Pipeline Error Record Handling

Pipeline error record handling determines how Data Collector processes error records that stages send to the pipeline for error handling. It also handles records deliberately dropped from the pipeline such as records without required fields.

The pipeline handles error records based on the Error Records property on the Error Records tab. When Data Collector encounters an unexpected error, it stops the pipeline and logs the error.

Pipelines provide the following error record handling options:
Discard
The pipeline discards the record. Data Collector includes the records in error record counts and metrics.
Send Response to Origin
The pipeline passes error records back to the microservice origin to be included in a response to the originating REST API client. Data Collector includes the records in error record counts and metrics. Use in microservice pipelines only.
Not valid in Data Collector Edge pipelines.
Write to Amazon S3
The pipeline writes error records and related details to Amazon S3. Data Collector includes the records in error record counts and metrics.

You define the Amazon S3 configuration properties.

Not valid in Data Collector Edge pipelines.

Write to Another Pipeline
The pipeline writes error records to an SDC RPC pipeline. Data Collector includes the records in error record counts and metrics.
When you write to another pipeline, Data Collector effectively creates an SDC RPC origin pipeline to pass the error records to another pipeline.
You need to create an SDC RPC destination pipeline to process the error records. The pipeline must include an SDC RPC origin configured to read error records from this pipeline.
For more information about SDC RPC pipelines, see SDC RPC Pipeline Overview (deprecated).
Not valid in Data Collector Edge pipelines.
Write to Azure Event Hub
The pipeline writes error records and related details to Microsoft Azure Event Hub. Data Collector includes the records in error record counts and metrics.
You define the configuration properties for the Azure Event Hub to use.
Not valid in Data Collector Edge pipelines.
Write to Elasticsearch
The pipeline writes error records and related details to Elasticsearch. Data Collector includes the records in error record counts and metrics.
You define the configuration properties for the Elasticsearch cluster to use.
Not valid in Data Collector Edge pipelines.
Write to File
The pipeline writes error records and related details to a local directory. Data Collector includes the records in error record counts and metrics.
You define the directory to use and the maximum file size. Error files are named based on the File Prefix pipeline property.
Write to file is not supported for cluster pipelines at this time.
Write to Google Cloud Storage
The pipeline writes error records and related details to Google Cloud Storage. Data Collector includes the records in error record counts and metrics.
You define the Google Cloud Storage configuration properties.
Not valid in Data Collector Edge pipelines.
Write to Google Pub/Sub
The pipeline writes error records and related details to Google Pub/Sub. Data Collector includes the records in error record counts and metrics.
You define the Google Pub/Sub configuration properties.
Not valid in Data Collector Edge pipelines.
Write to Kafka
The pipeline writes error records and related details to Kafka. Data Collector includes the records in error record counts and metrics.
You define the configuration properties for the Kafka cluster to use.
Not valid in Data Collector Edge pipelines.
Write to Kinesis
The pipeline writes error records and related details to Amazon Kinesis Streams. Data Collector includes the records in error record counts and metrics.
You define the configuration properties for the Kinesis stream to use.
Not valid in Data Collector Edge pipelines.
Write to MapR Streams
The pipeline writes error records and related details to MapR Streams. Data Collector includes the records in error record counts and metrics.
You define the configuration properties for the MapR Streams cluster to use.
Not valid in Data Collector Edge pipelines.
Write to MQTT
The pipeline writes error records and related details to an MQTT broker. Data Collector includes the records in error record counts and metrics.
You define the configuration properties for the MQTT broker to use.