MapR DB CDC

The MapR DB CDC origin performs parallel processing and enables the creation of a multithreaded pipeline. The origin reads changed data from MapR DB that has been written to MapR Streams.

The MapR DB CDC origin uses multiple concurrent threads based on the Number of Threads property. MapR Streams distributes partitions equally among all the consumers in a group.

When performing multithreaded processing, the MapR DB CDC origin checks the list of topics to process and creates the specified number of threads. Each thread connects to MapR Streams and creates a batch of data from a partition assigned by MapR Streams. Then, it passes the batch to an available pipeline runner.

A pipeline runner is a sourceless pipeline instance - an instance of the pipeline that includes all of the processors, executors, and destinations in the pipeline and handles all pipeline processing after the origin. Each pipeline runner processes one batch at a time, just like a pipeline that runs on a single thread. When the flow of data slows, the pipeline runners wait idly until they are needed, generating an empty batch at regular intervals. You can configure the Runner Idle Time pipeline property to specify the interval or to opt out of empty batch generation.

Multithreaded pipelines preserve the order of records within each batch, just like a single-threaded pipeline. But since batches are processed by different pipeline runners, the order that batches are written to destinations is not ensured.

For example, say you set the Number of Threads property to 5. When you start the pipeline, the origin creates five threads, and Data Collector creates a matching number of pipeline runners. The threads are assigned to different partitions as defined by MapR Streams. Upon receiving data, the origin passes a batch to each of the pipeline runners for processing.

At any given moment, the five pipeline runners can each process a batch, so this multithreaded pipeline processes up to five batches at a time. When incoming data slows, the pipeline runners sit idle, available for use as soon as the data flow increases.

For more information about multithreaded pipelines, see Multithreaded Pipeline Overview. For more information about the MapR Streams, see the MapR Streams documentation.

All MapR DB changed data includes an _id field. The MapR DB CDC origin includes the _id field as a field in the generated record. If needed, you can use the Field Remover processor in the pipeline to remove the _id field.

The _id field can contain string or binary data. The MapR DB CDC origin can process data that includes string or binary data. The origin cannot read _id fields with a combination of string and binary data.

When incoming data includes a string _id field, the origin creates the _id field in the record as a String. When incoming data includes a binary _id field, the origin converts the data to String and then includes the field in the record.

The MapR DB CDC origin includes the CRUD operation type in the sdc.operation.type record header attribute.

If you use a CRUD-enabled destination in the pipeline such as JDBC Producer or Elasticsearch, the destination can use the operation type when writing to destination systems. When necessary, you can use an Expression Evaluator processor or any scripting processor to manipulate the value in the header attribute. For an overview of Data Collector changed data processing and a list of CRUD-enabled destinations, see Processing Changed Data.

You can add custom configuration properties to the MapR DB CDC origin. You can use any MapR or Kafka property supported by MapR Streams. For more information, see the MapR Streams documentation.

When you add a configuration property, enter the exact property name and the value. The MapR DB CDC origin does not validate the property names or values.