Multiple Offset Value Handling

When processing partitions, the JDBC Multitable Consumer origin can process multiple records with the same offset value. For example, the origin can process multiple records with the same timestamp in a transaction_date offset column.

Warning: When processing multiple records with the same offset value, records can be dropped if you stop the pipeline when the origin is processing a series of records with the same offset value.

When you stop the pipeline as the origin is processing a series of records with the same offset value, the origin notes the offset. Then, when you restart the pipeline, it starts with a record with the next logical offset value, skipping any unprocessed records that use the same last-saved offset.

For example, say you specified a datetime column as a user-defined offset column, and five records in the table share the same datetime value. Now say you happen to stop the pipeline after it processes the second record. The pipeline stores the datetime value as the offset where it stopped. When you restart the pipeline, processing begins with the next datetime value, skipping the three unprocessed records with the last-saved offset value.