Skip Offset Tracking

You can configure any origin that tracks offsets to skip tracking offsets. You cannot configure the Surrogate Key Generator processor to skip tracking offsets.

Skip offset tracking when you want an origin to treat every batch like the pipeline just started running for the first time. This can be appropriate in certain situations.

For example, say you want a pipeline to process all data in a Hive table every time you run the pipeline. To get the desired results, you use the Hive origin in a batch pipeline to read all data in a single batch. Then, you enable the Skip Offset Tracking property in the origin to ensure that all data is processed with each pipeline run. If you allow offset tracking, the pipeline reads all available data in the first pipeline run, but in subsequent runs, it reads only the data that arrived since the last pipeline run.

Skipping offset tracking is critical in a slowly changing dimension streaming pipeline, where you want to compare change data against the latest master dimension data. In this case, you skip offset tracking in the master origin, so the master origin reads the master dimension data every time the pipeline processes data from the change origin. This allows the Slowly Changing Dimension processor to compare changes against the master dimension data. If you don't skip offset tracking, the master origin only reads new master dimension data, providing an incomplete master data set for comparison.

Skipping offset tracking can also be totally inappropriate, so you should skip offset tracking with care.

Note that most streaming pipelines require offset tracking to function as expected. For example, you typically want a Kafka origin to read messages from the specified initial offset, to process all existing messages from that point forward, and to continue processing newly arrived messages. If you skip offset tracking, the origin reprocesses data from the initial offset with each batch.

To skip tracking offsets, on the General tab of the origin, select the Skip Offset Tracking property. If the origin does not have the property, it does not track offsets.