Reset Pipeline Offsets

You can optionally reset all pipeline offsets before starting a pipeline. When you reset pipeline offsets, Transformer runs the pipeline like it is the very first pipeline run.

For example, say you have a batch pipeline that runs weekly. It includes an ADLS Gen2 origin that reads files from a /logs directory. After the pipeline processes all available data, the origin notes the offset - in this case, the last-modified timestamp of the last processed file. Then, the pipeline comes to a stop. The next time you run the pipeline, the pipeline processes only files with a last-modified timestamp after that offset.

Now, let's say you need to change the destination system that the pipeline writes to, and you want to reprocess all available data to write the results to the new destination system. To do this, you replace the destination in the pipeline. Then, when you start the pipeline, you use the Reset Offsets and Start option.

The pipeline processes all available data in a single batch and stops. As before, it stores the offset. Then on subsequent pipeline runs, it continues processing from the last-saved offset.

To reset pipeline offsets before starting a pipeline, click the menu arrow to the right of the Start button (), then click Reset Offsets & Start.