What is a Transformer Pipeline?

A Transformer pipeline describes the flow of data from origin systems to destination systems and defines how to transform the data along the way.

Transformer pipelines are designed in Control Hub or in Transformer and executed by Transformer.

You can include the following stages in Transformer pipelines:

Origins

An origin stage represents an origin system. A pipeline can include multiple origin stages, but a pipeline can include only one origin configured to read from multiple tables.

If you use more than one origin in a pipeline, you must use a Join processor to join the data read by the origins. Each Join processor can join data from two input streams.

Processors

You can use multiple processor stages to perform complex transformations on the data.

Destinations

A destination stage represents a destination system. A pipeline can include multiple destination stages.

When you develop a pipeline, you can also include development stages to provide sample data and generate errors to test error handling.