StreamSets TransformerTM is an execution engine that runs data processing pipelines on Apache Spark, an open-source cluster-computing framework. Because Transformer pipelines run on Spark deployed on a cluster, the pipelines can perform transformations that require heavy processing on the entire data set in batch or streaming mode.

Transformer functions as a Spark client that launches distributed Spark applications.

Transformer can run pipelines in batch mode. A batch pipeline processes all available data in a single batch, and then stops.

Transformer can run pipelines in streaming mode. A streaming pipeline maintains connections to origin systems and processes data at user-defined intervals. The pipeline runs continuously until you manually stop it.

For users already familiar with StreamSets Data Collector pipelines, here's how Transformer pipelines are similar... and different.

StreamSets provides tutorials and sample pipelines to help you learn about using Transformer.

You can preview data to help build or fine-tune a pipeline. You can preview complete or incomplete pipelines.

In Preview mode, Transformer displays different colors for different types of data. Transformer uses other codes and formatting to highlight changed fields.

When previewing data for a processor, you can preview both the input and the output data. You can display the output records in the order that matches the input records or in the order produced by the processor.

After running preview for a pipeline, you can view the input and output schema for each stage on the Schema tab in the pipeline properties panel. The schema includes each field path and data type.

When running preview, you can edit stage properties to see how the changes affect preview data. For example, you might edit the condition in a Stream Selector processor to see how the condition alters which records pass to the different output streams.