Overview

You build pipelines in Control Hub to define how data flows from origin to destination systems and how the data is processed along the way.

You can create Data Collector, Transformer, or Transformer for Snowflake pipelines. The pipeline types differ in the available functionality and in how the engine runs the pipeline. For more information, see Comparing IBM StreamSets Pipelines and "Comparing Snowflake and Other StreamSets Engines" in the Transformer for Snowflake documentation.

When you create a Data Collector or Transformer pipeline, you select the authoring engine to use for pipeline design.

Generally, you do not select an authoring engine for a Transformer for Snowflake pipeline since it runs on the hosted Transformer for Snowflake engine by default. You select an Transformer for Snowflake authoring engine only when the account agreement for your organization allows you to deploy Transformer for Snowflake engines.

Control Hub provides release management for your pipelines and pipeline fragments by maintaining a version history of each pipeline or fragment. You can compare one pipeline or fragment version with another version. You can also add tags to pipeline and fragment versions to easily differentiate between versions.

In the Pipelines view, you manage all IBM StreamSets pipelines the same way. For example, you use the same steps to create, duplicate, filter, publish, and delete all pipeline types.

As you build pipelines and fragments, you can use data preview to see how source data passes through the pipeline or fragment.