Fragments Overview

A pipeline fragment is a stage or set of connected stages that you can use in pipelines. Use pipeline fragments to easily add the same processing logic to multiple pipelines and to ensure that the logic is used as designed.

For example, to use the same set of processing logic for all pipelines that process log data, you create a pipeline fragment that includes the processors you want to use, such as the Field Merger and Field Remover to organize record fields. You can include runtime parameters to change configuration values in each pipeline that uses the fragment. After you publish the fragment, you add the fragment to pipelines as needed, reusing the fragment processing logic.

Design pipeline fragments in Control Hub. You cannot design fragments in the Data Collector or Transformer UI.

You can create fragments for both Data Collector and Transformer pipelines.

You can use published fragments in pipelines. When you add the fragment to the pipeline, you optionally specify a prefix to include in the names of runtime parameters. The pipeline canvas displays each fragment as a single stage. You can expand the fragment stage to view the stages and stage configuration defined in the fragment, but you cannot edit the fragment from within the pipeline. This ensures that the fragment processing logic is used as designed.

You can edit and update published pipeline fragments. To prevent the invalidation of pipelines that use fragments, pipelines do not automatically inherit changes to published fragments. To configure a pipeline to use a different version of a fragment, you must edit the pipeline and update the fragment version.

When viewing a fragment, you can also view the list of pipelines that use the fragment. After you publish updates to a fragment, you can easily navigate to the various pipelines that use the fragment to determine whether they should use the latest version of the fragment. Then, you can update the fragment version that the pipelines use, as needed.