Fragments Overview

A pipeline fragment is a stage or set of connected stages that you can use in pipelines. Use pipeline fragments to easily add the same processing logic to multiple pipelines and to ensure that the logic is used as designed.

For example, to use the same set of processing logic for all pipelines that process log data, you create a pipeline fragment that includes the processors you want to use, such as the Field Merger and Field Remover to organize record fields. You can include runtime parameters to change configuration values in each pipeline that uses the fragment. After you publish the fragment, you add the fragment to pipelines as needed, reusing the fragment processing logic.

You can create fragments for all engine types - Data Collector, Transformer, and Transformer for Snowflake. You can use fragments only in pipelines created using the same engine type. For example, you can use Data Collector fragments only in Data Collector pipelines, and Transformer fragments only in Transformer pipelines.

You can use published fragments in pipelines. When you add the fragment to the pipeline, you optionally specify a prefix to include in the names of runtime parameters. The pipeline canvas displays each fragment as a single stage. You can expand the fragment stage to view the stages and stage configuration defined in the fragment, but you cannot edit the fragment from within the pipeline. This ensures that the fragment processing logic is used as designed.

You can edit and update published pipeline fragments. To prevent the invalidation of pipelines that use fragments, pipelines do not automatically inherit changes to published fragments. To configure a pipeline to use a different version of a fragment, you must edit the pipeline and update the fragment version.

When viewing a fragment, you can also view the list of pipelines that use the fragment. After you publish updates to a fragment, you can easily navigate to the various pipelines that use the fragment to determine whether they should use the latest version of the fragment. Then, you can update the fragment version that the pipelines use, as needed.

Working with Fragments

The Fragments view lists all pipeline fragments that you have access to.

You can complete the following tasks in the Fragments view:

  • Create, configure, and publish a fragment.
  • Search for fragments.
  • View fragment details, including the user who committed the fragment and the commit time and message.
  • View the configuration of each fragment and each stage in the fragments.
  • Duplicate a fragment.
  • Import fragments that have been exported from Control Hub.
  • Export fragments from the repository.
  • Share a fragment with other users and groups.
  • Delete a fragment or fragment version.

The following image shows the available pipeline fragments and displays the details of one of the fragments:

Note the following icons that display for all fragments or when you hover over a single fragment. You'll use these icons frequently as you manage fragments:

Icon Name Description
Data Collector Engine Type Displayed for Data Collector fragments.
Transformer Engine Type Displayed for Transformer fragments.
Transformer for Snowflake Engine Type Displayed for Transformer for Snowflake fragments.
Create New Pipeline Fragment Create a new pipeline fragment.
Import Import fragments that have been exported from Control Hub.
Refresh Refresh the list of fragments.
Duplicate Duplicate the selected fragment.
Compare with Previous Version Compare a fragment version with a previous version.
Export Export the selected fragments.
Share Share the fragment with other users and groups, as described in Permissions.
Delete Delete the fragment or a fragment version.