Using Fragments in Pipelines

After a pipeline fragment is published, you can use the fragment in a pipeline that uses the same execution engine type.

You can use a fragment like any other stage. The stage library lists fragments in their own category. When appropriate, the pipeline creation help bar includes a list of fragments that you can add to the pipeline. You can use multiple fragments in a pipeline and reuse the same fragment in a pipeline.

Note: You cannot use a fragment in another fragment.
By default, the pipeline canvas displays each fragment in the pipeline as a single stage. The input and output streams of the fragment stage correspond to the input and output streams in the fragment. You can expand a fragment stage to view the stages in the fragment and to verify the fragment logic:
  • From published or draft pipelines, click the Expand Fragments icon () located on the right side of the canvas by the navigation icons to expand all the fragment stages in the pipeline.
  • From draft pipelines, click a fragment stage and then click the Expand Fragments icon () in the pop-up menu to expand a single fragment stage.

After you expand a fragment stage, the Expand Fragments icon becomes the Collapse Fragments icon (), which you can click to hide the fragment details.

You cannot edit a fragment when designing a pipeline. To navigate away from the pipeline to edit the fragment, click the fragment name in either of the following locations in the Properties panel:

Using Fragment Versions

If a fragment has multiple versions, be sure to select the correct version when you design a pipeline. A pipeline uses the selected fragment version until changed.

When you publish an updated fragment, you can have Control Hub update pipelines that use the previous version of the fragment to use the latest version. At any time, you can manually change pipelines to use particular versions of fragments. To manually change the fragment version that a pipeline uses, edit the pipeline and select the fragment version.

You can see the pipelines that use a fragment version on the Pipelines tab of the fragment details pane, as shown below:

You can click a pipeline name to quickly navigate to the pipeline.

To view or change the fragment version used in a pipeline, select the fragment, then configure the Fragment Commit/Tag property as needed:

Stream Order

The stream order of a fragment stage in a pipeline follows the order and location of the stages in the fragment - from left to right across the primary branch, then across any remaining upper and lower branches, with all event streams displaying below data streams.

For example, the following fragment stage has four output streams:

The expanded fragment looks like this:

The stream order follows the primary branch, JSON Parser to Hadoop FS, then returns to the remaining Aggregator branch. It also lists the event streams after the data streams. So the resulting stream order for this fragment is as follows:
  • Input stream - JSON Parser input stream.
  • Output stream 1 - Stream Selector output stream 2.
  • Output stream 2 - Aggregator output stream.
  • Output stream 3 - Hadoop FS event stream.
  • Output stream 4 - Aggregator event stream.