Using Fragments in Pipelines

After a pipeline fragment is published, you can use the fragment in a pipeline that uses the same engine type.

You can use a fragment like any other stage. The stage library lists fragments in their own category. When appropriate, the stage selector includes a list of fragments that you can add to the pipeline. You can use multiple fragments in a pipeline and reuse the same fragment in a pipeline.

Note: You cannot use a fragment in another fragment.

By default, the pipeline canvas displays each fragment in the pipeline as a single stage with a puzzle piece icon: . You can optionally modify the icon to represent the fragment processing logic.

The input and output streams of the fragment stage correspond to the input and output streams in the fragment. You can expand a fragment stage to view the stages in the fragment and to verify the fragment logic as follows:
  • From published or draft pipelines, click the Expand Fragments icon () located on the right side of the canvas by the navigation icons to expand all the fragment stages in the pipeline.
  • From draft pipelines, click a fragment stage and then click the Expand Fragments icon () in the pop-up menu to expand a single fragment stage.

After you expand a fragment stage, the Expand Fragments icon becomes the Collapse Fragments icon (), which you can click to hide the fragment details.

You cannot edit a fragment when designing a pipeline. To navigate away from the pipeline to edit the fragment, select the fragment in the canvas and then click the Edit Fragment icon () above the fragment or the fragment name in the properties panel:

Fragment Icons

By default when you add a published fragment to a pipeline, the fragment displays as a single stage with a puzzle piece icon. For example, when the Flatten and Mask fragment is added to a pipeline, the fragment displays as follows:

You can modify the icon to represent the fragment processing logic. For example, you might configure the Flatten and Mask fragment to use the predefined Process icon, as follows:

Modifying a Fragment Icon

When you modify a fragment icon, you can select from a set of predefined icons, or you can upload an image to use as the icon.

You can upload images with a .jpg, .jpeg, .gif, .png, or .svg file name extension. Uploaded images cannot exceed 100 KB in size.

  1. While viewing a fragment in edit mode, click the General tab in the properties panel.
  2. Click the Edit icon next to the Icon property:

  3. In the Icons dialog box, select one of the predefined icons or upload a custom image to use as the icon.
    Note: You can upload a single image to the dialog box. Uploading another image replaces the existing uploaded image.

    The properties panel displays the modified icon.

    After you publish the fragment, all pipelines that use that fragment version display the fragment as a single stage with the modified icon.

Using Fragment Versions

If a fragment has multiple versions, be sure to select the correct version when you design a pipeline. A pipeline uses the selected fragment version until changed.

When you publish an updated fragment, you can have Control Hub update pipelines that use the previous version of the fragment to use the latest version. At any time, you can manually change pipelines to use particular versions of fragments. To manually change the fragment version that a pipeline uses, edit the pipeline and select the fragment version.

You can see the pipelines that use a fragment version on the Pipelines tab of the fragment details pane, as shown below:

You can click a pipeline name to quickly navigate to the pipeline.

To view or change the fragment version used in a pipeline, select the fragment, then configure the Fragment Commit/Tag property as needed:

Stream Order

The stream order of a fragment stage in a pipeline follows the order and location of the stages in the fragment - from left to right across the primary branch, then across any remaining upper and lower branches.

For example, the following fragment stage has three output streams:

The expanded fragment looks like this:

The stream order follows the primary branch, JSON Parser to Amazon S3, then returns to the remaining Windowing Aggregator branch. So the resulting stream order for this fragment is as follows:
  • Input stream - JSON Parser input stream.
  • Output stream 1 - Stream Selector output stream 2.
  • Output stream 2 - Stream Selector output stream 3.
  • Output stream 3 - Windowing Aggregator output stream.