Managing Pipelines and Fragments

You manage pipelines and pipeline fragments in the same way.

Managing pipelines in the Pipelines view and fragments in the Fragments view includes duplicating an existing pipeline or fragment and filtering the list of displayed pipelines or fragments.

You can add sticky notes to the pipeline canvas to communicate with other users as you build a pipeline or fragment. As you build a pipeline, you can start a draft run of the pipeline to quickly test the pipeline logic.

When you have finished building a pipeline, you publish or check in the pipeline so that it can be added to a job. From the Pipelines view, you can also create a job for a published pipeline and view a list of jobs that include the published pipeline.

Similarly, when you have finished building a fragment, you publish or check in the fragment so that it can be added to a pipeline.

When needed, you can delete pipeline versions that are not included in jobs and delete fragment versions that are not included in pipelines.

For details about building pipelines in the canvas, including how to configure individual pipeline stages, see the appropriate engine documentation for the pipeline type:

Duplicating a Pipeline or Fragment

Duplicate a pipeline or fragment when you want to keep the existing version while continuing to configure a duplicate version. A duplicate is an exact copy of the original pipeline or fragment.

You can duplicate draft and published pipelines and fragments. When you duplicate a pipeline or fragment, you can rename the duplicate and specify the number of copies to make.

  1. To duplicate a pipeline, in the Navigation panel, click Build > Pipelines.
    To duplicate a fragment, in the Navigation panel, click Build > Fragments.
  2. Select a pipeline or fragment in the list and then click the Duplicate icon: .
  3. Enter a name and optional description for the duplicate.
  4. Enter the number of copies to make.

    When you create multiple copies, Control Hub appends an integer to the pipeline or fragment name. For example, if you enter the name test and create two copies of the pipeline, Control Hub names the duplicate pipelines test1 and test2.

  5. Click Duplicate.
    The duplicate pipelines and fragments display in a version 1 draft state.

Filtering Pipelines and Fragments

In the Pipelines and Fragments views, you can filter the list of displayed pipelines or fragments by execution engine type, status, or label. Or you can search for pipelines or fragments by name, description, or commit message.

You can filter only published pipelines and fragments by label. You cannot filter draft pipelines and fragments by label.
Note: By default, filtering is replaced by the new search functionality. If needed, you can disable search and revert to the original filtering functionality.
  1. To filter pipelines, in the Navigation panel, click Build > Pipelines.

    To filter pipeline fragments, in the Navigation panel, click Build > Fragments.

  2. If the Filter column does not display, click the Toggle Filter Column icon: .

    The following image displays the Filter column in the Pipelines view:

  3. To search for pipelines or fragments by name, description, or commit message, enter text in the search field, and then press Return.
  4. Select an execution engine type, published or draft status, or pipeline label to additionally filter the list of pipelines or fragments.
  5. Select the Keep Filter Persistent checkbox to retain the filter when you return to the view.
    Tip: To share the applied filter, copy the URL and send it to another user in your organization.

Using Sticky Notes

As you build a pipeline or pipeline fragment, you can add sticky notes to the pipeline canvas to include notes that might be useful to you or your team.

For example, you might add a note reminding you to test newly added expressions, as follows:

When you duplicate or export a pipeline or fragment with sticky notes, the notes are retained. However, if you add a published fragment with sticky notes to a pipeline, you cannot view the fragment notes when you expand the fragment in the pipeline. You can view fragment sticky notes only when you open the fragment in the pipeline canvas.

You can complete the following actions as you use sticky notes:
Add
To add a sticky note, select the pipeline canvas or select a stage in the canvas, and then click the Stickies icon (). When you add a sticky note with a stage selected, the note is added near the stage.
Click inside the note to add text to the note.
Edit
To edit the sticky note text, click inside the note and then add or remove text.
To change the sticky note color, click the Change Color icon () in the note header, and then select another color.
Move
To move a sticky note, select the note header and then drag the note to another location on the canvas.
Delete
To delete a sticky note, click the Delete icon () in the note header.

Starting a Draft Run of a Pipeline

While editing a pipeline in the pipeline canvas, you can start a draft run to quickly test the pipeline logic.

You can start a draft run of a fully configured draft pipeline. The Draft Run menu becomes active in the pipeline canvas when a draft pipeline is complete.

You cannot start a draft run of a published pipeline. To run a published pipeline, you must first create a job for the published pipeline and then start the job.

For more information, see Draft Runs.

Viewing Published Pipelines and Fragments

You can view the complete configuration details of each published pipeline or pipeline fragment in the pipeline canvas. To view a published pipeline, click the pipeline name in the Pipelines view. To view a published fragment, click the fragment name in the Fragments view.

When the pipeline or fragment is in development, the pipeline canvas displays the pipeline or fragment in edit mode. When the pipeline or fragment is published, it displays in read-only mode. You can navigate through the canvas to review pipeline or fragment details.

All properties display in the Configuration tab in the properties panel.

In a pipeline, you can click the Rules tab in the properties panel to view metric, data, and data drift rules configured for the pipeline. You can select any stage in the pipeline and then click the Configuration tab in the properties panel to view the stage properties.

In a fragment, you can view data rules and data drift rules configured in the fragment by selecting the stream where the rule is defined. You can also select any stage in the pipeline and then click the Configuration tab in the properties panel to view the stage properties.

Viewing Jobs that Include a Pipeline

When viewing a pipeline version in the pipeline canvas, you can view the complete list of job instances and job templates that include that pipeline version.

  1. In the Navigation panel, click Build > Pipelines.
  2. Click the name of a pipeline to open the pipeline in the pipeline canvas.

    By default, the latest version of the pipeline opens. To view an earlier pipeline version, click the version number next to the pipeline name in the toolbar, and then select another version, as follows:

  3. In the properties panel, click the Jobs tab.

    The Jobs tab lists all job instances and job templates that include that pipeline version.

    For example, the following image shows that version 3 of the Social Feeds Dataflows pipeline is included in the job named Job for Social Feeds Dataflows:

Publishing a Pipeline

You publish pipelines after you design them in the pipeline canvas. You can publish valid pipelines. Control Hub performs explicit validation before publishing a pipeline.

Publish a pipeline to create a job that runs the pipeline, to use the pipeline as a sample pipeline, or to retain the published pipeline version for historical reference. You can only use published pipelines in jobs or as sample pipelines.

  1. While viewing a pipeline in edit mode, click the Check In icon: .

    The Check In wizard appears.

  2. Enter a commit message.

    As a best practice, state what changed in this version so that you can track the commit history of the pipeline.

  3. Click one of the following buttons:
    • Cancel - Cancels publishing the pipeline.
    • Publish and Next - Publishes the pipeline and continues.
    • Publish and Close - Publishes the pipeline and exits the wizard.
  4. In the Share Pipeline step, share the pipeline with other users and groups.

    Or, you can share the pipeline with others at a later time. For details, see Pipeline Permissions.

  5. If the pipeline is not included in a job, click one of the following buttons:
    • Save & Create New Job - Saves permissions applied to the published pipeline, and then displays the Create Job Instances wizard so that you can create a new job for this pipeline version.
    • Save & Exit - Saves permissions applied to the published pipeline and exits the wizard.
  6. If the pipeline is included in a job, click one of the following buttons:
    • Save & Next - Saves permissions applied to the published pipeline and continues.
    • Save & Exit - Saves permissions applied to the published pipeline and exits the wizard.
  7. In the Update Jobs step, select whether to update jobs that currently use the published pipeline.

    Select the jobs that you want to update to use the latest published version and click Update. Review the jobs to be updated, and then click Close. You can also choose to create a new job for this pipeline version.

    Or, choose to skip the update and close the wizard. You can also choose to skip the update and create a new job for this pipeline version.

Publishing a Fragment

You publish fragments after you design them in the pipeline canvas. You can publish valid fragments. Control Hub performs explicit validation before publishing a fragment.

Publish a fragment to use the fragment in a pipeline. Pipelines can only use published fragments.

  1. While viewing a fragment in edit mode, click the Check In icon: .

    The Check In wizard appears.

  2. Enter a commit message.

    As a best practice, state what changed in this version so that you can track the commit history of the fragment.

  3. Click one of the following buttons:
    • Cancel - Cancels publishing the fragment.
    • Publish or Publish and Update - Publishes the fragment and continues.
    • Publish and Close - Publishes the fragment and exits the wizard.
  4. In the Share Fragment step, share the fragment with other users and groups.

    Or, you can share the fragment with others at a later time. For details, see Pipeline Fragment Permissions.

  5. If the fragment is not included in a pipeline, click Save & Exit.
  6. If the fragment is included in a pipeline, click one of the following buttons:
    • Save & Next - Saves permissions applied to the published fragment and continues.
    • Save & Exit - Saves permissions applied to the published fragment and exits the wizard.
  7. In the Update Pipelines step, select whether to update pipelines that currently use the published fragment.

    Select the pipelines that you want to update to use the latest published version and click Update. Review the pipelines to be updated, and then click Close.

    Or, choose to skip the update and close the wizard.

Deleting Pipelines or Fragments

From the Pipelines and Fragments views, you can delete all versions of a pipeline or fragment from the repository.
Note: You can delete a specific version of a pipeline or fragment when you view the version history from the pipeline canvas.

You can delete pipeline versions that are not included in jobs. To delete pipeline versions that are included in jobs, you must first delete the job or edit the job so that it no longer includes the pipeline version that you want to delete.

Similarly, you can only delete fragments that are not used in pipelines. To delete fragment versions that are included in pipelines, you must first remove the fragment versions from all pipelines or delete those pipelines.

  1. To delete all versions of a pipeline, in the Navigation panel, click Build > Pipelines.

    To delete all versions of a fragment, in the Navigation panel, click Build > Fragments.

  2. Hover over the pipeline or fragment that you want to delete, and then click the Delete All Versions icon:
  3. In the confirmation window, click Delete.