Pipeline, Fragment, and Topology Export and Import

Pipelines, pipeline fragments, and topologies are versioned objects with either a draft or published state. You can export and import published pipelines, fragments, and topologies only.

When you export a pipeline, fragment, or topology, you export a single version of the object. You do not export all published versions at once.

When you export an object, you also export any dependent objects. For example, topologies contain jobs and pipeline versions, and can contain pipeline fragments. When you export and import a topology, Control Hub also exports and imports all jobs, pipeline versions, and fragment versions included in the topology. Control Hub uses the same rules to import dependent jobs, pipelines, and fragments as it uses when you import them individually.

When you export an object that uses connections, Control Hub exports the connection metadata only. Control Hub does not export the connection credentials or other configured values.

Parent Versions

Each object version also tracks the parent object version.

For example, when you create and publish a new pipeline, the pipeline version is v1 and the parent version is v0. When you edit and publish the pipeline to create version v2, the parent version becomes v1.

When you import a pipeline, fragment, or topology, Control Hub always lists the parent version as v0. For example, if you export and import a v3 topology to a target organization, Control Hub lists the imported topology as v3 and lists the parent version as v0 - indicating that the topology was imported.

Versioning of Imported Objects

Because pipelines, fragments, and topologies are versioned objects, Control Hub retains the commit ID with each export and import in addition to the object ID. The commit ID uniquely identifies the version.

When you import an object that already exists in the target organization, Control Hub compares the commit IDs. If the commit IDs are different, Control Hub imports the object. If the commit IDs are the same, Control Hub does not import the object.

When you import a pipeline, fragment, or topology, Control Hub retains the same version number. If the version number already exists in the target organization, Control Hub updates the minor version number. For example, v2 becomes v2.1 in the target organization.

Because Control Hub retains version numbers, the target organization might include gaps in the version numbers.

Let's look at some examples of how Control Hub handles the versioning of imported pipelines, fragments, and topologies:
  • The source organization includes the following pipeline versions: v1, v2, v3. You export the pipeline and import it into the target organization for the first time. Control Hub imports the pipeline as v3. The target organization does not include a v1 or v2 version of the pipeline.
  • The source organization includes a v1 pipeline that uses v2 of a pipeline fragment. You export the pipeline and import it into the target organization for the first time. Control Hub imports the pipeline as v1 and the fragment as v2. You then import the v1 pipeline into the target organization again with no changes. Since the commit IDs for the source and target pipelines match, Control Hub does not import the pipeline again. As a result, the pipeline stays as v1 and the fragment as v2 in the target organization.
  • The source organization includes a v1 pipeline. You export the pipeline and import it into the target organization for the first time. Control Hub imports the pipeline as v1. You edit and publish the pipeline in both the source and target organizations, creating a v2 in both. You then export the pipeline from the source organization again and import it into the target organization. Because v2 already exists in the target organization with a unique commit ID, Control Hub imports the pipeline as v2.1.
  • The source organization includes the following pipeline fragment versions: v1, v2, v3. You have exported and imported each version individually to the target organization. You manually delete v2 and v3 from the source organization. And you delete v1 and v2 from the target organization. Then you export the v1 fragment from the source organization again and import it into the target organization. Control Hub imports the fragment as v1. The target organization now includes v1 and v3 versions of the fragment.
  • The source organization includes the following topology versions: v1 and v2. The v2 topology contains two jobs - JobA and JobB - each of which include a v3 pipeline. You export the topology and import it into the target organization for the first time. Control Hub imports the objects as the following versions:
    • v2 topology
    • JobA
    • JobB
    • v3 pipeline included in JobA
    • v3 pipeline included in JobB

    The target organization does not include a v1 version of the topology. The target organization does not include a v1 or v2 version of either pipeline.