Pipeline, Fragment, and Topology Export and Import

Pipelines, pipeline fragments, and topologies are versioned objects with either a draft or published state. You can export and import draft and published pipelines and fragments. You can export and import only published topologies.

When you export a pipeline, fragment, or topology, you export a single version of the object. You do not export all published versions at once.

When you export an object, you also export any dependent objects. For example, topologies contain jobs and pipeline versions, and can contain pipeline fragments. When you export and import a topology, Control Hub also exports and imports all jobs, pipeline versions, and fragment versions included in the topology. Control Hub uses the same rules to import dependent jobs, pipelines, and fragments as it uses when you import them individually.

When you export an object that uses connections, Control Hub exports the connection metadata only. Control Hub does not export the connection credentials or other configured values.

Because pipelines, fragments, and topologies are versioned objects, Control Hub retains the commit ID with each export in addition to the object ID. The commit ID uniquely identifies the version. Control Hub handles the version number of the imported object differently, based on the object type and on the following locations where you can perform the import:

Import Objects from a View

When you import a pipeline, fragment, or topology from the Pipelines, Fragments, or Topologies view, Control Hub retains the original version number of the imported object. Because Control Hub retains version numbers, the target organization might include gaps in the version numbers.

For example, the source organization includes the following pipeline versions: v1, v2, v3. You export the latest pipeline version and import it into the target organization for the first time. Control Hub imports the pipeline as v3. The target organization does not include a v1 or v2 version of the pipeline.

When you import an object from a view, Control Hub checks if the imported object already exists in the target organization by comparing commit and object IDs. After comparing the IDs, Control Hub handles the import as follows:
  • If the imported object has different IDs, Control Hub imports the object.
  • If importing a pipeline or fragment and the IDs are the same, you can choose to import the object as a new pipeline or fragment with a new commit and object ID.
  • If importing a topology and the IDs are the same, Control Hub does not import the topology.

Control Hub retains the original version number when importing a pipeline or fragment as a new object.

For example, you export a v4 pipeline from the source organization and import it into the target organization. Control Hub imports the pipeline as v4. If you import the same pipeline again, Control Hub informs you that the pipeline already exists. You choose to import the object as a new pipeline, and so Control Hub imports the pipeline as v4 with a new commit and object ID.

Import a Pipeline Version from the Pipeline Canvas

When a published pipeline is open in the pipeline canvas, you can import a new version of that pipeline. You cannot import a new version when a draft pipeline is open in the canvas.

When you import a pipeline version from the canvas, Control Hub updates the version number so that it is the next available version of the pipeline version currently open in the canvas.

For example, a pipeline includes the following versions: v1, v2, and v3. You have exported all three versions to create backups. A user mistakenly deletes v1, but now you want to revert to that version. You open v3 of the pipeline in the canvas and then import v1. Control Hub updates the imported version to v4. If you instead open v2 of the pipeline in the canvas and then import v1, Control Hub updates the imported version to v2.1.

Parent Versions

Each object version also tracks the parent object version. For example, when you create and publish a new pipeline, the pipeline version is v1 and the parent version is v0. When you edit and publish the pipeline to create version v2, the parent version becomes v1.

When you import a pipeline, fragment, or topology from a view, Control Hub always lists the parent version as v0. For example, if you export and import a v3 topology to a target organization, Control Hub lists the imported topology as v3 and lists the parent version as v0 - indicating that the topology was imported.

When you import a new version of a pipeline from the pipeline canvas, the pipeline version currently open in the canvas becomes the parent version of the imported version.