System Sample Pipelines

The default system organization with an ID of admin includes a set of published pipelines. When data engineers in other organizations design pipelines in Control Hub, they can use any published pipeline belonging to the system organization as a system sample pipeline.

You can edit the provided system sample pipelines and create new system sample pipelines. You can also configure another organization to manage the system samples.

For example, you might create your own set of system sample pipelines that include the stages and processing your pipelines require. Data engineers can use the samples as a basis for their pipeline development.

To view the list of provided system sample pipelines, log in as the system administrator and then in the Pipeline Repository view, filter the list of pipelines by the system organization. The provided samples include Data Collector Edge, Data Collector, and Transformer sample pipelines.

A user with the System Administrator role can view the name and description of the sample pipelines, but cannot view or edit the pipeline configuration in the canvas. To manage the sample pipelines, assign the following roles to the system administrator account:
  • Pipeline Editor
  • Engine Creator or Administrator
  • Job Operator

The following image displays a few of the provided Data Collector sample pipelines in the Pipeline Repository view of the system organization:

To manage system sample pipelines, you can take either of the following approaches:

Modify the provided samples and create new samples in the system organization
You can edit or delete the provided samples. You can also create new sample pipelines in the system organization. All published pipelines that belong to the system organization can be used as system sample pipelines by any other organization.
Important: If you create additional user accounts in the system organization to manage the sample pipelines, assign them the required roles to manage pipelines. However, take care not to grant those users the System Administrator role. If you enable permission enforcement for the system organization, grant the users the required pipeline permissions.
Create new samples in another organization
You can create and configure another organization that manages system sample pipelines. You might want to configure another organization dedicated to the management of system sample pipelines to ensure that only system administrators log into and access the system organization.
You cannot move the sample pipelines provided with the system organization to another organization. However, you can recreate the same sample pipelines in another organization.
To configure another organization to manage system sample pipelines, modify the following property in the $DPM_CONF/pipelinestore-app.properties file:
  • pipeline.templates.organization - Name of the organization that manages system sample pipelines. All published pipelines that belong to this configured organization can be used as system sample pipelines by any other organization.
Restart Control Hub for the changed property to take effect.
Create user accounts for this organization and assign them the required roles to manage pipelines. If you enable permission enforcement for the organization, grant the user accounts the required pipeline permissions.
Note: The pipeline.templates.organizationUser property in the $DPM_CONF/pipelinestore-app.properties file defines the user account assigned to the Committed By property for the provided sample pipelines during installation or upgrade. There is no need to change the value of this property after completing the Control Hub installation or upgrade.