Start Pipelines (deprecated)

The Start Pipelines origin starts one or more pipelines in parallel when the pipeline starts.

Important: This stage is deprecated and may be removed in a future release.
The Start Pipelines origin is an orchestration stage that you use in orchestration pipelines. Orchestration stages perform tasks, such as schedule and start pipelines and Control Hub jobs, that you can use to create an orchestrated workflow across the StreamSets platform.

The Start Pipelines origin can start pipelines that run on any StreamSets execution engine, such as Data Collector or Transformer. The origin can start pipelines that run on the execution engine specified in the stage. To start pipelines on a different execution engine, you can use a Start Pipelines processor.

The origin generates a record that contains a list of the started pipelines and information about those pipelines. You can pass the record to an orchestration stage to trigger another task. Or, you can pass it to a non-orchestration stage to perform other processing. For example, you might use a Stream Selector processor to pass the record to different stages based on a pipeline completion status.

When you configure the Start Pipelines origin, you define the URL of the execution engine that runs the pipelines. You specify the names or IDs of the pipelines to start along with any runtime parameters to use. For an execution engine registered with Control Hub, you specify the Control Hub URL, so the origin starts the pipelines through Control Hub.

You can configure the origin to reset the origins in the pipelines when possible, and to run the pipelines in the background. When running pipelines in the background, the origin immediately passes its generated record downstream instead of waiting for the pipelines to finish.

You also configure the user name and password to run the pipeline and can optionally configure SSL/TLS properties.

Pipeline Execution and Data Flow

The Start Pipelines origin starts the specified pipelines when the pipeline starts. The origin creates a record with task details and passes it downstream based on how the pipelines run:

Run pipelines in the foreground
By default, the origin starts pipelines that run in the foreground. When the pipelines run in the foreground, the origin passes the orchestration record downstream after all the started pipelines complete.
Run pipelines in the foreground to ensure that the task defined in the subsequent stage is not performed until all of the pipelines complete.
Run pipelines in the background
You can configure the origin to start pipelines that run in the background. When pipelines run in the background, the origin updates and passes the orchestration record downstream immediately after starting the pipelines.
The origin does not track or provide information about whether the pipelines finish successfully.
Run pipelines in the background to enable the task defined in the subsequent orchestration stage to occur in parallel with the pipelines started by the origin.

Generated Record

The Start Pipelines origin creates an orchestration record that includes information about the pipelines that it starts.

The record includes the following fields:
Field Name Description
orchestratorTasks List Map field that contains task details for the orchestration pipeline.

Most orchestration stages add details about their completed tasks within this field.
<unique task name> List Map field within the orchestratorTasks field that contains the following fields:
  • pipelineIds - IDs of the pipelines that the stage is configured to start.
  • pipelineResults - List Map of pipeline run details for each pipeline.
  • success - Boolean field that indicates whether all pipelines completed successfully.

    Included only when the pipelines run in the foreground.
<pipeline ID> List Map field within the pipelineResults field that provides details about each pipeline. Contains the following fields:
  • pipelineId - ID of the pipeline.
  • pipelineTitle - Name of the pipeline.
  • startedSuccessfully - Boolean field that indicates whether a pipeline was started successfully.
  • finishedSuccessfully - Boolean field that indicates whether a pipeline completed successfully.

    Included only when the pipeline runs in the foreground.

  • pipelineMetrics - Map field that contains metrics for the pipeline and individual pipeline stages.

    Included only when the pipeline runs in the foreground.

  • pipelineStatus - Status of the pipeline. For more information, see Understanding Pipeline States.
  • pipelineStatusMessage - Status message for the pipeline.
  • committedOffsetsStr - Offset details for the pipeline.

    Included only when the pipeline runs in the foreground.

For example, the following preview shows information provided by a Start Pipelines origin with the Load_ADLS task name. The origin runs one pipeline in the background:

Note that the startedSuccessfully and pipelineStatus fields indicate that pipeline was started successfully. There is no finishedSuccessfully field because the pipeline has not yet completed.

For an example of an orchestration record, see Example.

Configuring a Start Pipelines Origin

Configure a Start Pipelines origin to start a Data Collector or Transformer pipeline. The Start Pipelines origin is an orchestration stage that you use in orchestration pipelines.
Important: This stage is deprecated and may be removed in a future release.
  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the pipeline for error handling.
    • Stop Pipeline - Stops the pipeline.
  2. On the Pipeline tab, configure the following properties:
    Pipeline Property Description
    Task Name Name for the task to perform. The name must be unique within the pipeline.
    The task name is used to group the data that the stage adds to the generated record.
    Tip: Avoid using spaces in the task name if you want to access information added to the record, such as a pipeline status, from within the pipeline.
    Execution Engine URL URL of the execution engine that runs the pipelines. Execution engines include Data Collector and Transformer.
    Pipelines List of pipelines to start in parallel. For each pipeline, enter:
    • Identifier Type - Information used to identify the pipeline. Select Pipeline ID or Pipeline Title.
    • Identifier - ID or name of the pipeline.

      To find the pipeline ID from the execution engine UI, click the pipeline canvas and then click the General tab in the Properties panel.

      To find the pipeline ID from Control Hub, in the Pipelines view, expand the pipeline and click Show Additional Info.

    • Runtime Parameters - Parameters defined in a pipeline and specified when starting the pipeline.
      Use the following format: 
      {
      "<parameter name>": <numeric value>,
      "<parameter name>": "<string value>"
}

    To include another pipeline, click the Add icon.

    You can use simple or bulk edit mode to specify pipelines.
    Reset Origin Resets the origin before starting a pipeline, if the origin can be reset. For a list of origins that can be reset, see Resetting the Origin in the Data Collector documentation.
    Control Hub Enabled Starts pipelines through Control Hub. Select this property when the execution engine is registered with Control Hub.
    Control Hub URL URL of Control Hub where the execution engine is registered:
    • For Control Hub cloud, enter https://cloud.streamsets.com.
    • For Control Hub on-premises, enter the URL provided by your system administrator. For example, https://<hostname>:18631.
    Run in Background Runs started pipelines in the background.

    When running pipelines in the background, the stage passes the orchestration record downstream immediately after starting the pipelines.

    By default, the stage runs pipelines in the foreground, passing the record downstream only after all started pipelines complete.
    Status Check Interval Milliseconds to wait between checks for the completion status of the started pipelines. Available when running started pipelines in the foreground.
  3. On the Credentials tab, configure the following properties:
    Credentials Property Description
    User Name User that runs the pipeline. Enter a user name for the execution engine or enter a Control Hub user name if the engine is registered with Control Hub.
    Password Password for the user.
    Tip: To secure sensitive information such as user names and passwords, you can use runtime resources or credential stores. For more information about credential stores, see Credential Stores in the Data Collector documentation.
  4. To use SSL/TLS, click the TLS tab and configure the following properties.
    TLS Property Description
    Use TLS Enables the use of TLS.
    Use Remote Truststore Enables loading the contents of the truststore from a remote credential store or from values entered in the stage properties. For more information, see Remote Keystore and Truststore.
    Trusted Certificates Each PEM certificate used in the remote truststore. Enter a credential function that returns the certificate or enter the contents of the certificate.

    Using simple or bulk edit mode, click the Add icon to add additional certificates.
    Truststore File

    Path to the local truststore file. Enter an absolute path to the file or enter the following expression to define the file stored in the Data Collector resources directory:

    ${runtime:resourcesDirPath()}/truststore.jks

    By default, no truststore is used.

    Truststore Type Type of truststore to use. Use one of the following types:
    • Java Keystore File (JKS)
    • PKCS #12 (p12 file)

    Default is Java Keystore File (JKS).

    Truststore Password Password to the truststore file. A password is optional, but recommended.
    Tip: To secure sensitive information such as passwords, you can use runtime resources or credential stores. For more information about credential stores, see Credential Stores in the Data Collector documentation.
    Truststore Trust Algorithm Algorithm to manage the truststore.

    Default is SunX509.
    Use Default Protocols Uses the default TLSv1.2 transport layer security (TLS) protocol. To use a different protocol, clear this option.
    Transport Protocols TLS protocols to use. To use a protocol other than the default TLSv1.2, click the Add icon and enter the protocol name. You can use simple or bulk edit mode to add protocols.
    Note: Older protocols are not as secure as TLSv1.2.
    Use Default Cipher Suites Uses a default cipher suite for the SSL/TLS handshake. To use a different cipher suite, clear this option.
    Cipher Suites Cipher suites to use. To use a cipher suite that is not a part of the default set, click the Add icon and enter the name of the cipher suite. You can use simple or bulk edit mode to add cipher suites.

    Enter the Java Secure Socket Extension (JSSE) name for the additional cipher suites that you want to use.