Managing Job Templates

Create a job template for a published pipeline to hide the complexity of job details from business analysts. Analysts can then start job instances from the template by modifying pipeline parameter values only.

When you edit a job template, Control Hub makes the same updates to all attached job instances. If any attached job instances are currently running, Control Hub restarts those instances to apply the updates to them.

You can archive a job template when you do not want new job instances to be created from the template, but want existing job instances to continue to run.

You can delete a job template that has no attached job instances.

Creating a Job Template

To create a job template, in the Navigation panel, click Run > Job Templates, and then click the Create a Job Template icon: .

Then, complete the following steps in the job template wizard:
  1. Define the Job Template
  2. Select the Pipeline
  3. Configure the Job Details
  4. Set Parameter Defaults
  5. Review the Job Template

Define the Job Template

Define the job template essentials, including the template name and optionally a description and tags to identify similar jobs.

Important: Before you can define a job template, you must publish a pipeline built in the pipeline canvas. For instructions, see Publishing a Pipeline.
  1. Enter the following information to define the job template:
    Property Description
    Name Name of the job template.

    Use a brief name that informs your team of the job template use case.
    Description Optional description.

    Use the description to add additional details about the job template use case.
    Job Tags Tags that identify similar job templates. Job instances created from the job template inherit all tags added to the template. Use tags to easily search and filter jobs and job templates.

    Enter nested tags using the following format:

    <tag1>/<tag2>/<tag3>
  2. Click one of the following buttons:
    • Cancel - Cancels creating the job template and exits the wizard.
    • Next - Saves the job template definition and continues.

Select the Pipeline

Select the published pipeline that you want the job template to run.

Tip: To take full advantage of the flexibility that job templates offer, use job templates for pipelines that use parameters.
  1. To select a published pipeline, click Click here to select next to the Pipeline property.

    In the Select a Pipeline window, select the pipeline to use, and then click Save to return to the job template wizard.

    By default, the latest published version is selected.

  2. To select an earlier published version, click Click here to select next to the Pipeline Version property.

    In the Select a Pipeline Version window, select an earlier version, and then click Save to return to the job template wizard.

  3. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Next - Saves the job template definition and continues.

Configure the Job Details

Configure the job details to determine how engines run the pipeline. Users cannot override or change these details when they create job instances for this job template.

Note: When your organization uses the Transformer for Snowflake engine hosted by IBM StreamSets, IBM StreamSets automatically handles how the engine runs pipelines. As such, the job template wizard skips this step for Transformer for Snowflake jobs that run on the hosted engine.
  1. Configure the following job details:
    Property Description
    Deployment Deployment associated with the group of engines that run the pipeline.
    Engine Labels Label or labels that determine the group of engines that run the pipeline. Labels are case sensitive.

    By default, the labels assigned to the deployment are automatically selected as the engine labels. In most cases, you can use the default labels so that the pipeline runs on a deployment of identical engine instances.

    Enable Failover

    Enables Control Hub to restart a pipeline on another available engine when the original engine shuts down unexpectedly.

    Default is enabled.

    Control Hub manages pipeline failover differently based on the engine type, as described in the following topics:
  2. Optionally, click Show Advanced Options and configure the following advanced properties.

    The defaults should work in most cases.

    Advanced Job Property Description
    Number of Instances Number of pipeline instances to run for the job. Increase the value only when the pipeline is designed for scaling out.

    Default is 1, which runs one pipeline instance on an available Data Collector running the fewest number of pipelines. An available Data Collector is an engine assigned all labels specified for the job.

    Available for Data Collector jobs only.
    Failover Retries per Data Collector Maximum number of pipeline failover retries to attempt on each available Data Collector.

    When a Data Collector reaches the maximum number of failover retries, Control Hub does not attempt to restart additional failed pipelines for the job on that Data Collector.

    Use -1 to retry indefinitely.

    Available for Data Collector jobs when failover is enabled.
    Global Failover Retries Maximum number of pipeline failover retries to attempt across all available engines. When the maximum number of global failover retries is reached, Control Hub stops the job.

    Use -1 to retry indefinitely.

    Control Hub manages failover retries differently based on the engine type, as described in the following topics:

    Available when failover is enabled.
    Statistics Refresh Interval (ms) Milliseconds to wait before automatically refreshing statistics when you monitor the job.

    The minimum and default value is 60,000 milliseconds.
    Pipeline Force Stop Timeout (ms) Number of milliseconds to wait before forcing remote pipeline instances to stop.

    In some situations when you stop a job, a remote pipeline instance can remain in a Stopping state. For example, if a scripting processor in the pipeline includes code with a timed wait or an infinite loop, the pipeline remains in a Stopping state until it is force stopped.

    Default is 120,000 milliseconds, or 2 minutes.
    Require Job Error Acknowledgement Requires that users acknowledge an inactive error status due to connectivity issues before the job can be restarted.
    Clear the property for a scheduled job so the job can automatically be restarted without requiring user intervention.
    Important: Clear the property with caution, as doing so might hide errors that the job has encountered.
  3. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the job details and continues.
    • Save & Exit - Saves the job details and exits the wizard, displaying the new template in the Job Templates view. If the selected pipeline uses runtime parameters, the job template uses the default parameter values defined in the pipeline.

Set Parameter Defaults

If the selected pipeline uses runtime parameters, define the default parameter values to use for job instances created from the template.

Note: If the selected pipeline does not use runtime parameters, this step does not display in the wizard.
  1. Override the default value of each parameter as needed.

    The wizard displays the parameter names and default values as defined in the pipeline.

    If the pipeline uses a parameter for a connection property, the default value displays the connection ID. To view the name of the default connection or to select another connection of the same type, click the Select Connection icon: .

    You can override the default values using simple or bulk edit mode. In bulk edit mode, configure parameter values in JSON format. You can also use functions to define parameter values.

  2. To define a parameter as a static parameter that cannot be overridden in job instances created from the template, select Static Parameter.

    By default, parameters are dynamic and can be overridden in job instances. For more information, see Dynamic and Static Parameters.

  3. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the parameter defaults and continues.
    • Save & Exit - Saves the parameter defaults and exits the wizard, displaying the new template in the Job Templates view.

Review the Job Template

You've successfully finished creating the job template. Click Exit to display the new template in the Job Templates view.

For details about creating and starting job instances from the template, see Creating a Job from a Job Template.

Filtering Job Templates

In the Job Templates view, you can filter the list of displayed job templates by engine type or label, by job tag, or by archive status. Or you can search for job templates by name.
Note: By default, filtering is replaced by the new search functionality. If needed, you can disable search and revert to the original filtering functionality.
  1. In the Navigation panel, click Run > Job Templates.
  2. If the Filter column does not display, click the Toggle Filter Column icon: .

    The following image displays the Filter column in the Job Templates view:

  3. To search for job templates by name, enter text in the search field, and then press Return.
  4. Select an engine type or label, job tag, or archive status to additionally filter the list of job templates.
  5. Select the Keep Filter Persistent checkbox to retain the filter when you return to the view.
    Tip: To share the applied filter, copy the URL and send it to another user in your organization.

Editing Job Templates

You can edit job templates to change the job definition. When you edit a job template, Control Hub makes the same updates to all attached job instances. If any attached job instances are currently running, Control Hub restarts those instances to apply the updates to them.

Edit job templates from the Job Templates view. Hover over the job template, and click the Edit icon: .

You can edit job templates to change the following information:
  • Name
  • Description
  • Pipeline version - You can select a different pipeline version to run.

    For example, after you start a job instance from the template, you realize that the developer forgot to enable a metric rule for the pipeline, so you stop the job instance. You inform your developer, who edits the pipeline rules in the pipeline canvas and republishes the pipeline as another version. You edit the job template to select that latest published version of the pipeline, and then start the job instance again.

    Important: If you edit the job template so that it contains a new pipeline version with a different origin, you must reset the origin before restarting job instances.
  • Engine Labels - You can assign and remove labels from the job template to change the group of engines that run the pipeline.
  • Job Template Tags - You can assign and remove tags from the job template to identify the template in a different way.
  • Statistics Refresh Interval - You can change the milliseconds to wait before Control Hub refreshes the statistics when you monitor job instances created from the template.
  • Number of Instances - You can change the number of pipeline instances run for Data Collector jobs.
  • Pipeline Force Stop Timeout - You can change the number of milliseconds to wait before Control Hub forces remote pipeline instances to stop.
  • Enable or disable failover - You can enable or disable pipeline failover.
    Control Hub manages pipeline failover differently based on the engine type, as described in the following topics:
  • Parameter defaults - You can edit default parameter values or change whether a parameter functions as a static or dynamic parameter.

Upgrading to the Latest Pipeline Version

You can upgrade a job template to use the latest published pipeline version.

When a job template includes a pipeline that has a later published version, Control Hub notifies you by displaying the New Pipeline Version icon () next to the template.

You can simply click the icon to upgrade the job template to use the latest pipeline version. Or, you can select job templates in the Job Templates view, click the More icon () and then click Use Latest Pipeline Version.

When you upgrade to the latest pipeline version, Control Hub makes the same updates to all attached job instances. If any attached job instances are currently running, Control Hub restarts those instances to apply the updates to them.

Duplicating Job Templates

You duplicate job templates from the Job Templates view in Control Hub.

  1. In the Navigation panel, click Run > Job Templates.
  2. Select a job template in the list and then click the Duplicate icon: .
  3. Enter a name for the duplicate and the number of copies to make.

    When you create multiple copies, Control Hub appends an integer to the job template name. For example, if you enter the name MyJob and create two copies of a job template, Control Hub names the duplicate templates MyJob1 and MyJob2.

  4. Click Duplicate.
    Control Hub adds the duplicated job templates to the list of jobs template. You can edit them as necessary.

Archiving Job Templates

Archive a job template when you do not want new job instances to be created from the template, but want existing job instances to continue to run.

When you archive a job template, you cannot create new job instances from the template or edit the template. You can continue to run and restart existing job instances originally created from the archived template.

  1. In the Navigation panel, click Run > Job Templates.
  2. Click the name of a job template to view the template details.
  3. Click Archive.
  4. Click Ok to confirm.

Deleting Job Templates

You can delete job templates when all attached job instances created from that template are inactive.

When you delete a job template, Control Hub also deletes all attached job instances.

  1. In the Navigation panel, click Run > Job Templates.
  2. Select templates in the list, and then click the Delete icon: .
  3. Click Ok to confirm.