Job Templates

When you create a job for a pipeline that uses runtime parameters, you can enable the job to work as a job template. A job template lets you run multiple job instances with different runtime parameter values from a single job definition.

Like a regular job, a job instance can run multiple pipeline instances, based on the number of pipeline instances configured for the job template. When a job instance runs multiple pipeline instances, each of those pipeline instances uses the same parameter values.

For example, you design a pipeline with a Google Cloud Storage origin that uses a runtime parameter for the Bucket property for the origin. You create a job template for the pipeline and configure the job template to run one pipeline instance. You create and start five job instances from the template, defining a unique value for the bucket parameter for each job instance. Control Hub starts five active jobs from the single job template, running one pipeline instance for each job. Each job and pipeline instance has an identical configuration, except for the bucket parameter value.

Control Hub lists job templates in the Jobs view, identifying each template with a Job Template label in the Status column. A job template is simply a job definition and does not have an active or inactive status. When you start a job template, you create and start one or more job instances from the template. You specify a suffix to uniquely name each job instance, the number of job instances to create from the template, and the parameter values to use for each job instance.

For example, the following image displays a job template, identified with the Job Template label in the Status column, and two job instances started from that template. The job instance names use a counter as the suffix:

You manage job instances started from a template just as you manage any other job. For example, you can individually start, stop, and monitor job instances.

However, you cannot edit inactive job instances started from a template. Instead, you must edit the job template to edit the job definition. Active job instances reflect edits made to the job template after you stop and restart the job instances.

Job Template Creation

To create a job template, you simply create a job and select the Enable Job Template property in the Add Job window.

You can enable a job to work as a job template only during job creation. You cannot enable an existing job to work as a job template.

Job Instance Creation

To create and start one or more job instances from a job template, you start the job template. You specify a suffix to uniquely name each job instance, the number of job instances to create from the template, and the parameter values to use for each job instance.

You can start a job template multiple times. For example, you might start a job template to create and start three job instances. After those job instances are active for several days, you realize that you need to run more instances from the same job template. You start the job template again to create additional instances.

Suffix for Job Instance Names

For job instances created or started from a job template, Control Hub appends a suffix to uniquely name each job instance.

The suffix is added to the job template name after a hyphen, as follows:

<job template name> - <suffix>

Select one of the following methods to generate the suffix:

Counter

Control Hub appends a number to the job template name. For example, job instances created from the Web Log Collection Job are named as follows:

Web Log Collection Job - 1
Web Log Collection Job - 2

Timestamp

Control Hub appends a timestamp indicating when the job instance is started to the job template name. For example, job instances created from the Web Log Collection Job are named as follows:

Web Log Collection Job - 2021-10-22
Web Log Collection Job - 2021-10-23

Use a timestamp for the suffix when you plan to create and start job instances from the template at different times. For example, if you plan to start a single job instance every 24 hours. If you start multiple job instances from the template at the same time, Control Hub appends the same timestamp to each job instance name.

Parameter Value

Control Hub appends the value of the specified parameter to the job template name. For example, job instances created from the Web Log Collection Job are named as follows:

Web Log Collection Job - /server1/logs
Web Log Collection Job - /server2/logs

Parameter Values for Job Instances

When you create job instances from a job template, you define unique parameter values for each job instance.

You can use functions included in the StreamSets expression language to define parameter values for each job instance.

Use one of the following methods to define the parameter values when you create job instances:

Bulk edit: Enter parameter values in JSON format in the Create and Start Job Instances window. When you add another job instance, Control Hub adds another group containing all the runtime parameters defined for the pipeline with the default values. Modify the parameter values, but be sure not to modify the parameter names.; For example, the following bulk edit configuration creates two job instances using different parameter values:
From file: Define the parameter values in JSON format in a text file and then select the file to upload to Control Hub in the Create and Start Job Instances window. Each group of parameter names and values creates another job instance. Configuring parameter values in a file can be particularly useful when you have a large number of parameter values to enter.