Job Templates Overview

A job template is a definition of a job that you can use to create and start multiple job instances. A job template defines the pipeline to run, the StreamSets engine that runs the pipeline, advanced job runtime details, and default values for pipeline parameters. Job templates allow data engineers to define the job details, while analysts can start job instances from the templates by modifying pipeline parameter values only.

Using job templates allows you to hide the complexity of job details from business analysts. For example, let's say that analysts from different departments need to run a pipeline that moves data from their department tables in Oracle to their department tables in Snowflake. All analysts require the same business logic to move the data, but need to specify their unique source and target table names.

A data engineer uses Control Hub to build a single pipeline, using pipeline runtime parameters to define the source and target tables. The data engineer creates a job template for the published pipeline that defines how the Oracle to Snowflake pipeline is run - specifying the engines that run the pipeline and enabling pipeline failover. When an analyst creates and starts a job instance from the template, they only need to specify the values of the source and target table parameters. Each job instance started from the template has an identical configuration, except for the parameters.

When you create a job template for a pipeline that uses runtime parameters, you determine whether each parameter functions as a dynamic parameter that can be overridden in child job instances or as a static parameter that cannot be overridden in child job instances.

When you create instances from a job template, each job instance is attached to the parent job template by default. When you edit the job template, Control Hub makes the same updates to all attached job instances. You can optionally detach job instances from the parent job template when you create the job instances.