Runtime Parameters

Runtime parameters are parameters that you define for a pipeline and call from within that same pipeline. When you define a runtime parameter, you enter the default value to use. When you create or edit a job that includes a pipeline with runtime parameters, you can specify another value to override the default. When the job starts, the value replaces the default value of the runtime parameter.

You configure runtime parameters when you develop a pipeline. Use runtime parameters to represent any stage or pipeline property with a value that must change for each pipeline run - such as batch sizes and timeouts, directories, or URI.

For more information about configuring runtime parameters for a pipeline, see the appropriate engine documentation for the pipeline type:

After you publish the pipeline, you can change the parameter values for each job that runs the pipeline without having to edit the pipeline. If you do not define runtime parameter values for the job, Control Hub uses the default values defined in the pipeline.

When Control Hub runs multiple pipeline instances for a job, the same parameter values are used for each pipeline instance.

Using Functions for Parameter Values

You can use functions included in the StreamSets expression language to define parameter values for a job.

Enter job functions for parameter values within the expression language syntax, as follows:

${<function>}

For example, you need to run a batch job that processes all available data from a database table, uses a Hadoop FS destination to write the data to HDFS, and then stops. You want each job run to write to a directory that includes the job start time. You define a StartTime parameter in the pipeline and then call that parameter from the Directory Template property in the Hadoop FS destination.

When you define the job for the pipeline, you define the value of the StartTime parameter as follows:

StartTime : ${job:startTime()}

For more information about the IBM StreamSets expression language functions, see Functions in the Data Collector documentation.