Runtime Parameters

Runtime parameters are parameters that you define in a pipeline and then call from within the same pipeline. When the pipeline runs, the value replaces the name of the runtime parameter. Use runtime parameters to define values for stage and pipeline properties.

When you configure a job that runs the pipeline, you can override the parameter values defined in the pipeline. For example, you can define a targetTable parameter that points to a production table. Then during development, you can override it to point to a test table.

Or, you might define a set of parameters, one that defines the target table and one that defines the write mode, so you can use the same pipeline to write to different tables in different ways by defining different sets of parameters for each job.

Using Parameters

You can create and then use a new parameter when you define the value for a stage or pipeline property.

Note: Alternatively, you can first create parameters on the pipeline Parameters tab, and then use those existing parameters from the pipeline and stage properties. However, when you create parameters on the pipeline Parameters tab, you must ensure that you type a valid default value for the property where you plan to use the parameter.

Click the Use Parameter icon () next to a stage or pipeline property to use a parameter for that property.

For example, click the Use Parameter icon next to the Additional Where Clause property to use a parameter for that property:
To create and use a new parameter, in the Parameter Selector dialog box, select Create New Parameter.
1. Enter a name for the new parameter, without any spaces.
  
  The default name is a combination of the stage and property name. You can specify a different name.
2. Enter or select a default value for the parameter.
  
  When defining a parameter for a property that displays as a checkbox or drop-down menu, the dialog box displays a list of valid default values.
  
  When defining a parameter for a property that displays as a list, the dialog box displays the required JSON format for the default value.
For example, the following image displays the creation of a new parameter named SnowflakeTable_where with a default value of ID > 1000:
To use an existing parameter, in the Parameter Selector dialog box, select Use Existing Parameter and then select the parameter that you want to use.

The Existing Parameters field lists all pipeline parameters with their default values.

For example, the following image displays the selection of an existing parameter named SnowflakeTable_previewPredicate:
Click Ok.

The parameter name displays as the property value, using the following required syntax:

${<parameter name>}

Modifying Parameter Default Values

When you create a new parameter, you define a default value to use for that parameter. You can modify the default value from the pipeline Parameters tab.

In the pipeline properties, click the Parameters tab.
Locate the parameter, and then modify the default value in the right text box.

Note: You can also modify the parameter name in the left text box. However, if any pipeline or stage properties are currently using the parameter, those properties are not automatically updated to use the new parameter name. You must manually update each property that uses the parameter with the changed name.

Overriding Parameter Values in Jobs

When needed, you can override a runtime parameter in the Control Hub job that runs the pipeline.

For more information about configuring parameters in jobs or job templates, see the Control Hub documentation.

Parameter Syntax

When you use the Parameter Selector to create a new parameter or to select an existing parameter, the parameter name automatically displays as the property value, using the following required syntax:

${<parameter name>}

You can type the parameter name for a property, as long as you use the required syntax.

You can call a runtime parameter from within a StreamSets function by simply entering the parameter name. For example, the following expression trims leading and trailing spaces from the data in the address_column runtime parameter:

 ${str:trim(address_column)}

You can use parameters to represent any stage or pipeline property that allows the use of the StreamSets expression language, including properties that display as checkboxes, drop-down menus, or lists.

Calling from Text Boxes

When calling a parameter from a property that displays as a text box, you can use a parameter to represent a part of the property. For example, say you want to use a set of UDFs in the Applied Functions processor that all have the same EU_ prefix. You can specify the prefix in the UDF Name property and use a UDFname parameter for the rest of the name, as follows:

EU_${UDFname}

Calling from Checkboxes and Drop-Down Menus

When calling a parameter from a property that displays as a checkbox or drop-down menu, the parameter must evaluate to a valid option for the property type:

Checkboxes: Parameters called from properties that display as checkboxes must evaluate to true or false.; When you use the Parameter Selector to create a new parameter for a property that displays as a checkbox, the dialog box displays the possible default values for the parameter, for example:
Drop-down menus: Parameters called from properties that display as drop-down menus must evaluate to a valid key value. Each option in the menu has an associated key value.; For example, to use a parameter for the Write Mode property that displays as a drop-down menu, the parameter must evaluate to one of the valid key values, such as APPEND or OVERWRITE, and not to one of the menu options, such as Append to Table or Overwrite Table.; When you use the Parameter Selector to create a new parameter for a property that displays as a drop-down menu, the dialog box displays each menu option with the corresponding valid key values for the possible default values for the parameter, for example:

Calling from Lists

When calling a parameter from a property that displays as a list, the parameter must evaluate to a valid JSON string for the list type:

List of values

Parameters called from properties that display as a list of values must evaluate to a JSON string in the following format:

[
    "<value1>",
    "<value2>"
]

List of key-value pairs

Parameters called from properties that display as a list of key-value pairs must evaluate to a JSON string in the following format:

[
   {
      "<key1>": "<value1>",
      "<key2>": "<value2>"
   },
   {
      "<key3>": "<value3>",
      "<key4>": "<value4>"
   }
]

To view the valid JSON string for a list, first configure the list with the desired values, and then click the Use Parameter icon () next to the property.

For example, to define a parameter for the Column Type Conversions list, configure the list with the desired values:

Then, click the Use Parameter icon () and select Create New Parameter. The default value for the new parameter uses the configured values in the required JSON format, as follows: