Runtime Values
Runtime values are values that you define outside of the pipeline and use for stage and pipeline properties. You can change the values for each pipeline run without having to edit the pipeline.
You can use runtime values for any pipeline property that allows the use of the expression language. You can, for example, use runtime values to represent batch sizes, timeouts, directories, and URI. You cannot use runtime values to represent fields.
You can use the following methods of passing runtime values to pipelines:
- Runtime parameters
- Use runtime parameters when you want to define runtime values for a single pipeline.
- Runtime properties
- Use runtime properties when you want to define values for multiple pipeline properties in a single file.
- Runtime resources
- Use runtime resources when you want to store common pipeline configuration properties in files with restricted permissions.
Using Runtime Parameters
Runtime parameters are parameters that you define in a pipeline and then call from within that same pipeline. When the pipeline runs, the value replaces the name of the runtime parameter. Use runtime parameters to define values for stage and pipeline properties.
When you start the
pipeline, you can override the parameter values defined in the pipeline. For example,
you can define a directory
parameter that points to a directory on a
production system and override it during testing to point to a directory on a test
system.
- Define runtime parameters.
- Use an expression in the pipeline to call a runtime parameter.
Step 1. Define Runtime Parameters
Define runtime parameters when you configure the pipeline.
Step 2. Call the Runtime Parameters
Use an expression in the pipeline to call a runtime parameter.
You can use runtime parameters to represent any stage or pipeline property that allows the use of the StreamSets expression language, including properties that display as text boxes, checkboxes, drop-down menus, or lists. You can also call a runtime parameter in the code developed for a scripting processor.
If you call a runtime parameter from a property that conditionally displays child properties, all child properties display so that you can configure valid values for all dependent properties.
Calling from Text Boxes
${<parameter name>}
JDBCConnectionString
runtime parameter for the
JDBC Multitable Consumer origin, enter the following syntax for the JDBC Connection
String property:${JDBCConnectionString}
JDBCConnectionString
runtime
parameter: ${record:value(JDBCConnectionString)}
RootDir
runtime parameter and append the rest of the directory in
the property as follows:${RootDir}/logfiles
Calling from Checkboxes and Drop-Down Menus
To call a runtime parameter in a stage or pipeline property that displays as a checkbox or drop-down menu, you first must convert the property to a text box.

${<parameter name>}
For example, the following image shows that the Delivery Guarantee property that displays as a drop-down menu has been converted to a text box so that a parameter can be called from the property:
- Checkboxes
- Parameters called from properties that display as checkboxes must evaluate to true or false.
- Drop-down menus
- Parameters called from properties that display as drop-down menus must evaluate to a valid key value. Each option in the menu has an associated key value.
Calling from Lists
To call a runtime parameter in a stage or pipeline property that displays as a list of values or a list of key-value pairs, you first must convert the property to a text box.

${<parameter name>}
For example, the following image shows that the Email IDs property that displays as a list of values has been converted to a text box so that a parameter can be called from the property:
- List of values
- Parameters called from properties that display as a list of values must
evaluate to a JSON string in the following
format:
[ "<value1>", "<value2>" ]
- List of key-value pairs
- Parameters called from properties that display as a list of key-value pairs
must evaluate to a JSON string in the following
format:
[ { "<key1>": "<value1>", "<key2>": "<value2>" }, { "<key3>": "<value3>", "<key4>": "<value4>" } ]
To view the valid JSON string for a list, configure the list with the desired values, and then switch to bulk edit mode.
For example, to define a runtime parameter for the Email IDs list on the pipeline Notifications tab, configure the list with the desired values:
Then, click Bulk Edit Mode to display the valid property value in JSON format, as follows:
Use the bulk edit mode value to define the parameter value on the pipeline Parameters tab, as follows:
Calling from Scripting Processors
You can call a runtime parameter in the code developed for a scripting processor.
The method you use to call the runtime parameter depends on the following scripting processor types:
- JavaScript Evaluator processor
- Use the following syntax in any of the processor scripts:
${<parameter name>}
. For example, the following line of JavaScript code assigns the value of theNewFieldValue
parameter to a map field:records[i].value.V= ${NewFieldValue}
- Jython Evaluator processor
- You can use either of the following methods:
- Groovy Evaluator processor
- Use the
sdc.pipelineParameters()
method in any of the processor scripts to return a map of all runtime parameters defined for the pipeline. For example, the following line of Groovy code assigns the value of theCompanyParam
parameter to the Company Name field:record.value['Company Name'] = sdc.pipelineParameters()['CompanyParam']
Configuring Dependent Properties
Some pipeline and stage properties conditionally display child properties. For example, if you configure an origin to use the Delimited data format, the origin displays a set of Delimited configuration properties. If you configure that origin to use the JSON data format, it displays a different set of JSON configuration properties.
However, if you use a runtime parameter to define a parent property, all child properties display so that you can configure valid values for all dependent properties.
For example, if you convert the Data Format drop-down menu to a text box and then call
the dataformat
parameter using the required syntax, the origin displays
all of the Delimited, JSON, and Text configuration properties:
Using Runtime Properties
Runtime properties are properties that you define in an external location and call from within a pipeline.
With runtime properties, you can define different sets of values for different deployments. Runtime properties allow you to easily update the property definitions without editing the pipeline.
For example, you might define a set of runtime properties that points to test systems for pipeline development. When you move to production, you simply update the runtime property definitions in the file. This effectively updates every pipeline that uses those runtime properties without having to edit the pipelines.
When defining a runtime property, you can use a static value or an environment variable.
When calling the runtime property, you can use it as part of a larger property definition. For example, you can set a runtime property to the HOME environment variable, which would differ on different machines, and then call the runtime property as a base directory for a longer directory.
- Define runtime properties.
- Use an expression in the pipeline to call a runtime property.
Step 1. Define Runtime Properties
You can define runtime properties in the Data Collector configuration properties, or in a separate runtime properties file.
- Data Collector configuration properties
- Use the following steps to define runtime properties in the Data Collector
configuration properties:
- In Control Hub, edit the
deployment. In the Configure Engine section, click
Advanced Configuration. Then, click Data
Collector Configuration. Configure the runtime.conf.location
property as
follows:
runtime.conf.location=embedded
- To define the runtime properties in the Data Collector
configuration properties, use either of the following formats:
- To define a static value for a runtime property, use the following format:
For example, the following runtime property defines a Hadoop FS directory template:runtime.conf_<property name>=<value>
runtime.conf_HDFSDirTemplate=/HDFS/DirectoryTemplate
- To define an environment variable for a runtime property, use the following
format:
runtime.conf_<property name>=${env("<environment_variable>")}
For example, the following runtime property defines a base directory, setting it to the HOME environment variable:runtime.conf_BaseDir=${env("HOME")}
- To define a static value for a runtime property, use the following format:
- Save the changes to the deployment and restart all engine instances.
- In Control Hub, edit the
deployment. In the Configure Engine section, click
Advanced Configuration. Then, click Data
Collector Configuration. Configure the runtime.conf.location
property as
follows:
- Separate runtime properties file
- Use the following steps to define runtime properties in a separate runtime properties
file:
- Create a text file and then use
either of the following formats to define the runtime properties:
-
To define a static value for a runtime property, use the following format:
For example, the following runtime property defines a Hadoop FS directory template:<property name>=<value>
HDFSDirTemplate=/HDFS/DirectoryTemplate
- To define an environment variable for a runtime property, use the
following
format:
<property name>=${env("<environment_variable>")}
For example, the following runtime property defines a base directory, setting it to the HOME environment variable:BaseDir=${env("HOME")}
-
- In Control Hub, upload the file as an external resource for the engine.
- In Control Hub, edit the deployment. In the Configure Engine section, click Advanced Configuration. Then, click Data Collector Configuration. Configure the runtime.conf.location property to point to the name of the separate runtime properties file.
- Save the changes to the deployment and restart all engine instances.
- Create a text file and then use
either of the following formats to define the runtime properties:
For more information, see Configuring Data Collector.
Step 2. Call the Runtime Property
Use the runtime:conf
function to call a runtime property. You can
use runtime properties to represent any stage or pipeline property that allows the use of
the expression language.
${runtime:conf(<property name>)}
HDFSDirTemplate
runtime property to provide the
directory template value for the Hadoop FS destination, enter the following syntax for
the Directory Template
property:${runtime:conf('HDFSDirTemplate')}
RootDir
runtime property and append the rest of the
directory, as follows:${runtime:conf('RootDir')}/logfiles
Using Runtime Resources
Similar to runtime properties, runtime resources are values that you define in an external file and call from within a pipeline. But with runtime resources, you can restrict the permissions for the files to secure information.
Use runtime resources to store common configuration properties for multiple pipelines, like URLs for external systems. Note that any user who can create a pipeline can access the data stored in the resource files.
- Define each runtime resource.
- Use an expression in the pipeline to call a runtime resource.
Step 1. Define Runtime Resources
- For each resource, create a text file.
A file must contain one piece of information to be used when the resource is called.
- Optionally, restrict the permissions for the file.
Generally, anyone can read a file. To restrict permissions, configure the file so only the owner has read or write permissions for the file - in octals, that's 600 or 400. And the owner must be the system user that runs the Data Collector.
When you use the resource in the pipeline, you specify whether the file is restricted.
- In Control Hub, upload the file as an external resource for the engine.
Step 2. Call the Runtime Resource
Use the runtime:loadResource
or
runtime:loadResourceRaw
function to call a runtime resource. You can
use runtime resources to represent information in any stage or pipeline property that allows
the use of the expression language.
runtime:loadResource
function which
trims any leading or trailing whitespace characters from the file. However, if
needed, you can also use the runtime:loadResourceRaw
function which
includes any leading or trailing whitespace characters in the file.To call a runtime resource, use the following syntax:
${runtime:loadResource(<file name>, <restricted: true | false>)}
JDBC.txt
file, trimming any leading or trailing whitespace
characters. The file contains a connection string and is restricted so only the owner
can read the file:${runtime:loadResource("JDBC.txt", true)}