Runtime Values

Runtime values are values that you define outside of the pipeline and use for stage and pipeline properties. You can change the values for each pipeline run without having to edit the pipeline.

You can use runtime values for any pipeline property that allows the use of the expression language. You can, for example, use runtime values to represent batch sizes, timeouts, directories, and URI. You cannot use runtime values to represent fields.

You can use the following methods of passing runtime values to pipelines:

Runtime parameters: Use runtime parameters when you want to define runtime values for a single pipeline.; You define runtime parameters when you configure the pipeline, and then you call the parameters from within that pipeline. When you start the pipeline, you can optionally override the parameters defined in the pipeline.; Runtime parameters are defined for a single pipeline - only that pipeline can call them.
Runtime properties: Use runtime properties when you want to define values for multiple pipeline properties in a single file.; You define runtime properties in a file local to Transformer, and then you call the properties from within a pipeline. At runtime, Transformer loads the property values from the file. A runtime properties file can contain multiple properties.; Runtime properties are defined for Transformer - any pipeline can call them.
Runtime resources: Use runtime resources when you want to store common pipeline configuration properties in files with restricted permissions.; You define runtime resources in a file local to Transformer, and then you call the resources from within a pipeline. You can restrict the permissions for the resource files, but any user who can create a pipeline can access the data stored in the files. Transformer loads the property values from the files each time it processes a new batch. A runtime resource file can contain a single piece of information.; Runtime resources are defined for Transformer - any pipeline can call them.; Tip: Runtime resources can impact pipeline performance because Transformer loads runtime resource files for each batch. Consider using runtime parameters or runtime properties instead. Or, to more securely define sensitive values, consider using credential stores.

Note: When configuring runtime values for pipelines run by StreamSets Control Hub, all Transformers that run the pipeline must have the values locally defined in the expected locations.

Runtime Parameters

Runtime parameters are parameters that you define in a pipeline and then call from within the same pipeline. When the pipeline runs, the value replaces the name of the runtime parameter. Use runtime parameters to define values for stage and pipeline properties.

When you start the pipeline, you can override the parameter values defined in the pipeline. For example, you can define a directory parameter that points to a directory on a production system and override it during testing to point to a directory on a test system.

Note: If you shut down and then restart Transformer without stopping the pipeline, the pipeline continues running with the last set of parameter values.

To implement runtime parameters, perform the following steps:

Define runtime parameters.
Use an expression in the pipeline to call a runtime parameter.

Step 1. Define a Runtime Parameter

Define runtime parameters when you configure the pipeline.

In the pipeline properties, click the Parameters tab.
Using simple or bulk edit mode, click the Add icon and define the name and the default value for each parameter.
For example, you might define a parameter named JDBCConnectionString with the default value of jdbc:mysql://localhost:3306/sample to be used in the JDBC Table origin.

Step 2. Call the Runtime Parameter

Use an expression in the pipeline to call a runtime parameter.

You can use runtime parameters to represent any stage or pipeline property that allows the use of the StreamSets expression language, including properties that display as text boxes, checkboxes, or drop-down menus. You can also call a runtime parameter in the code developed for a scripting processor.

Calling from Text Boxes

To call a runtime parameter in a stage or pipeline property that displays as a text box, use the following syntax:

${<parameter name>}

For example, to use the JDBCConnectionString runtime parameter with the JDBC Table origin, enter the following syntax for the JDBC Connection String property:

${JDBCConnectionString}

You can call a runtime parameter from within a StreamSets function by simply entering the parameter name. For example, the following expression decodes the Base64 encoded keytab defined for the KerberosKeytab runtime parameter:

 ${base64:decodeBytes(KerberosKeytab)}

You can use a runtime parameter to represent a part of a property. For example, you could use a RootDir runtime parameter and append the rest of the directory in the property as follows:

${RootDir}/logfiles

Calling from Checkboxes and Drop-Down Menus

To call a runtime parameter in a stage or pipeline property that displays as a checkbox or drop-down menu, you first must convert the property to a text box.

Click the Use Parameter icon (

) next to the checkbox or drop-down menu to convert the property to a text box, and then call the parameter using the required syntax:

${<parameter name>}

For example, the following image shows the Lookup Behavior property in the Delta Lake Lookup processor. Though it usually displays as a drop-down menu, it was converted to a text box to call the lookupReturn runtime parameter:

The parameter must evaluate to a valid option for the property type:

Checkboxes: Parameters called from properties that display as checkboxes must evaluate to true or false.
Drop-down menus: Parameters called from properties that display as drop-down menus must evaluate to a valid key value. Each option in the menu has an associated key value.; For example, to use a parameter for the Lookup Behavior property, the parameter must evaluate to one of the valid key values, such as RETURN_ONE, and not to the listed menu options, in this case, Return first matching row.; To view the valid keys for a menu, configure the pipeline with the desired option, then export the pipeline and view the exported pipeline JSON file.

Calling from Scripting Processors

You can call a runtime parameter in the code developed for a scripting processor, such as the PySpark or Scala processors. Use the following syntax in any processor script:

${<parameter name>}

Runtime Properties

Runtime properties are properties that you define in a file local to Transformer and call from within a pipeline. Use runtime properties to define values for stage and pipeline properties.

Runtime properties allow you to easily update the property definitions without editing the pipeline. You can also define different sets of definitions for different Transformers.

For example, you might define a set of runtime properties that points to test systems for pipeline development. When you move to production, you simply update the runtime property definitions in the file. This effectively updates every pipeline that uses those runtime properties without having to edit the pipelines.

When defining a runtime property, you can use a static value or an environment variable.

When calling a runtime property, you can use it as part of a larger property definition. For example, you can set a runtime property to the HOME environment variable, which would differ on different machines, and then call the runtime property as a base directory for a longer directory.

To implement runtime properties, perform the following steps:

Define runtime properties.
Use an expression in the pipeline to call a runtime property.

Step 1. Define Runtime Properties

You can define runtime properties in the Transformer configuration file, $TRANSFORMER_CONF/transformer.properties or in a separate runtime properties file:

Transformer configuration file

Use the following steps to define runtime properties in the Transformer configuration file:

In the $TRANSFORMER_CONF/transformer.properties file, configure the runtime.conf.location property as follows:In Control Hub, edit the deployment. In the Configure Engine section, click Advanced Configuration. Then, click Transformer Configuration.
```
runtime.conf.location=embedded
```
To define the runtime properties in the Transformer configuration file, use either of the following formats:
- To define a static value for a runtime property, use the following format:
```
runtime.conf_<property name>=<value>
```
  For example, the following runtime property defines an Amazon S3 bucket:
```
runtime.conf_S3bucket=EUsales
```
- To define an environment variable for a runtime property, use the following format:
```
runtime.conf_<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
runtime.conf_BaseDir=${env("HOME")}
```
Restart Transformer to enable the changes.

Separate runtime properties file

Use the following steps to define runtime properties in a separate runtime properties file:

Create a text file and save it in a directory relative to the $TRANSFORMER_CONF directory.
To define runtime properties in a separate text file, use either of the following formats:
- To define a static value for a runtime property, use the following format:
```
<property name>=<value>
```
  For example, the following runtime property defines an Amazon S3 bucket:
```
S3bucket=EUsales
```
- To define an environment variable for a runtime property, use the following format:
```
<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
BaseDir=${env("HOME")}
```
In the Transformer configuration file, $TRANSFORMER_CONF/transformer.properties, configure the runtime.conf.location property to point to the relative location of the separate runtime properties file. Specify a relative path for a file inside the $TRANSFORMER_CONF directory or an absolute path for a file outside the $TRANSFORMER_CONF directory.
For example, the following separate runtime properties file is located in a runtime directory that is relative to the $TRANSFORMER_CONF directory:
```
runtime.conf.location=runtime/test-runtime.properties
```
Restart Transformer to enable the changes.

Step 2. Call the Runtime Property

Use the runtime:conf function to call a runtime property. You can use runtime properties to represent any stage or pipeline property that allows the use of the expression language.

To call a runtime property, use the following syntax:

${runtime:conf(<property name>)}

For example, to call the ADLS_SAcct runtime property to provide the storage account for an ADLS Gen2 origin, enter the following syntax for the Storage Account stage property:

${runtime:conf('ADLS_SAcct')}

You can call a runtime property to represent a part of a property value. For example, to call the S3bucket runtime property, enter the following syntax for the Amazon S3 Bucket property:

s3a://${runtime:conf('S3bucket')}/

Runtime Resources

Similar to runtime properties, runtime resources are values that you define in a file local to the Transformer and call from within a pipeline. However, with runtime resources, you can restrict the permissions for the files to secure sensitive information. Without restrictive permissions, any user who can create a pipeline can access the data stored in the resource files.

Transformer loads runtime resource files for each pipeline run, then removes them when the pipeline completes. When a pipeline uses a large number of runtime resource files, this can impact pipeline performance.

You can configure Transformer to cache runtime resource files for Databricks pipelines. For other pipelines that use a large number of runtime resource files, consider using runtime parameters or runtime properties instead. Or, to more securely define sensitive values, consider using a credential store.

To implement runtime resources, perform the following steps:

Define each runtime resource.
Use an expression in the pipeline to call a runtime resource.

Step 1. Define Runtime Resources

Use the following steps to define runtime resources:

For each resource, create a text file and save it in the $TRANSFORMER_RESOURCES directory. This directory should be defined outside of $TRANSFORMER_DIST, the Transformer runtime directory.
A file must contain one piece of information to be used when the resource is called.
Optionally, restrict the permissions for the file.
Generally, anyone can read a file. To restrict permissions, configure the file so only the owner has read or write permissions for the file - in octals, that's 600 or 400. And the owner must be the system user that runs Transformer.
When you use the resource in the pipeline, you specify whether the file is restricted.

Step 2. Call Runtime Resources

Use the runtime:loadResource or runtime:loadResourceRaw function to call a runtime resource. You can use runtime resources to represent information in any stage or pipeline property that allows the use of the StreamSets expression language.

Note: In most cases, you should use the runtime:loadResource function which trims any leading or trailing whitespace characters from the file. However, if needed, you can also use the runtime:loadResourceRaw function which includes any leading or trailing whitespace characters in the file.

To call a runtime resource, use the following syntax:

${runtime:loadResource(<file name>, <restricted: true | false>)}

For example, the following expression returns the contents of the JDBC.txt file, trimming any leading or trailing whitespace characters. The file contains a connection string and is restricted so only the owner can read the file:

${runtime:loadResource("JDBC.txt", true)}