Runtime Values

Runtime values are values that you define outside of the pipeline and use for stage and pipeline properties. You can change the values for each pipeline run without having to edit the pipeline.

You can use runtime values for any pipeline property that allows the use of the expression language. You can, for example, use runtime values to represent batch sizes, timeouts, directories, and URI. You cannot use runtime values to represent fields.

You can use the following methods of passing runtime values to pipelines:

Runtime parameters: Use runtime parameters when you want to define runtime values for a single pipeline.; You define runtime parameters when you configure the pipeline, and then you call the parameters from within that pipeline. When you start the pipeline, you can optionally override the parameters defined in the pipeline.; Runtime parameters are defined for a single pipeline - only that pipeline can call them.
Runtime properties: Use runtime properties when you want to define values for multiple pipeline properties in a single file.; You define runtime properties in an external file, and then you call the properties from within a pipeline. At runtime, Data Collector loads the property values from the file. A runtime properties file can contain multiple properties.; Runtime properties are defined for the entire Data Collector - any pipeline can call them.
Runtime resources: Use runtime resources when you want to store common pipeline configuration properties in files with restricted permissions.; You define runtime resources in an external file, and then you call the resources from within a pipeline. You can restrict the permissions for the resource files, but any user who can create a pipeline can access the data stored in the files. Data Collector loads the property values from the files each time it processes a new batch. A runtime resource file can contain a single piece of information.; Runtime resources are defined for the entire Data Collector - any pipeline can call them.; Tip: Runtime resources can impact pipeline performance because Data Collector loads runtime resource files for each batch. Consider using runtime parameters or runtime properties instead. Or, to more securely define sensitive values, consider using credential stores.

Runtime Parameters

Runtime parameters are parameters that you define in a pipeline and then call from within that same pipeline. When the pipeline runs, the value replaces the name of the runtime parameter. Use runtime parameters to define values for stage and pipeline properties.

When you create a new parameter, you define a default value for that parameter.

When you start the pipeline, you can override the default parameter values. For example, you can define a directory parameter that points to a directory on a production system and override it during testing to point to a directory on a test system.

Note: If you shut down and then restart Data Collector without stopping the pipeline, the pipeline continues running with the last set of parameter values.

Using Parameters

You can create and then use a new parameter when you define the value for a stage or pipeline property.

Note: Alternatively, you can first create parameters on the pipeline Parameters tab, and then use those existing parameters from the pipeline and stage properties. However, when you create parameters on the pipeline Parameters tab, you must ensure that you type a valid default value for the property where you plan to use the parameter.

Click the Use Parameter icon () next to a stage or pipeline property to use a parameter for that property.

For example, click the Use Parameter icon next to the JDBC Connection String property to use a parameter for that property:
To create and use a new parameter, in the Parameter Selector dialog box, select Create New Parameter.
1. Enter a name for the new parameter, without any spaces.
  
  Control Hub uses a combination of the stage and property name to suggest a default name for the parameter. You can specify a different name.
2. Enter or select a default value for the parameter.
  
  When defining a parameter for a property that displays as a checkbox or drop-down menu, the dialog box displays a list of valid default values.
  
  When defining a parameter for a property that displays as a list, the dialog box displays the required JSON format for the default value.
For example, the following image displays the creation of a new parameter named JDBCQuery_connectionString with a default value of jdbc:mysql://server:3306/sales:
To use an existing parameter, in the Parameter Selector dialog box, select Use Existing Parameter and then select the parameter that you want to use.

The Existing Parameters field lists all pipeline parameters with their default values.

For example, the following image displays the selection of an existing parameter named AmazonS3_bucket with a default value of western_region:
Click Ok.

Control Hub displays the parameter name as the property value, using the following required syntax:

${<parameter name>}

Modifying Parameter Default Values

When you create a new parameter, you define a default value to use for that parameter. You can modify the default value from the pipeline Parameters tab.

In the pipeline properties, click the Parameters tab.
Locate the parameter, and then modify the default value in the right text box.

Note: You can also modify the parameter name in the left text box. However, if any pipeline or stage properties are currently using the parameter, those properties are not automatically updated to use the new parameter name. You must manually update each property that uses the parameter with the changed name.

Parameter Syntax

When you use the Parameter Selector to create a new parameter or to select an existing parameter, Control Hub automatically displays the parameter name as the property value, using the following required syntax:

${<parameter name>}

You can type the parameter name for a property, as long as you use the required syntax.

You can also call a parameter from within an expression language function by simply entering the parameter name. For example, the following expression returns the value of the JDBCConnectionString runtime parameter:

${record:value(JDBCConnectionString)}

You can use parameters to represent any stage or pipeline property that allows the use of the StreamSets expression language, including properties that display as checkboxes, drop-down menus, or lists. You can also use a parameter in a connection property. In addition, you can call a parameter in the code developed for a scripting processor.

If you call a parameter from a property that conditionally displays child properties, all child properties display so that you can configure valid values for all dependent properties.

Calling from Text Boxes

When calling a parameter from a property that displays as a text box, you can use the parameter to represent a part of the property. For example, you could use a RootDir parameter and append the rest of the directory in the property as follows:

${RootDir}/logfiles

Calling from Checkboxes and Drop-Down Menus

When calling a parameter from a property that displays as a checkbox or drop-down menu, the parameter must evaluate to a valid option for the property type:

Checkboxes: Parameters called from properties that display as checkboxes must evaluate to true or false.; When you use the Parameter Selector to create a new parameter for a property that displays as a checkbox, the dialog box displays the possible default values for the parameter, for example:
Drop-down menus: Parameters called from properties that display as drop-down menus must evaluate to a valid key value. Each option in the menu has an associated key value.; For example, to use a parameter for the Delivery Guarantee property that displays as a drop-down menu, the parameter must evaluate to one of the valid key values, AT_LEAST_ONCE or AT_MOST_ONCE, and not to one of the menu options, At Least Once or At Most Once.; When you use the Parameter Selector to create a new parameter for a property that displays as a drop-down menu, the dialog box displays each menu option with the corresponding valid key values for the possible default values for the parameter, for example:

Calling from Lists

When calling a parameter from a property that displays as a list, the parameter must evaluate to a valid JSON string for the list type:

List of values

Parameters called from properties that display as a list of values must evaluate to a JSON string in the following format:

[
    "<value1>",
    "<value2>"
]

List of key-value pairs

Parameters called from properties that display as a list of key-value pairs must evaluate to a JSON string in the following format:

[
   {
      "<key1>": "<value1>",
      "<key2>": "<value2>"
   },
   {
      "<key3>": "<value3>",
      "<key4>": "<value4>"
   }
]

To view the valid JSON string for a list, first configure the list with the desired values, and then click the Use Parameter icon () next to the property.

For example, to define a parameter for the Email IDs list on the pipeline Notifications tab, configure the list with the desired values:

Then, click the Use Parameter icon () and select Create New Parameter. The default value for the new parameter uses the configured values in the required JSON format, as follows:

Calling from Connection Properties

A connection property displays a drop-down menu of existing connection names valid for that stage type. For example, when you configure a connection for an Amazon S3 origin, you can select from existing Amazon S3 connections.

When calling a parameter from a connection property, the parameter must evaluate to the connection ID. When you use the Parameter Selector to create a new parameter for a connection property, the dialog box displays the name and ID of valid connections as the possible default values, for example:

When you view the default parameter values from the pipeline Parameters tab, the default value displays the connection ID only. You can modify the default value by copying and pasting another connection ID. To locate a connection ID, click Set Up > Connections in the Navigation panel, and then click the name of the connection. The connection details displays the connection ID.

Calling from Scripting Processors

You can call a runtime parameter in the code developed for a scripting processor.

The method you use to call the runtime parameter depends on the following scripting processor types:

JavaScript Evaluator processor

Use the following syntax in any of the processor scripts: ${<parameter name>}. For example, the following line of JavaScript code assigns the value of the NewFieldValue parameter to a map field:

records[i].value.V= ${NewFieldValue}

Jython Evaluator processor

You can use either of the following methods:

Use the following syntax in any of the processor scripts: '${<parameter name>}'. For example, the following line of Jython code assigns the value of the NewFieldValue parameter to a map field:
```
records[i].value = '${NewFieldValue}'
```
Use the sdc.pipelineParameters() method in any of the processor scripts to return a map of all runtime parameters defined for the pipeline. For example, the following line of Jython code assigns the value of the NewFieldValue parameter to a map field:
```
records[i].value = sdc.pipelineParameters()['NewFieldValue']
```

Groovy Evaluator processor

Use the sdc.pipelineParameters() method in any of the processor scripts to return a map of all runtime parameters defined for the pipeline. For example, the following line of Groovy code assigns the value of the CompanyParam parameter to the Company Name field:

record.value['Company Name'] = sdc.pipelineParameters()['CompanyParam']

Configuring Dependent Properties

Some pipeline and stage properties conditionally display child properties. For example, if you configure an origin to use the Delimited data format, the origin displays a set of Delimited configuration properties. If you configure that origin to use the JSON data format, it displays a different set of JSON configuration properties.

However, if you use a runtime parameter to define a parent property, all child properties display so that you can configure valid values for all dependent properties.

For example, if you convert the Data Format drop-down menu to a text box and then call the dataformat parameter using the required syntax, the origin displays all of the Delimited, JSON, and Text configuration properties:

Runtime Resources

Similar to runtime properties, runtime resources are values that you define in an external file and call from within a pipeline. But with runtime resources, you can restrict the permissions for the files to secure information.

Use runtime resources to store common configuration properties for multiple pipelines, like URLs for external systems. Note that any user who can create a pipeline can access the data stored in the resource files.

Tip: Runtime resources can impact pipeline performance because Data Collector loads runtime resource files for each batch. Consider using runtime parameters or runtime properties instead. Or, to more securely define sensitive values, use credential stores.

To implement runtime resources, perform the following steps:

Define each runtime resource.
Use an expression in the pipeline to call a runtime resource.

Step 1. Define Runtime Resources

Use the following steps to define runtime resources:

For each resource, create a text file.
A file must contain one piece of information to be used when the resource is called.
Optionally, restrict the permissions for the file.
Generally, anyone can read a file. To restrict permissions, configure the file so only the owner has read or write permissions for the file - in octals, that's 600 or 400. And the owner must be the system user that runs the Data Collector.
When you use the resource in the pipeline, you specify whether the file is restricted.
In Control Hub, add the file as an external resource for the deployment, as described in the Control Hub documentation.

Step 2. Call the Runtime Resource

Use the runtime:loadResource or runtime:loadResourceRaw function to call a runtime resource. You can use runtime resources to represent information in any stage or pipeline property that allows the use of the expression language.

Note: In most cases, you'll use the runtime:loadResource function which trims any leading or trailing whitespace characters from the file. However, if needed, you can also use the runtime:loadResourceRaw function which includes any leading or trailing whitespace characters in the file.

To call a runtime resource, use the following syntax:

${runtime:loadResource(<file name>, <restricted: true | false>)}

For example, the following expression returns the contents of the JDBC.txt file, trimming any leading or trailing whitespace characters. The file contains a connection string and is restricted so only the owner can read the file:

${runtime:loadResource("JDBC.txt", true)}

Runtime Properties

Runtime properties are properties that you define in an external location and call from within a pipeline.

With runtime properties, you can define different sets of values for different deployments. Runtime properties allow you to easily update the property definitions without editing the pipeline.

For example, you might define a set of runtime properties that points to test systems for pipeline development. When you move to production, you simply update the runtime property definitions in the file. This effectively updates every pipeline that uses those runtime properties without having to edit the pipelines.

When defining a runtime property, you can use a static value or an environment variable.

When calling the runtime property, you can use it as part of a larger property definition. For example, you can set a runtime property to the HOME environment variable, which would differ on different machines, and then call the runtime property as a base directory for a longer directory.

To implement runtime properties, perform the following steps:

Define runtime properties.
Use an expression in the pipeline to call a runtime property.

Step 1. Define Runtime Properties

You can define runtime properties in the Data Collector configuration properties, or in a separate runtime properties file.

Data Collector configuration properties

Use the following steps to define runtime properties in the Data Collector configuration properties:

In Control Hub, edit the deployment. In the Configure Engine section, click Advanced Configuration. Then, click Data Collector Configuration. Configure the runtime.conf.location property as follows:
```
runtime.conf.location=embedded
```
To define the runtime properties in the Data Collector configuration properties, use either of the following formats:
- To define a static value for a runtime property, use the following format:
```
runtime.conf_<property name>=<value>
```
  For example, the following runtime property defines a Hadoop FS directory template:
```
runtime.conf_HDFSDirTemplate=/HDFS/DirectoryTemplate
```
- To define an environment variable for a runtime property, use the following format:
```
runtime.conf_<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
runtime.conf_BaseDir=${env("HOME")}
```
Save the changes to the deployment and restart all engine instances.

Separate runtime properties file

Use the following steps to define runtime properties in a separate runtime properties file:

Create a text file and then use either of the following formats to define the runtime properties:
- To define a static value for a runtime property, use the following format:
```
<property name>=<value>
```
  For example, the following runtime property defines a Hadoop FS directory template:
```
HDFSDirTemplate=/HDFS/DirectoryTemplate
```
- To define an environment variable for a runtime property, use the following format:
```
<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
BaseDir=${env("HOME")}
```
In Control Hub, add the file as an external resource for the deployment, as described in the Control Hub documentation.
In Control Hub, edit the deployment. In the Configure Engine section, click Advanced Configuration. Then, click Data Collector Configuration. Configure the runtime.conf.location property to point to the location of the separate runtime properties file. Specify an absolute path to the file in the externalResources directory.
For example:
runtime.conf.location=/sdc/streamsets-datacollector-5.8.0/externalResources/resources/test-runtime.properties
Save the changes to the deployment and restart all engine instances.

For more information, see Configuring Data Collector.

Step 2. Call the Runtime Property

Use the runtime:conf function to call a runtime property. You can use runtime properties to represent any stage or pipeline property that allows the use of the expression language.

To call a runtime property, use the following syntax:

${runtime:conf(<property name>)}

For example, to call the HDFSDirTemplate runtime property to provide the directory template value for the Hadoop FS destination, enter the following syntax for the Directory Template property:

${runtime:conf('HDFSDirTemplate')}

You can call a runtime property to represent a part of a property value. For example, you can call the RootDir runtime property and append the rest of the directory, as follows:

${runtime:conf('RootDir')}/logfiles