Miscellaneous Functions

You can replace any argument with a literal or an expression that evaluates to the argument. String literals must be enclosed in single or double quotation marks.

The StreamSets expression language provides the following miscellaneous functions:

runtime:availableProcessors()

Returns the number of processors available to the Java virtual machine. You can use this function when you want to configure multithreaded processing based on the number of processors available to Transformer.

Return type: Integer.

runtime:conf(<runtime property>)

Returns the value for the specified runtime configuration. Use to call a runtime property.

Uses the following argument:

• runtime property - Name of the configuration property to use. The property must defined in the Transformer configuration fileconfiguration properties or in a separate runtime properties file.

  When the property is defined in the Transformer configuration fileconfiguration properties, simply specify the runtime property name.

  When the property is defined in a runtime configuration file, use the following format: runtime.conf_<property name>.

For example, the following expression returns the value of the ADLS_SAcct runtime property defined in the Transformer configuration fileconfiguration properties:

${runtime:conf('ADLS_SAcct')}

In contrast, the following expression returns the value of the ADLS_SAcct runtime property defined in a runtime properties file:

${runtime:conf('runtime.conf_ADLS_SAcct')}

runtime:loadResource(<file name>, <restricted: true | false>)

Returns the value in the specified file, trimming any leading or trailing whitespace characters from the file. Use to call a runtime resource.

Uses the following arguments:

• file name - Name of the file that contains the information to be loaded. The file must reside in the $TRANSFORMER_RESOURCES directory.
• restricted - Whether the file has restricted permissions. If set to true, the file must be owned by the system user who runs the Transformer and read and writable only by the owner.

For example, the following expression returns the contents of the restricted JDBCpassword.txt file, trimming any leading or trailing whitespace characters:

${runtime:loadResource("JDBCpassword.txt", true)}

runtime:loadResourceRaw(<file name>, <restricted: true | false>)

Returns the entire contents in the specified file, including any leading or trailing whitespace characters in the file. Use to call a runtime resource.

Uses the following arguments:

• file name - Name of the file that contains the information to be loaded. The file must reside in the $TRANSFORMER_RESOURCES directory.
• restricted - Whether the file has restricted permissions. If set to true, the file must be owned by the system user who runs the Transformer and read and writable only by the owner.

For example, the following expression returns the entire contents of the restricted JDBCpassword.txt file, including any leading or trailing whitespace characters:

${runtime:loadResourceRaw("JDBCpassword.txt", true)}

sdc:hostname()

Returns the host name of the Transformer machine.

Return type: String.

sdc:id()

Returns the Transformer ID.

For a pipeline that runs in standalone execution mode, the ID is a unique identifier associated with the Transformer, such as 58efbb7c-faf4-4d8e-a056-f38667e325d0. The ID is stored in the following file: $TRANSFORMER_DATA/transformer.id.

For a pipeline that runs in cluster mode, the ID is the Transformer worker partition ID generated by a cluster application, such as Spark or MapReduce.

size()

Returns the size of a map.

Return type: Integer.

uuid:uuid()

Returns a randomly generated UUID.

This function uses a lot of entropy on Linux systems and can cause your entropy pools to run dry. When this happens, your pipelines slow to a halt but continue to run. Throughput effectively goes to zero while the system waits for entropy to again become available. As a best practice, we recommend running the haveged daemon on any Transformer machine where you use this function. The haveged daemon regenerates your entropy pools.

Return type: String.