StreamSets Expression Language

The StreamSets expression language is based on the JSP 2.0 expression language. In Transformer, you can use the StreamSets expression language in pipeline or stage properties that are evaluated only once, before pipeline processing begins.

For example, you might use the StreamSets expression language to perform the following tasks:
  • Retrieve secrets from a credential store
  • Decrypt a Base64-encoded Kerberos keytab
  • Use the pipeline title to name the Spark application
  • Call a runtime parameter defined in the pipeline

To configure properties that manipulate data, or perform other tasks during pipeline processing, use Spark SQL. For more on this, see Expressions in Pipeline and Stage Properties.

Use the StreamSets expression language to define properties that represent numeric or string values. You can use expression completion to determine where you can use a StreamSets expression and the expression elements that you can use in that location.

You can use the following elements in a StreamSets expression:

Basic Syntax

Precede all StreamSets expressions with a dollar sign and enclose them with curly brackets, as follows: 
${<expression>}

For example, to add 2 + 2, use the following syntax: ${2 + 2}.

Expression Completion

When you configure a property, you can use Ctrl + Space Bar to invoke the StreamSets expression completion helper.

Expression completion provides a list of functions and other elements of the StreamSets expression language that are valid for use at that location. The list includes any runtime parameters that are defined for the pipeline. When an element does not display in the list, it is not a valid element at the specified location.
Tip: You can use expression completion to determine if specifying a StreamSets expression is valid for a property. Simply click in the property text box, and press Ctrl + Space Bar. When a list of elements appears, you know that creating a StreamSets expression is a valid option. If no list appears, configuring a StreamSets expression is not allowed.

Usage and Tips

Use the following information and tips to work with expression completion:

  1. To invoke expression completion, place the cursor where you want to create an expression and click Ctrl + Space Bar.

    A list of valid expression elements displays. Scroll to view the entire list.

    You can use expression completion anywhere where you can use the StreamSets expression language.

  2. Elements are listed in case-sensitive alphabetical order using the following icons:
    • Green M for functions.
    • Red C for constants
    • Purple P for runtime parameters defined in the pipeline.

    In the following example, eastDir is a runtime parameter:

  3. To view more information about an element, use the down arrow key to navigate to and highlight the element name:

  4. To add an element to an expression, click the element name, or highlight the element and press the Enter/Return key.
  5. You can filter the element list by typing the first few letters of the element name.
  6. To view the syntax of a function, after you add the function, click within the parentheses.

Expression Examples

The following table includes some examples of how you might use StreamSets expressions in Transformer:
Description Example
Decodes a Base64 encoded password using the UTF-8 character set. 
${base64:decodeString("bXlwYXNzd29yZA==", "UTF-8"}
Returns the password stored in Azure Key Vault as the DevOpsGen2Pw secret and caches it for two seconds. The expression allows any user belonging to the devops group access to the secret when validating, previewing, or running the pipeline: 
${credential:getWithOptions("azure", "devops", "DevOpsGen2Pw", "refresh=2000")}
Returns the contents of the restricted JDBCpassword.txt file, trimming any leading or trailing whitespace characters: 
${runtime:loadResource("JDBCpassword.txt", true)}
Replaces the expression with the value of the Directory runtime parameter defined in the pipeline. 
${Directory}