StreamSets Expression Language

The StreamSets expression language is based on the JSP 2.0 expression language. In Transformer, you can use the StreamSets expression language in pipeline or stage properties that are evaluated only once, before pipeline processing begins.

For example, you might use the StreamSets expression language to perform the following tasks:
  • Retrieve secrets from a credential store
  • Decrypt a Base64-encoded Kerberos keytab
  • Use the pipeline title to name the Spark application
  • Call a runtime parameter defined in the pipeline

To configure properties that manipulate data, or perform other tasks during pipeline processing, use Spark SQL. For more on this, see Expressions in Pipeline and Stage Properties.

Use the StreamSets expression language to define properties that represent numeric or string values. You can use expression completion to determine where you can use a StreamSets expression and the expression elements that you can use in that location.

You can use the following elements in a StreamSets expression:

Basic Syntax

Precede all StreamSets expressions with a dollar sign and enclose them with curly brackets, as follows: 
${<expression>}

For example, to add 2 + 2, use the following syntax: ${2 + 2}.

Expression Completion

When you configure a property, you can use Ctrl + Space Bar to invoke the StreamSets expression completion helper.

Expression completion provides a list of functions and other elements of the StreamSets expression language that are valid for use at that location. The list includes any runtime parameters that are defined for the pipeline. When an element does not display in the list, it is not a valid element at the specified location.
Tip: You can use expression completion to determine if specifying a StreamSets expression is valid for a property. Simply click in the property text box, and press Ctrl + Space Bar. When a list of elements appears, you know that creating a StreamSets expression is a valid option. If no list appears, configuring a StreamSets expression is not allowed.

Usage and Tips

Use the following information and tips to work with expression completion:

  1. To invoke expression completion and see all available elements, place the cursor where you want to create an expression and click Ctrl + Space Bar.

    A list of valid expression elements displays. When an element does not display in the list, it is not a valid element at the specified location. Scroll to view the entire list.

    You can use expression completion anywhere where you can use the StreamSets expression language.

  2. You can filter the element list by typing a few letters of the element name.

    Element names that include your input appear the top of the list. Below those are elements that include individual letters that you entered.

  3. Expression completion automatically displays when possible when you type in an expression.

    For example, after typing in re in an expression, the following list displays:

  4. Expression completion lists elements using the following icons:
    • c icon for constants

    • f icon for functions

    • Square icon for fields
    • x icon for runtime parameters defined in the pipeline

    In the following example, RootDir is a runtime parameter:

  5. To view more information about an element, highlight or select the element name:

  6. To add an element to an expression, click the element name or use the Enter/Return key on the highlighted element.
  7. When you add a function, it displays with syntax placeholders. You can view the full syntax by clicking within the parentheses.

Expression Examples

The following table includes some examples of how you might use StreamSets expressions in Transformer:
Description Example
Decodes a Base64 encoded password using the UTF-8 character set. 
${base64:decodeString("bXlwYXNzd29yZA==", "UTF-8"}
Returns the password stored in Azure Key Vault as the DevOpsGen1Pw secret and caches it for two seconds. The expression allows any user belonging to the devops group access to the secret when validating, previewing, or running the pipeline: 
${credential:getWithOptions("azure", "devops", "DevOpsGen1Pw", "refresh=2000")}
Returns the contents of the restricted JDBCpassword.txt file, trimming any leading or trailing whitespace characters: 
${runtime:loadResource("JDBCpassword.txt", true)}
Replaces the expression with the value of the Directory runtime parameter defined in the pipeline. 
${Directory}