Credential Functions

Credential functions provide access to sensitive information, such as user names and passwords, that is secured in a credential store. Use credential functions in pipeline and stage properties to enable Transformer to access external systems without exposing the those values.

Before you use a credential function, you must configure Transformer to use one of the supported credential stores.

You can use credential functions in any property that displays a key icon next to the property name, as follows:

You cannot use credential functions in all stages. For example, StreamSets intentionally does not allow the use of credential functions in the Spark SQL Expression processor. If credential functions were allowed in stages such as the Spark SQL Expression processor, any user with access to the pipeline could access or print sensitive values, compromising the security of the external system.
Important: When you use a credential function in a stage or pipeline property, the function must be the only value defined in the property. For example, you cannot include another function or a literal value along with the credential function.
You can replace any argument with a literal or an expression that evaluates to the argument. String literals must be enclosed in single or double quotation marks.
The StreamSets expression language provides the following credential functions:
credential:get(<cstoreId>, <userGroup>, <name>)
Returns the secret from the credential store. Uses the following arguments:
  • cstoreId - Unique ID of the credential store to use. Use the ID specified in the $TRANSFORMER_CONF/credential-stores.properties fileTransformer credential store configuration properties. For more information, see Enabling Credential Stores.
  • userGroup - Group that a user must belong to in order to access the secret. Only users that have execute permission on the pipeline and that belong to this group can validate, preview, or run the pipeline that retrieves the secret.
    If working with Control Hub, sSpecify the group using the required naming convention: <group ID>@<organization ID>. To grant access to all users, specify the default all group when working only with Transformer. When working with Control Hub and Transformer version 3.14.0 or later, you can specify the default group using all or all@<organization ID>. StreamSets recommends using all so that you do not need to modify credential functions when migrating pipelines from Transformer to Control Hub.
    Note: When working with Control Hub and a Transformer version earlier than 3.14.0, you must use the default all@<organization ID> group.
  • name - Name of the secret to retrieve from the credential store. Use the required format for the credential store:
    • AWS Secrets Manager - Enter the name of the secret to retrieve from Secrets Manager. Use the following format: "<name><separator><key>", where:
      • <name> is the name of the secret in Secrets Manager to read.
      • <separator> is the separator defined in the $TRANSFORMER_CONF/credential-stores.properties fileTransformer credential store configuration properties.
      • <key> is the key for the value that you want returned.
    • Azure Key Vault - Enter the name of the key or secret to retrieve from Azure Key Vault.
    • Java keystore - Enter the name of the secret added to the Java keystore file using the jks-cs add command.
Return type: String.
AWS Secrets Manager example: The following expression returns the value from the key SQLk1 of the secret SQLpassword from the awsdev credential store. Note that the expression uses an ampersand (&) as the separator argument because that is how the separator is defined in the $TRANSFORMER_CONF/credential-stores.properties fileTransformer credential store configuration properties. The expression allows any user in the devops group to access the key when validating, previewing, or running the pipeline:
${credential:get("awsdev", "devops@MyCompany", "SQLpassword&SQLk1")}
JKS example: The following expression returns the value of the OracleDBPassword secret defined in the devjks credential store and allows any user belonging to the devops group access to the secret when validating, previewing, or running the pipeline:
${credential:get("devjks", "devops@MyCompany", "OracleDBPassword")}
credential:getWithOptions(<cstoreId>, <userGroup>, <name>, <storeOptions>)
Returns the secret from the credential store using additional options to communicate with the credential store. Not applicable for the Java keystore credential store.
For example, you might use this function with Azure Key Vault to specify a different vault URL to use.
Uses the following arguments:
  • cstoreId - Unique ID of the credential store to use. Use the ID specified in the $TRANSFORMER_CONF/credential-stores.properties fileTransformer credential store configuration properties. For more information, see Enabling Credential Stores.
  • userGroup - Group that a user must belong to in order to access the secret. Only users that have execute permission on the pipeline and that belong to this group can validate, preview, or run the pipeline that retrieves the secret.
    If working with Control Hub, sSpecify the group using the required naming convention: <group ID>@<organization ID>. To grant access to all users, specify the default all group when working only with Transformer. When working with Control Hub and Transformer version 3.14.0 or later, you can specify the default group using all or all@<organization ID>. StreamSets recommends using all so that you do not need to modify credential functions when migrating pipelines from Transformer to Control Hub.
    Note: When working with Control Hub and a Transformer version earlier than 3.14.0, you must use the default all@<organization ID> group.
  • name - Name of the secret to retrieve from the credential store. Use the required format for the credential store:
    • AWS Secrets Manager - Enter the name of the secret to retrieve from Secrets Manager. Use the following format: "<name><separator><key>", where:
      • <name> is the name of the secret in Secrets Manager to read.
      • <separator> is the separator defined in either the $TRANSFORMER_CONF/credential-stores.properties fileTransformer credential store configuration properties or using the separator option, below.
      • <key> is the key for the value that you want returned.
      Azure Key Vault - Enter the name of the key or secret to retrieve from Azure Key Vault.
  • storeOptions - Additional options to communicate with the credential store.
    For AWS Secrets Manager, you can use the following options to override several properties in the $TRANSFORMER_CONF/credential-stores.properties fileTransformer credential store configuration properties:
    • separator - Specifies the separator for name and key values in the credential functions, overriding the credentialStore.<cstore ID>.config.nameKey.separator property.
    • alwaysRefresh - When set to true, forces the key to refresh its cached value before Transformer retrieves the value, overriding the credentialStore.<cstore ID>.config.cache.ttl.millis property.
    For Azure Key Vault, you can use the following options to override several properties in the $TRANSFORMER_CONF/credential-stores.properties fileTransformer credential store configuration properties:
    • url - Overrides the credentialStore.<cstore ID>.config.vault.url property.
    • retry - Overrides the credentialStore.<cstore ID>.config.credential.retry.millis property.
    • refresh - Overrides the credentialStore.<cstore ID>.config.credential.refresh.millis property.
    Use the following format to specify options:
    "<option1>=<value>,<option2>=<value>"
    For example, to set the Azure Key Vault retry property to 1000, enter the following for the options argument: 
    "retry=1000"
Return type: String.
AWS Secrets Manager example: The following expression returns the value from the key SQLk1 of the secret SQLpassword from the awsdev credential store, overriding the separator defined in the $TRANSFORMER_CONF/credential-stores.properties fileTransformer credential store configuration properties with a pipe ( | ). The expression allows any user in the devops group to access the key when validating, previewing, or running the pipeline:
${credential:getWithOptions("awsdev", "devops@MyCompany", "SQLpassword|SQLk1", "separator=|")}
Azure example: The following expression returns the value stored in the DevOpsGen1Pw secret from the azureprod credential store and caches it for two seconds. The expression allows any user belonging to the devops group access to the secret when validating, previewing, or running the pipeline:
${credential:getWithOptions("azureprod", "devops@MyCompany", "DevOpsGen1Pw", "refresh=2000")}