Security Best Practices

When your organization uses deployed Transformer for Snowflake engines, you can use the following security measures to improve the security of your organization and data:

Credential stores for connection information: When your organization uses deployed Transformer for Snowflake engines, you define how to connect to Snowflake using a connection.; To secure the connection to Snowflake, use a credential store to protect sensitive connection information, such as user names, passwords, or access keys.; For more information about connections, or details about the Snowflake connection, see the Control Hub documentation.; For information about using credential stores, see Credential Stores.

HTTPS and direct engine REST APIs for communication: When you preview pipeline data, that data passes through Control Hub, by default. To avoid passing preview data through the StreamSets infrastructure, configure the web browser to use direct engine REST APIs to communicate with engines.; Enabling the use of direct engine REST APIs requires performing several tasks, such as enabling the engine to use HTTPS.; For more information, see the Control Hub documentation.
Avoid including sensitive information in configuration properties: When configuring a Transformer for Snowflake deployment, you can specify sensitive information such as the access key or secret key to your AWS credential store. To avoid including such sensitive information in plain text, save the information elsewhere and call it using StreamSets functions.; For more information, see Protecting Sensitive Data in Configuration Properties.

Protecting Sensitive Data in Configuration Properties

You configure Transformer for Snowflake configuration properties in the advanced configuration properties of the deployment.

Some properties in the configuration properties, such as the https.keystore.password property, require that you enter a password. Instead of entering the password in clear text, you can store the password outside of the configuration properties and then use a function to retrieve the sensitive data.

Tip: To access sensitive data in a credential store, develop a script that retrieves the sensitive data, then use the exec function to call the script.

You can use functions to retrieve sensitive data in the following ways:

From a file with the file function

Store the sensitive data in a separate file in the Transformer for Snowflake configuration directory, $TRANSFORMER_CONF. Then, use the file function in the configuration properties to retrieve the data, as follows:

${file("<filename>")}

For example, with the xmail.username property configured as shown below, Transformer for Snowflake retrieves the user name from the email_username.txt file located in the Transformer for Snowflake configuration directory:

xmail.username=${file("email_username.txt")}

Retrieving sensitive data from another file provides some level of security. However, the sensitive data in the additional file is still entered in clear text and thus vulnerable for others to access. For increased security, use a script or executable to retrieve the sensitive data.

Using a script or executable with the exec function

For increased security, store the sensitive data in a separate location, such as a credential store or file. Then, develop a script or executable that retrieves the sensitive data.

For example, you can develop a script that decrypts an encrypted file containing a password. Or you can develop a script that calls an external REST API to retrieve a password from a credential store.

Store the script or executable, and any related files, in the Transformer for Snowflake configuration directory, $TRANSFORMER_CONF.

Use the exec function in the configuration properties to call the script or executable as follows:

${exec("<script name>")}

For example, with the xmail.password property configured as shown below, Transformer for Snowflake runs the email_pwd.sh script in the Transformer for Snowflake configuration directory to retrieve the password:

xmail.password=${exec("email_pwd.sh")}