Pipeline Defaults and Credentials

When you configure a Snowflake pipeline, you specify the Snowflake URL for your account, the warehouse, and the role for the pipeline. The pipeline uses this information to access Snowflake data.

As a new user, when you specify these details in your first pipeline, they are stored in your StreamSets account as your Snowflake pipeline defaults. Snowflake pipeline defaults are added in new pipelines so you do not have to configure that information over and over again. You can change your Snowflake pipeline defaults at any time in your StreamSets account.

The user credentials required to connect to Snowflake are requested when needed. They are also securely stored in your StreamSets account for reuse.

Default Role and Overrides

The role used to connect to Snowflake can help limit access to your Snowflake account. You can specify the Snowflake role used for a pipeline in the following ways:
Snowflake pipeline defaults
When you specify a role as a Snowflake pipeline default in your StreamSets account, then that role is used in all new pipelines by default.
You can update the role as needed in individual pipelines.
Pipeline properties
If you do not have a role defined as a Snowflake pipeline default, the pipeline uses the Public role, by default. You can change this by configuring the Role property on the General tab of pipeline properties.
Note: If no role is defined in the locations above, Snowflake uses the default role for the user. If the user has no default role, Snowflake uses the Public role. For more information, see the Snowflake documentation.

Specifying Snowflake Credentials

If you are a new user who has not provided your Snowflake credentials, Transformer for Snowflake requests the Snowflake credentials when you create your first Transformer for Snowflake pipeline. The credentials are then validated and stored securely for reuse.

You can update your Snowflake credentials as needed in your StreamSets account. You can also update the credentials when you configure a pipeline.
Important: When you specify Snowflake credentials, they replace any previously-stored credentials and are used for all subsequent connections to Snowflake.
  1. In the pipeline canvas, click More > Enter Snowflake Credentials.
  2. In the Snowflake Credentials dialog box, configure the following properties, and then click Save:
    Snowflake Credentials Property Description
    Username Snowflake user name.
    Authentication Method Authentication method to use: password or private key.
    Password Password for the Snowflake account.

    Available when using password authentication.
    Private Key Private key for the Snowflake account. Enter a PKCS#1 or PKCS#8 private key and include the key delimiters.

    For example, when entering a PKCS#8 private key, include the -----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY----- key delimiters.

    Available when using private key authentication.
    Role Optional role to use when a role is not defined in the pipeline.

    When you do not specify a role in a pipeline or in this property, Snowflake uses the default role for the user. If the user has no default role, Snowflake uses the Public role. For more information, see the Snowflake documentation.

    Use to limit access to the Snowflake account. However, note that the following permissions are generally required to run Snowflake pipelines:
    • Read
    • Write
    • Create Table

Applying Snowflake Pipeline Defaults

If you have Snowflake pipeline defaults defined in your StreamSets account settings, they are only used in new pipelines by default. You can apply Snowflake pipeline defaults to existing pipelines on a case-by-case basis.

When you apply Snowflake pipeline defaults, the Snowflake pipeline defaults defined in your account are added to the pipelines, overriding the corresponding property configurations. No changes occur for properties that are not defined in your StreamSets account.

For example, say you define the Account URL and Warehouse Snowflake pipeline defaults. When you apply the Snowflake pipeline defaults to an existing pipeline, the definitions for the Snowflake URL and Warehouse properties in the pipeline are replaced with the Snowflake pipeline defaults. Other pipeline properties remain untouched.

You might apply Snowflake pipeline defaults to an existing pipeline if the pipeline was created before you defined or updated Snowflake pipeline defaults.

  1. In the pipeline canvas, click More > Use Snowflake Defaults.
    A warning message indicates that applying Snowflake pipeline defaults overwrites existing pipeline properties.
  2. If this is acceptable, click OK.
    On the General tab, pipeline properties are replaced with the Snowflake pipeline defaults that are defined in your StreamSets account.