Snowflake Pipeline Properties

When you configure a Snowflake pipeline, you specify the Snowflake URL for your account. You can optionally define the Snowflake warehouse, database, schema, and role for the pipeline.

The pipeline uses this information to access Snowflake data, by default. You can override the database and schema in individual stages as needed.

If you find that you are entering the same Snowflake details in each pipeline, you can simplify pipeline configuration by configuring Snowflake pipeline defaults in your StreamSets account.

The user credentials required to connect to Snowflake are requested when needed, and then securely stored with your account details. For more information, see Snowflake Credentials.

Default Role and Overrides

The role used to connect to Snowflake can help limit access to your Snowflake account. You can specify the Snowflake role used for a pipeline in the following ways:
Snowflake pipeline defaults
When you specify a role as a Snowflake pipeline default in your StreamSets account, then that role is used in all new pipelines by default.
You can update the role as needed in individual pipelines.
Pipeline properties
If you enter a specific role in the Role property on the General tab of pipeline properties, then that role is used to connect to Snowflake when you run the pipeline.
Snowflake credentials
If you specify a role as part of the Snowflake credentials in your StreamSets account, then that role is used when one is not defined in any of the locations above.
Note: If no role is defined in any of the specified locations, Snowflake uses the default role for the user. If the user has no default role, Snowflake uses the Public role. For more information, see the Snowflake documentation.

Applying Snowflake Pipeline Defaults to Existing Pipelines

If you have Snowflake pipeline defaults defined in your StreamSets account settings, they are only used in new pipelines by default. You can apply Snowflake pipeline defaults to existing pipelines on a case-by-case basis.

When you apply Snowflake pipeline defaults, the Snowflake pipeline defaults defined in your account are added to the pipelines, overriding the corresponding property configurations. No changes occur for properties that are not defined in your StreamSets account.

For example, say you define the Account URL and Warehouse Snowflake pipeline defaults. When you apply the Snowflake pipeline defaults to an existing pipeline, the definitions for the Snowflake URL and Warehouse properties in the pipeline are replaced with the Snowflake pipeline defaults. Other pipeline properties remain untouched.

You might apply Snowflake pipeline defaults to an existing pipeline if the pipeline was created before you defined or updated Snowflake pipeline defaults.

  1. In the pipeline canvas, click More > Use Snowflake Defaults.
    A warning message indicates that applying Snowflake pipeline defaults overwrites existing pipeline properties.
  2. If this is acceptable, click OK.
    On the General tab, pipeline properties are replaced with the Snowflake pipeline defaults that are defined in your StreamSets account.