Snowflake

You can create the following types of Snowflake connections:

Snowflake Connection

Available when using an authoring Data Collector version 3.20.0 or later.

To create a Snowflake connection, the appropriate Snowflake stage library must be installed on the selected authoring Data Collector:
  • Data Collector 5.4.0 or later - Requires the Snowflake stage library, streamsets-datacollector-sdc-snowflake-lib.
  • Data Collector 3.20.0 to 5.3.x - Requires version 1.7.x or later of the Snowflake Enterprise stage library, streamsets-datacollector-snowflake-lib.

For a description of the Snowflake connection properties, see Snowflake Connection Properties.

After you create a Snowflake connection, you can use the connection in the following stages:
Engine Stage
Data Collector 3.20.0 or later
  • Snowflake destination
  • Snowflake File Uploader destination
  • Snowflake executor

Transformer 3.18.0 or later
  • Snowflake origin
  • Snowflake Lookup processor
  • Snowflake destination

Snowflake Roles and Privileges

When you configure a Snowflake stage, the Snowflake user that you specify in the stage must have a Snowflake role that includes all of the privileges required to perform the necessary Snowflake tasks.

Before using a connection in a Snowflake stage, ensure that the Snowflake user has the appropriate privileges. All Snowflake stages require specific privileges, except for the Data Collector Snowflake File Uploader destination.

For details about privilege requirements, see the documentation for the stage:
If you create a custom role with the required privileges, define the role that the stage uses in one of the following ways:
Assign the custom role as the default role
In Snowflake, assign the custom role as the default role for the Snowflake user account specified in the stage. A Snowflake user account is associated with a single default role.
Override the default role with the custom role
In the stage, select Use Snowflake Role on the connection information tab. Then, specify the name of the custom role. The custom role overrides the default role assigned to the Snowflake user account specified in the stage.
For example, you might define custom roles in Snowflake for specific data sources, and then specify one of the roles when using a Snowflake destination to load data.

Snowflake Connection Properties

When creating a Snowflake connection, configure the following properties on the Snowflake Connection Info tab:
Snowflake Connection Info Property Description
Include Organization Enables specifying the Snowflake organization.

Available with Snowflake Enterprise stage library 1.12.0 or later.
Snowflake Region Region where the Snowflake warehouse is located. Select one of the following regions:
  • An available Snowflake region.
  • Other - Use to specify a Snowflake region not listed in the property.
  • Custom JDBC URL - Use to specify a virtual private Snowflake installation.

Available when Include Organization is disabled.
Custom Snowflake Region Custom Snowflake region. Available when using Other as the Snowflake region.
Custom Snowflake URL Custom JDBC URL to use when using a virtual private Snowflake installation.
Organization Snowflake organization.

Available with Snowflake Enterprise stage library 1.12.0 or later.
Account Snowflake account name.
User Snowflake user name.

The user account or the custom role that overrides the default role for this user account must have the required Snowflake privileges.

The required privileges depend on the stage where you use the connection. For details, see Snowflake Roles and Privileges.
Password Snowflake password.
Use Snowflake Role Overrides the default role for the specified user account.

The custom role must have the required Snowflake privileges, which are dependent on the stage where you use the connection. For details, see Snowflake Roles and Privileges.
Snowflake Role Name Name of the custom Snowflake role to use.
Connection Properties Additional Snowflake connection properties to use.

To add properties, click Add and define the property name and value. Use the property names and values as expected by Snowflake.

Available with Snowflake Enterprise stage library 1.11.0 or later.

Snowpipe Connection

Available when using an authoring Data Collector version 3.20.0 or later.

To create a Snowpipe connection, the appropriate Snowflake stage library must be installed on the selected authoring Data Collector:
  • Data Collector 5.4.0 or later - Requires the Snowflake stage library, streamsets-datacollector-sdc-snowflake-lib.
  • Data Collector 3.20.0 to 5.3.x - Requires version 1.7.x or later of the Snowflake Enterprise stage library, streamsets-datacollector-snowflake-lib.

For a description of the Snowpipe connection properties, see Snowpipe Connection Properties.

After you create a Snowpipe connection, you can use the connection in the following stage:
Engine Stage
Data Collector 3.20.0 or later Snowflake destination

Snowpipe Connection Properties

Define a Snowpipe connection when configuring the Data Collector Snowflake destination to use the Snowpipe load method. Before creating a Snowpipe connection, make sure to complete the Snowpipe prerequisite tasks.

When creating a Snowpipe connection, configure the following properties on the Snowpipe tab:

Snowpipe Property Description
Private Key PEM Private key PEM. Generated in Snowflake as part of the Snowpipe prerequisite tasks.
Tip: To secure sensitive information, you can use credential stores or runtime resources.
Private Key Password Private key password. Generated in Snowflake as part of the Snowpipe prerequisite tasks.
Public Key PEM Public key PEM. Generated in Snowflake as part of the Snowpipe prerequisite tasks.
Use Custom Snowpipe Endpoint Enables using a custom Snowpipe endpoint.
Use TLS Enables the use of TLS to connect to the custom Snowpipe endpoint.
Custom Snowpipe Host Host name for the custom Snowpipe endpoint.
Custom Snowpipe Port Port number for the custom Snowpipe endpoint.