Snowflake
Snowflake Connection
Available when using an authoring Data Collector version 3.20.0 or later.
- Data Collector 5.4.0 or later - Requires the Snowflake stage library,
streamsets-datacollector-sdc-snowflake-lib
. - Data Collector
3.20.0 to 5.3.x - Requires
version 1.7.x or later of the Snowflake Enterprise stage library,
streamsets-datacollector-snowflake-lib
.
For a description of the Snowflake connection properties, see Snowflake Connection Properties.
Engine | Stage |
---|---|
Data Collector 5.6.0 or later |
|
Data Collector 3.20.0 or later |
|
Transformer 3.18.0 or later |
|
Snowflake Roles and Privileges
When you configure a Snowflake stage, the Snowflake user that you specify in the stage must have a Snowflake role that includes all of the privileges required to perform the necessary Snowflake tasks.
Before using a connection in a Snowflake stage, ensure that the Snowflake user has the appropriate privileges. All Snowflake stages require specific privileges, except for the Data Collector Snowflake File Uploader destination.
- Assign the custom role as the default role
- In Snowflake, assign the custom role as the default role for the Snowflake user account specified in the stage. A Snowflake user account is associated with a single default role.
- Override the default role with the custom role
- In the stage, select Use Snowflake Role on the connection information tab. Then, specify the name of the custom role. The custom role overrides the default role assigned to the Snowflake user account specified in the stage.
Snowflake Connection Properties
Snowflake Connection Info Property | Description |
---|---|
Include Organization | Enables specifying the Snowflake
organization. Available with Snowflake Enterprise stage library 1.12.0 or later. |
Snowflake Region | Region where the Snowflake warehouse is located. Select one of the
following:
Available when Include Organization is disabled. |
Custom Snowflake Region | Custom Snowflake region. Available when using Other as the Snowflake region. |
Custom Snowflake URL | Custom JDBC URL to use when using a virtual private Snowflake installation. |
Organization | Snowflake organization. Available with Snowflake Enterprise stage library 1.12.0 or later. |
Account | Snowflake account name. |
User | Snowflake user name. The user account or the custom role that overrides the default role for this user account must have the required Snowflake privileges. The required privileges depend on the stage where you use the connection. For details, see Snowflake Roles and Privileges. |
Password | Snowflake password. |
Use Snowflake Role | Overrides the default role for the specified
user account. The custom role must have the required Snowflake privileges, which are dependent on the stage where you use the connection. For details, see Snowflake Roles and Privileges. |
Snowflake Role Name | Name of the custom Snowflake role to use. |
Connection Properties | Additional Snowflake connection properties to
use. To add properties, click Add and define the property name and value. Use the property names and values as expected by Snowflake. Available with Snowflake Enterprise stage library 1.11.0 or later. |
Snowpipe Connection
Available when using an authoring Data Collector version 3.20.0 or later.
- Data Collector 5.4.0 or later - Requires the Snowflake stage library,
streamsets-datacollector-sdc-snowflake-lib
. - Data Collector
3.20.0 to 5.3.x - Requires
version 1.7.x or later of the Snowflake Enterprise stage library,
streamsets-datacollector-snowflake-lib
.
For a description of the Snowpipe connection properties, see Snowpipe Connection Properties.
Engine | Stage |
---|---|
Data Collector 3.20.0 or later | Snowflake destination |
Snowpipe Connection Properties
Define a Snowpipe connection when configuring the Data Collector Snowflake destination to use the Snowpipe load method. Before creating a Snowpipe connection, make sure to complete the Snowpipe prerequisite tasks.
When creating a Snowpipe connection, configure the following properties on the Snowpipe tab:
Snowpipe Property | Description |
---|---|
Private Key PEM | Private key PEM. Generated in
Snowflake as part of the Snowpipe prerequisite
tasks. Tip: To secure sensitive
information, you can use credential stores or runtime resources. |
Private Key Password | Private key password. Generated in Snowflake as part of the Snowpipe prerequisite tasks. |
Public Key PEM | Public key PEM. Generated in Snowflake as part of the Snowpipe prerequisite tasks. |
Use Custom Snowpipe Endpoint | Enables using a custom Snowpipe endpoint. |
Use TLS | Enables the use of TLS to connect to the custom Snowpipe endpoint. |
Custom Snowpipe Host | Host name for the custom Snowpipe endpoint. |
Custom Snowpipe Port | Port number for the custom Snowpipe endpoint. |