Snowpipe Prerequisites

When processing new data, you can use Snowpipe, the Snowflake continuous ingestion engine, to load data to Snowflake tables.

Before using Snowpipe, complete the following prerequisites:
  1. In Snowflake, create a pipe for Snowpipe to use to load data.

    For more information about creating a pipe, see the Snowflake documentation.

  2. In Snowflake, generate a private key PEM and a public key PEM.

    For details about key-pair authentication, see the Snowflake documentation. You do not need to generate JSON Web Tokens (JWT) as described in Step 5.

    When you configure the destination, you specify the private key PEM and password and the public key PEM.

  3. In Snowflake, assign the public key to the Snowflake user account configured in the destination.

    You can use the Snowflake console or the ALTER USER command.

  4. Optionally, to secure the private key PEM and password, use runtime resources or credential stores.credential stores. For more information about credential stores, see Credential Stores in the Data Collector documentation.
  5. In Snowflake, create a custom role and grant the role the access privileges required to use Snowpipe.

    For a list of the required Snowpipe access privileges, see the Snowflake documentation.

  6. Then in Snowflake, assign the custom role as the default role for the Snowflake user account specified in the destination. Or in the destination, specify the custom role that overrides the user's default role.

    For more information about defining a role for the destination, see Define a Role.