Hosted or Deployed Engines

By default, organizations use the Transformer for Snowflake engine hosted by StreamSets to run pipelines.

The hosted engine suits the needs of most organizations and provides the easiest way to work with Transformer for Snowflake. When using the hosted engine, you do not need to install or manage engines. All you do is start configuring pipelines.

When necessary, you can deploy Transformer for Snowflake engines to your private network, which can be on-premises or on a protected cloud computing platform such as AWS. You might need to deploy Transformer for Snowflake engines, rather than using the StreamSets hosted engine, due to company policies or security requirements.

An organization can use either the hosted engine or deployed engines, not both. Deploying Transformer for Snowflake engines requires that your organization have the appropriate account agreement. For more information about your account agreement, contact your StreamSets account team.

Most Transformer for Snowflake functionality is the same for both engines, such as the stages you can use in pipelines. As a result, most of this Transformer for Snowflake guide is appropriate for all users. However, there are sections of documentation that apply only to organizations using hosted or deployed engines.

The following information summarizes the key engine differences that affect how you use this documentation. For more information about the differences between the hosted and deployed engines, see the Control Hub documentation.

Hosted engine
When your organization uses the hosted engine:
  1. You can get started with Transformer for Snowflake by simply creating a new pipeline.

    You can use the Transformer for Snowflake tutorial as a guide.

  2. Transformer for Snowflake ensures that you provide the required credentials and connection information, and securely saves the information to your StreamSets account. In addition:
    • The connection information that you initially provide is stored as Snowflake pipeline defaults and used in subsequent pipelines.

    • You can update your credentials and Snowflake pipeline defaults on the Snowflake Settings tab of your StreamSets account.

    • You can also override Snowflake pipeline defaults in individual pipelines and jobs as needed.

  3. Since StreamSets hosts and manages the engine, the Installation chapter in this guide is not relevant to you.
Deployed engines
When your organization uses deployed engines:
  1. Before you create Transformer for Snowflake pipelines, you create a deployment and install your engines, just like Data Collector and Transformer engines.

    You can use the Installation chapter in this guide to install, configure, and maintain deployed engines. For more information about deployments, see the Control Hub documentation.

  2. You must also create one or more Snowflake connections where you specify the credentials and connection information to use.

    For more information about connections, or details about the Snowflake connection, see the Control Hub documentation.

  3. When you configure a pipeline, you simply select the appropriate Snowflake connection.

    As a result:
    • Your StreamSets account does not include a Snowflake Settings tab for configuring credentials or connection information.
    • Snowflake pipeline defaults are not available for use, but you can override the following connection details in pipelines: role, warehouse, database, and schema.