Pipelines Overview

Transformer for Snowflake pipelines enable you to process Snowflake data without writing a SQL query or template.

Snowflake pipelines process Snowflake data with batch processing. That is, each time a Snowflake pipeline runs, all data specified in pipeline origins is processed, then the pipeline stops.

Snowflake pipelines can include the following stages:
  • One or more origins to read from Snowflake
  • Processors, as needed, to transform the data
  • One or more destinations to write to Snowflake or to perform tasks such as creating a Snowflake view or removing data from the pipeline.
  • Executors, as needed, to perform tasks in Snowflake

You can also use the features described in this chapter.

After you configure a pipeline, you create a job to run the pipeline.

For general information about Snowflake pipelines, see Understanding Snowflake Pipelines.