Enabling Data Drift Handling

The Snowflake destination can automatically compensate for changes in column or table requirements, also known as data drift.

Note: If you change the destination table schema manually rather than enabling data drift handling, you must restart the pipeline to allow the destination to discover schema changes.

The destination can handle data drift in the following ways:

Create new columns

The destination can create new columns in Snowflake tables when new fields appear in records. For example, if a record suddenly includes a new Address2 field, the destination creates a new Address2 column in the target table.

By default, the destination creates new columns based on the data in the new fields, such as creating a Double column for decimal data. You can, however, configure the destination to create all new columns as Varchar.

To enable the automatic creation of new columns, select the Data Drift Enabled property on the Snowflake tab. To create new columns as Varchar, select the Create New Columns as Varchar property.

Create new tables

When data drift handling is enabled, you can also configure the destination to create new tables as needed. For example, say the destination writes data to tables based on the region name in the Region field. When a new SW-3 region shows up in a record, the destination creates a new SW-3 table in Snowflake and writes the record to the new table.

You can use this functionality to create all necessary tables in an empty Snowflake database schema.

To enable the creation of new tables, first enable data drift, and then select the Table Auto Create property.