Previewing Pipelines and Fragments

The Preview icon () becomes active when data preview is available.

To display preview data in list view, click the List View icon: . List view displays output data for origins, both input and output data for processors, and input data for destinations.

In list view, the preview uses codes and formatting to highlight different types of data and changed data.

By default when you preview a Data Collector pipeline or fragment, the data displays in list view.

To display preview data in table view, click the Table View icon: . By default, table view displays output data for origins and processors and input data for destinations.

To display both input and output data for processors in table view, select a processor in the canvas, and then toggle the Show Input switch. You might want to display both input and output data when a processor updates the data.

For example, let's say that you configure a Field Remover processor to remove the Bin field. When you display both input and output data in table view, the preview displays the Bin field in the input data only, as follows:

By default when you preview a Transformer or Transformer for Snowflake pipeline or fragment, the data displays in table view.

Preview displays data types differently, depending on the following engine types:

Data Collector

Preview displays Data Collector data types.

Transformer

Preview displays generic data types, such as Boolean, String, and List. These data types represent the Spark data types that are being used. For example, in preview, List represents the Array Spark data type, and Map can represent either the Map or Struct Spark data types.

Transformer for Snowflake

Preview displays Snowflake data types.

For all engine types, preview displays date, datetime, and time data using the default format of the browser locale. For example, if the browser uses the en_US locale, preview displays dates using the following format: MMM d, y h:mm:ss a.

Preview displays date, datetime, and time data using the time zone that you select in the preview configuration. By default, data preview displays data using the browser time zone.

When previewing data for a processor in list or table view, you can preview both the input and the output data. You can display the output records in the order that corresponds with the input records or in the order produced by the processor. The preview displays the most logical choice by default, but you can change the view as needed.

In most cases when you preview data for a processor, you want to compare input and output records side by side because the processor updates the records. For example, when you preview data for a Field Remover processor, the preview displays corresponding input and output records. The preview panel highlights the changed field in each record, as follows:

Some processors such as the Transformer or Transformer for Snowflake Aggregate processor don’t update records – they create new records. And other processors such as the Transformer or Transformer for Snowflake Sort processor reorder the records. In these cases, comparing matching input and output records isn’t relevant. It's more helpful to display the output records in the order produced by the processor.

To change the preview display order in list view, toggle the Output Order / Match with Input Order switch.

To change the preview display order in table view, enable the Show Input switch and then toggle the Match with Input Order switch.

After running preview for a pipeline or fragment, you can view the input and output schema for each stage on the Schema tab in the properties panel. The schema includes each field path and data type.

If you change the schema, for example if you remove a field, rename a field, or change the data type of a field, then you must run preview again so that the schema reflects the change.

In most cases as you configure stage properties, you can use expression completion or the Field Selector dialog box to specify a field path. However, in some cases, you might use the Schema tab to copy a field path.

For example, let’s say you are configuring a Field Type Converter processor for a Data Collector pipeline to convert the data type of a field by name. After running preview, you select the Field Type Converter in the pipeline canvas, and then click the Schema tab in the pipeline properties panel. You click the Copy Field Path to Clipboard icon () to copy the field path from the Schema tab, and then paste the field path into the processor configuration.

The following image displays a sample Schema tab with the time of the last data preview:

When you preview a Data Collector pipeline or fragment, you can choose from several options to provide source data for the preview. You can also edit preview data to test and tune the processing logic.

When previewing Transformer data, you can choose to preview using Spark libraries embedded in the Transformer installation or using the Spark cluster configured for the pipeline.

When previewing Transformer for Snowflake data, you can choose whether to limit the stage input to the data from upstream stages.

Also note that the Preview Timeout property is not honored when you preview Transformer for Snowflake pipelines.

Since data preview is a tool for development, by default, it does not write data to destination systems or display the expected output for destinations. You can, however, view the data that is passed to a destination stage, which is typically similar to what is written to destination systems.

For a preview of Data Collector data, preview also does not pass data to executors in the pipeline, by default.

If you like, you can configure the preview to write data to destination systems. When previewing Data Collector data, writing to destinations also enables passing data to executors, which triggers their actions. For example, you might enable passing data to an executor to verify that it performs the configured task as expected.