Data Preview

Data Preview Overview

You can preview data to help build or fine-tune a pipeline. When using Control Hub, you can also use data preview when developing pipeline fragments.

You can use data preview with complete or incomplete pipelines and fragments. And you can choose from several options to provide source data for the preview.

When you preview data, source data passes through the pipeline or fragment, allowing you to review how the data passes and changes through each stage. You can edit stage properties and run the preview again to see how your changes affect the data. You can also edit preview data to test and tune the pipeline logic.

You can preview data for one stage at a time or for a group of stages. You can also view the data in list or table view, and refresh the preview data.

Data Preview Availability

You can preview complete and incomplete pipelines and Control Hub pipeline fragments. The Data Preview icon becomes active when data preview is available.

You can preview data under the following conditions:

All stages in the pipeline are connected
All required properties are defined

Tip: Stage configuration does not have to be accurate or complete to preview data. After you connect all stages, you can enable data preview by entering any valid value for required properties.

Source Data for Data Preview

You can use the following types of data for a data preview:

Data from the origin - Use available data from the origin.
Data from the test origin - Use data from the test origin configured in the pipeline or fragment properties.
Data from a snapshot - Use snapshot data from the same pipeline or another pipeline. Available for pipelines only.

Writing to Destinations and Executors

Since data preview is a tool for development, by default, it does not write data to destination systems or pass data to executors in the pipeline.

Data preview also does not display the data that is written by destinations in the pipeline. You can, however, view the data that is passed to a destination stage, which is typically similar to what is written to destination systems.

If you like, you can configure the preview to write data to destination systems and to pass data to executors in the pipeline, which triggers their actions. For example, you might enable writing to an executor to verify that it performs the configured task as expected.

To write to destination systems and pass data to executors, in the Preview Configuration dialog box, select Write to Destinations and Executors.

Important: StreamSets advises against writing preview data to or triggering executor actions in production systems.

Notes

Keep the following notes in mind when previewing your data:

Date, datetime, and time data - Data preview displays date, datetime, and time data using the default format of the browser locale. For example, if the browser uses the en_US locale, preview displays dates using the following format: MMM d, y h:mm:ss a.
Oracle CDC pipelines - When previewing a pipeline with the Oracle CDC origin, preview generates a maximum of 10 records, ignoring the Preview Batch Size property when set to a higher value.
Also, preview allows up to 5 seconds for the origin to receive records, which starts after connecting to Oracle. As a result, when previewing a pipeline with an Oracle CDC origin, preview ignores the configured Preview Timeout property when set to a higher value.
Oracle CDC Client pipelines - Due to the complex nature of the Oracle CDC Client origin, initiating preview can take longer than expected. If preview times out, try increasing the Preview Timeout property incrementally to allow the origin time to connect.
Also, to generate more than one preview record, set the Max Batch Wait Time property for the origin to a value greater than 0.
Whole file data format - When previewing a pipeline that processes whole file data, data preview displays only one record.

Data Collector UI - Preview Mode

You can use Data Collector to view how data passes through the pipeline.

The following image shows Data Collector in Preview mode:

Area/Icon	Name	Description
1	Pipeline canvas	Displays the pipeline.
2	Preview panel	Displays the data that enters and exits the selected stage or group of stages. It can also display stage properties and preview configuration.
	StreamSets Control Hub icon	Provides information about StreamSets Control Hub and lets you register this Data Collector with Control Hub.
	Home icon	Displays a home page with a list of pipelines and their statuses, allowing you to perform pipeline maintenance and navigate to individual pipelines.
	Package Manager icon	Displays the Package Manager which allows you to install additional stage libraries for a core or common Data Collector installation.
	Notifications icon	Displays notifications.
	Administration icon	Provides access to Data Collector configuration properties, directories, and log. Also allows you to shut down Data Collector.
	User icon	Displays the active user and the roles assigned to the user. Also allows you to log out of Data Collector.
	Help icon	Provides context-sensitive help based on the information in the panel. Allows you to configure display settings and to specify whether to use a local or hosted version of the help. Provides access to the REST API and the Data Collector version.
	Link to a pipeline list	Link to a pipeline list on the Home page. Use to view a list of available pipelines, perform pipeline maintenance like starting or sharing a pipeline, and navigate to individual pipelines.
	Records icon	Displays data preview records.
	Stage Configuration icon	Displays stage properties.
	Preview Configuration icon	Displays data preview properties.
	Single button	Displays input and output data for a single stage.
	Multiple button	Displays input and output data for a group of stages.
	List View icon	Displays preview data in a list.
	Table View icon	Displays preview data in a table.
	Previous Stage icon	Moves the preview to the previous stage.
	Next Stage icon	Moves the preview to the next stage.
	Refresh Preview icon	Provides a fresh set of data from the origin.
	Run Preview with Changes icon	Runs the data preview using changed data. Use to see how edited data is processed by the pipeline.
	Revert Changes icon	Reverts all changes to preview data and returns the preview to the origin.
	Close Preview icon	Closes the data preview.

Note: Some icons and options might not display. The items that display are based on the task that you are performing and roles assigned to your user account.

For information about maintaining pipelines on the Home page, see Data Collector UI - Pipelines on the Home Page.

For information about configuring pipelines, see Data Collector UI - Edit Mode.

For information about pipeline monitoring options, see Data Collector UI - Monitor Mode.

Preview Codes

Data preview displays different colors for different types of data. Preview also uses other codes and formatting to highlight changed fields.

The following table describes the color and asterisk coding:

Preview Code	Description
Black values	Date data
Blue values	Numeric data
Green values	String data
Red values	Boolean data
Asterisk	Records that include edited field values
Red italic field labels	Fields that contain edited data
Light red background	Fields removed by a stage
Italic values	Edited data
Green stage	First stage in a multiple-stage preview
Red stage	Last stage in a multiple-stage preview

Previewing a Single Stage

You can preview data for a single stage. In the Preview panel, you can review the values for each record to determine if the stage transforms data as expected.

Above the pipeline canvas, click the Preview icon: .
If the Preview icon is disabled, check the Issues list for unconnected stages and required properties that are not defined.

In the Preview Configuration dialog box, configure the following properties:


Preview Property	Description
Preview Source	Source data for the preview: Configured Source - Provides data from the origin system. Test Origin - Provides data from the test origin configured for the pipeline. Snapshot Data - Uses available snapshot data. Available for pipelines only.
Preview Batch Size	Number of records to use in the preview. Honors values up to the Data Collector preview batch size. Default is 10. The Data Collector default is 10.
Preview Timeout	Milliseconds to wait for preview data. Use to limit the time data preview waits for data to arrive at the origin. Relevant for transient origins only.
Write to Destinations and Executors	Determines whether the preview passes data to destinations or executors. By default, does not pass data to destination or executor stages.
Execute Pipeline Lifecycle Events	Triggers the generation of any appropriate pipeline events, typically the Start event. If the event is configured to be used, event consumption is also triggered.
Show Record/Field Header	Displays record header attributes and field attributes when in List view. Attributes do not display in Table view.
Show Field Type	Displays the data type for fields in List view. Field types do not display in Table view.
Snapshot Data	When using a snapshot for source data, select the snapshot to use.
Remember the Configuration	Stores the current preview configuration for use every time you request a preview for this pipeline. After you run data preview, you can change this option in the Preview panel by selecting the Preview Configuration icon () and clearing the option. The change takes effect the next time you run data preview.

Click Run Preview.
The Preview panel highlights the origin stage and displays preview data in list view. Since this is the origin of the pipeline, no input data displays.
To view preview data in table view, click the Table View icon: .
To delete a record that you do not want to use, click the Delete icon.
To view data for the next stage, click the Next Stage icon. Or, to view data for a different stage, select the stage in the pipeline canvas.
To refresh the preview, click the Refresh Preview icon: .
Refreshing the preview provides a new set of data.
To exit data preview, click the Close Preview icon: .

Previewing Multiple Stages

You can preview data for a group of linked stages within a pipeline.

When you preview multiple stages, you select the first stage and the last stage in the group. The Preview panel then displays the output data of the first stage in the group and the input data of the last stage in the group.

In the Preview panel, you can review the values for each record to determine if the group of stages transforms data as expected.

Above the pipeline canvas, click the Preview icon: .
If the Preview icon is disabled, check the Issues list for unconnected stages and required properties that are not defined.

In the Preview Configuration dialog box, configure the following properties:


Preview Property	Description
Preview Source	Source data for the preview: Configured Source - Provides data from the origin system. Test Origin - Provides data from the test origin configured for the pipeline. Snapshot Data - Uses available snapshot data. Available for pipelines only.
Preview Batch Size	Number of records to use in the preview. Honors values up to the Data Collector preview batch size. Default is 10. The Data Collector default is 10.
Preview Timeout	Milliseconds to wait for preview data. Use to limit the time data preview waits for data to arrive at the origin. Relevant for transient origins only.
Write to Destinations and Executors	Determines whether the preview passes data to destinations or executors. By default, does not pass data to destination or executor stages.
Execute Pipeline Lifecycle Events	Triggers the generation of any appropriate pipeline events, typically the Start event. If the event is configured to be used, event consumption is also triggered.
Show Record/Field Header	Displays record header attributes and field attributes when in List view. Attributes do not display in Table view.
Show Field Type	Displays the data type for fields in List view. Field types do not display in Table view.
Snapshot Data	When using a snapshot for source data, select the snapshot to use.
Remember the Configuration	Stores the current preview configuration for use every time you request a preview for this pipeline. After you run data preview, you can change this option in the Preview panel by selecting the Preview Configuration icon () and clearing the option. The change takes effect the next time you run data preview.

Click Run Preview.
The Preview panel highlights the origin stage and displays preview data in list view. Since this is the origin of the pipeline, no input data displays.
To view preview data in table view, click the Table View icon: .
To preview multiple stages, click Multiple.
The Preview panel displays two lists of stages.
From the list on the left, select the first stage to use.
In the list on the right, select the last stage to use.
The Preview panel displays the input and output data for the group of stages. You can review the details of each record.
To refresh the preview, click the Refresh Preview icon: .
Refreshing the preview provides a new set of data.
To exit data preview, click the Close Preview icon: .

Editing Preview Data

You can edit preview data to view how a stage or group of stages processes the changed data. Edit preview data to test for data conditions that might not appear in the preview data set.

For example, when the stage filters integer data based on an expression, you might change the input data to test positive and negative integer values, as well as zero.

You can edit preview data in the following locations:

The output data column for an origin.
The input data column for processors.

When you edit preview data, you can pass the changed data through the pipeline, or you can revert your changes to return to the original data.

To change field values, in the Output Data column of an origin or the Input Data column for all other stages, click the value that you want to change and enter a new value.
You can edit values for any input data.
To process changed data, click the Run With Changes icon: .
This runs the data preview with the current set of data and stage configuration.

In the Input Data column, records with changed values display with an asterisk and the changed values are highlighted. The Output Data column displays the results of processing. You can change and process preview data as often as necessary.
To refresh the preview, click the Refresh Preview icon: .
Refreshing the preview provides a new set of data.
To revert changes to data, click the Revert Data Changes icon: .

Editing Properties

In data preview, you can edit stage properties to see how the changes affect preview data. For example, you might edit the expression in an Expression Evaluator to see how the expression alters data.

When you edit properties, you can test the change with the existing preview data or you can refresh the preview data.

When changing properties in the origin, refresh the preview data to test your changes. Refreshing the preview data allows the Data Collector to use the latest origin properties to process preview data instead of using the cached data.

Note: Unlike changes to data, you cannot automatically revert property changes. Manually revert any changes that you do not want to preserve.

To edit stage properties while in data preview, select the stage you want to edit and click the Stage Configuration icon: .
Change properties as needed.
To test properties changed in the origin, click the Refresh Preview icon: .
This refreshes the preview data. Based on the origin type, it might use the same data or a new set of data with the updated origin properties.

To test properties in any non-origin stage using the same set of data, click the Run With Changes icon: .
If you want to revert your change, manually change the property back.