Fragments Tips and Best Practices

Use the following tips and best practices when designing and working with pipeline fragments.

Design Best Practices

Use identifiable fragment names and descriptions

Make sure that the pipeline fragment name and description offers enough information so the developer can easily select the fragment to use from a larger list of pipeline fragments.

Use detailed commit messages to differentiate between fragment versions

Use detailed commit messages to help a data engineer determine the version of a fragment to use.

Optionally, use pipeline labels or tags to codify fragment versions

When you have a version that is ready for testing or production, you can use pipeline labels or pipeline tags to clearly define the fragment.

Choose an appropriate authoring engine

When configuring a pipeline fragment, choose an authoring engine that is the same version as the engines that you intend to use to run the pipelines that use the pipeline fragment. Using a different engine version can result in developing a pipeline fragment that is invalid for execution engines.

Transformer for Snowflake fragments do not require an authoring engine when your organization uses the hosted Transformer for Snowflake engine.

Evaluate the number of input and output streams to use

Before publishing a pipeline fragment, carefully consider the number of input and output streams that the fragment uses. After you publish a pipeline fragment, you cannot change the number of input or output streams in subsequent fragment versions.

This prevents invalidating pipelines when updating the fragment version that they use. For more information, see Fragment Input and Output.

Use a Dev Identity processor to create additional input or output streams

You can use the Dev Identity processor to create additional input or output streams in a Data Collector or Transformer fragment, creating a corresponding input or output stream in the resulting fragment stage. For more information, see Creating Additional Streams.

Use data preview to test fragment logic

You can use data preview to help design and test the processing logic of a fragment. For a Data Collector fragment, you can use a test origin to provide source data for data preview. This can be especially useful when working with a fragment that does not include an origin. When the fragment contains an origin, you can also use the origin to provide source data for the preview.

Use pipeline explicit validation to validate fragments

At this time, you cannot use explicit validation when designing a fragment. Best practice is to validate the fragment in a test pipeline before using the fragment in a production pipeline. For more information, see Explicit Validation.

After publishing a new version of a fragment, review the pipelines that use previous versions of the fragment

You can view the pipelines that use each version of a fragment in the fragment details pane. After publishing a new version of a fragment, you might want to check the pipelines that use earlier versions to see if they should use the updated fragment version.

Set appropriate prefixes when adding fragments to pipelines

Consider how you will reuse the fragment and whether you want the values of runtime parameters to be the same or different in each instance of the fragment:

Same values for runtime parameters - To use the same values for the runtime parameters in each fragment instance, enter the same prefix or remove the prefix for those fragment instances.
Different values for runtime parameters - To use different values for the runtime parameters in each fragment instance, enter a unique prefix for those fragment instances.

Usage Tips

Select the correct version of the pipeline fragment: Use the Fragment Commit/Tag stage property to select the version of the fragment to use in the pipeline. For more information, see Using Fragment Versions.
If needed, override runtime parameter values that are defined in the fragment: The pipeline inherits runtime parameters defined in the fragment. You can override the default values for those runtime parameters. You can also delete the parameter if you end up removing the fragment from the pipeline.; For more information about working with runtime parameters in pipeline fragments, see Runtime Parameters.
Use data preview or explicit validation to verify and validate fragment processing: When using a fragment in a pipeline, you can use pipeline validation and data preview to verify that the fragment processes data as expected.