Using Connections in Pipeline Fragments

You can use connections in pipeline fragments to design pipeline fragments as your data sources. When you use connections in fragments, you reduce the possibility of user errors when defining connection information and you standardize the processing logic for the data sources.

For example, you might have a DevOps engineer create all connections to external systems and then share those connections with pipeline developers and data engineers.

A pipeline developer creates pipeline fragments that include origins and destinations using the connections and a processor or two to handle the expected processing logic for those data sources. A customer database contains highly sensitive customer data that must be encoded. The pipeline developer creates a fragment named CustomerSource that includes a JDBC Query Consumer origin that uses a connection and a Field Hasher processor that encodes the sensitive data.

A data engineer creates the final pipelines primarily using fragments. When creating a pipeline that reads from the customer database, the data engineer uses the CustomerSource fragment which is already configured with the correct connection information and the required encoding of the sensitive data.

For more information, see Fragments Overview.