Overview

A connection defines the information required to connect to an external system.

Pipelines communicate with external systems to read and write data. Most of these external systems require sensitive information, such as user names or passwords, to access the data. When you configure pipelines and pipeline fragments, you can enter the details needed to connect to the external system, or you can select an existing connection that contains the details.

Using connections provides the following benefits:
Increased security
When you use connections, you can limit the number of users that need to know the security credentials for external systems.
For example, you want to ensure that only the DevOps team knows the security credentials required to access external systems. A DevOps engineer logs into Control Hub to create all connections to the external systems, and then shares the connections with data engineers who design pipelines, granting them the ability to use the connections. Data engineers select the appropriate connection name for a pipeline stage, but cannot view the connection details.
Reusability
You can create a connection once and then reuse that connection in multiple pipelines. Reusing connections reduces the possibility of user errors and simplifies updates to connection values.
For example, you might create a single connection to your source data stored in Amazon S3. You name the connection SourceData. You develop multiple pipelines to process this source data. Each time you add an Amazon S3 origin to a pipeline, you simply select the existing SourceData connection. You do not need to re-enter the AWS authentication details for each Amazon S3 origin. When you need to update the authentication details, you make a single update to the connection. All Amazon S3 origins using that connection reflect the updated values in subsequent pipeline runs.

When you create a connection, you select an available authoring Data Collector. The Data Collector version and the installed stage libraries determine the connection types, such as Amazon S3 or JDBC, that you can create.

You can use connections in Control Hub when designing Data Collector or Transformer pipelines. You cannot use connections in the Data Collector or Transformer pipeline canvas.

For more information on the supported connection types, see Connection Types Overview.