Managing Connections

Managing connections includes filtering the list of displayed connections, testing connections, upgrading connections, and editing connection values. You can also view the pipelines and fragments that use each connection.

When needed, you can delete a connection when it is not used in any pipelines or pipeline fragments.

Tip: Your organization administrator can view a history of the connections that have been created, updated, and deleted by viewing the connection audit entries.

Filtering Connections

In the Connections view, you can filter the list of displayed connections by connection type or tag. Or you can search for connections by name or description.

  1. In the Navigation panel, click Connections.
  2. If the Filter column does not display, click the Toggle Filter Column icon: Toggle Filter Column icon.

    The following image displays the Filter column in the Connections view:

    Filter column displayed in the Connections view

  3. To search for connections by name or description, enter text in the search field, and then press Return.
  4. Select a connection type or tag to additionally filter the list of connections.

    You can filter by connection types that you have access to. For example, if you do not have access to any Kudu connections, you cannot filter by the Kudu connection type.

  5. Select the Keep Filter Persistent checkbox to retain the filter when you return to the view.
    Tip: To share the applied filter, copy the URL and send it to another user in your organization.

Testing Connections

When you create or edit connections, you can test that the configured connection values are valid.

When you test a connection, you verify that the selected authoring Data Collector can connect to the external system using the configured values. Testing a connection does not guarantee that a running pipeline using the connection can also connect to the external system.

For example, let's say that you use credential functions in connection properties to retrieve sensitive values from a credential store. When you test the connection, you verify that the selected authoring Data Collector can retrieve the values from the credential store and connect to the external system. A successful test verifies that the credential store is correctly configured for the selected authoring Data Collector. If the same credential store is not correctly configured for the execution Data Collector, then when you start a job that includes a pipeline using this connection, the pipeline fails to connect to the external system.

When a connection type supports multiple stage libraries, such as a Kafka or Kudu connection, you select the stage library installed on the selected authoring Data Collector to use for testing. For example, let's say that both Apache Kafka and Cloudera CDH stage libraries are installed on the selected authoring Data Collector. To test a Kafka connection to brokers running in the Apache Kafka cluster, you select the Apache Kafka stage library.

  1. In the Navigation panel, click Connections.
  2. Select a connection and then click the Edit icon: Edit icon.
  3. In the Define Connection section, verify that a valid authoring Data Collector is selected.
  4. In the Configure Connection section, verify that all required properties are configured.
  5. Click Test Connection.

    If the connection supports multiple stage libraries installed on the selected Data Collector, expand the list of libraries next to the Test Connection button and select the stage library to use for testing.

Viewing Connection Usage

You can view the pipelines and pipeline fragments that use a connection. You might want to view the list of pipelines and fragments before editing, upgrading, or deleting a connection.

  1. In the Navigation panel, click Connections.
  2. Expand the details of a connection.

    The Pipelines/Fragments section lists the most recently updated version of the pipelines and fragments that use the connection:

    Pipelines and fragments that use the connection

  3. Click a pipeline or fragment name to open the pipeline or fragment in the canvas.

Editing Connections

You can edit connections to change most connection properties. You cannot change the connection type.

When you edit a connection, all pipelines using that connection reflect the updated values in subsequent pipeline runs.
Tip: Before editing a connection, view the list of pipelines and pipeline fragments that use the connection. That way, you can ensure that you make valid changes for all objects that use the connection.

Edit connections from the Connections view. Hover over a connection, and click the Edit icon: Edit icon.

In the Define Connection section of the dialog box, you can change the following information:
  • Name and description
  • Authoring Data Collector - When editing a connection, you can select a different authoring Data Collector.

    Ensure that the stage library required by the connection is installed on the selected authoring Data Collector.

    If you select an authoring Data Collector of a later version that includes changes to connection properties, you can choose to upgrade the connection.

  • Tags - You can assign and remove tags from the job to identify the connection in a different way.

In the Configure Connection section of the dialog box, you can change configuration properties for the connection type. For a description of each property, see Overview.

Upgrading a Connection

The Data Collector version associated with a connection determines the properties available in a connection. A later Data Collector version can include changes to connection properties.

When you edit a connection and choose a later authoring Data Collector version that includes changes to connection properties, you can upgrade the connection to use the changed properties. Alternatively, you can create a copy of the connection and then upgrade the copy to use the changed properties.

Before upgrading a connection, you might want to view the connection usage to determine the pipelines and fragments using the connection.

Important: After you upgrade a connection, earlier engine versions can no longer run pipelines using that connection.
  1. In the Navigation panel, click Connections.
  2. Hover over a connection, and click the Edit icon: .
  3. In the Define Connection step, select an authoring Data Collector of a later version.
  4. Click Save.
  5. If the later Data Collector version includes changes to connection properties, the Configure Connection step asks if you would like to upgrade the connection.

    To skip upgrading the connection, edit the property values as needed and then click Save. You can upgrade the connection at a later time.

  6. To upgrade the connection, click Upgrade.
  7. In the Upgrade Connection dialog box, click one of the following buttons:
    • Cancel - Cancels upgrading the connection and returns to the Edit Connection dialog box. Click Cancel again.
    • Upgrade - Upgrades the connection to use the changed properties and returns to the Edit Connection dialog box. Configure the changed properties as needed, and then click Save.
    • Copy & Upgrade - Creates a copy of the connection, upgrades the copy to use the changed properties, and then displays the details of the copied and upgraded connection. Edit the copied connection to configure the changed properties.

Deleting Connections

You can delete a connection when it is not used in any pipelines or pipeline fragments.
Tip: Before deleting a connection, view the list of pipelines and pipeline fragments that use the connection. Edit these pipelines and fragments to select another valid connection.
  1. In the Navigation panel, click Connections.
  2. Select connections in the list, and then click the Delete icon: Delete icon.
  3. Click OK to confirm the deletion.