Data Collector Communication
StreamSets Control Hub works with Data Collector to design pipelines and to execute standalone and cluster pipelines.
Control Hub runs on a public cloud service hosted by StreamSets - you simply need an account to get started. You install Data Collectors in your corporate network, which can be on-premises or on a protected cloud computing platform, and then register them to work with Control Hub.
- Authoring Data Collector
- Use an authoring Data Collector to design pipelines and to create connections. You can design pipelines in the Control Hub after selecting an available authoring Data Collector. The selected authoring Data Collector determines the stages, stage libraries, and functionality that display in Pipeline Designer.
- Execution Data Collector
- Use an execution Data Collector to execute standalone and cluster pipelines run from Control Hub jobs.
A single Data Collector can serve both purposes. However, StreamSets recommends dedicating each Data Collector as either an authoring or execution Data Collector.
Registered Data Collectors use encrypted REST APIs to communicate with Control Hub. Data Collectors initiate outbound connections to Control Hub over HTTPS on port number 443.
The web browser that accesses Control Hub Pipeline Designer uses encrypted REST APIs to communicate with Control Hub. The web browser initiates outbound connections to Control Hub over HTTPS on port number 443.
The authoring Data Collector selected for Pipeline Designer or connection creation accepts inbound connections from the web browser on the port number configured for the Data Collector. Similarly, the execution Data Collector accepts inbound connections from the web browser when you monitor real-time summary statistics, error information, and snapshots for active jobs. The connections must be HTTPS.
The following image shows how authoring and execution Data Collectors communicate with Control Hub:
