Engine Communication

Control Hub runs on a public cloud service hosted by StreamSets - you simply need an account to get started. You set up and deploy Data Collector and Transformer engines in your corporate network, which can be on-premises or on a protected cloud computing platform.

Control Hub works with the engines when you design pipelines and when you run pipelines from jobs.

Engines communicate with the following components:

Control Hub: Engines use encrypted REST APIs to communicate with Control Hub. Engines initiate outbound connections to Control Hub over HTTPS on port number 443.; Engines send requests and information to Control Hub. Control Hub does not directly send requests to engines. Instead, Control Hub sends requests using encrypted REST APIs to a messaging queue managed by Control Hub. Engines periodically check with the queue to retrieve Control Hub requests. For more information, see Engine Requests to Control Hub.
Web browser: The web browser also uses encrypted REST APIs to communicate with Control Hub, initiating outbound connections to Control Hub over HTTPS on port number 443.; For some user actions, including when you design a pipeline, install additional stage libraries on engines, or monitor a job, the browser requests must reach the engines. By default for these actions, the browser initiates outbound connections to Control Hub over HTTPS, and then Control Hub forwards the requests to the engines using an encrypted WebSocket tunnel.; WebSocket tunnel communication is sufficient for most use cases and does not require additional setup. However, you can configure the engines to use the HTTPS protocol to directly connect to the web browser instead.