Control Hub Requirements

StreamSets Control Hub has the following minimum requirements:

Component Minimum Requirement
StreamSets Data Collector StreamSets recommends using the latest version of Data Collector with Control Hub.
The minimum supported Data Collector version depends on how you use Data Collector:
  • Version 2.1.0.0 or later is required to design pipelines in Data Collector and to run standalone and cluster pipelines from jobs.
  • Version 3.0.0.0 or later is required as the authoring Data Collector used to design pipelines in Control Hub.
  • Version 3.2.0.0 or later is required as the authoring Data Collector used to design pipeline fragments.
  • Version 3.4.0 or later is required to monitor the CPU load and memory usage of each Data Collector from within Control Hub.
  • Version 3.19.0 or later is required to create and use connections.
StreamSets Transformer StreamSets recommends using the latest version of Transformer to design and execute Transformer pipelines from Control Hub.

Version 3.16.0 or later is required to use connections.
Statistics aggregator Use one of the following systems to aggregate pipeline statistics when jobs run on multiple Data Collectors:
  • Amazon Kinesis Streams
  • Kafka version supported by Data Collector
  • MapR Streams version supported by Data Collector
Note: In a development environment, you can also use SDC RPC to aggregate pipeline statistics. Using SDC RPC to aggregate statistics is not highly available and might cause the loss of some data. It should be used for development purposes only.
Browser Use the latest version of one of the following browsers:
  • Google Chrome
  • Firefox
  • Safari

Ensure that the browser can access registered Data Collectors and Transformers.

Browser Access to Data Collector and Transformer

The web browser used to access Control Hub must be able to reach the following components:
Authoring engines
Authoring Data Collectors and Transformers accept inbound connections from the web browser when you design pipelines using Pipeline Designer.
Execution engines
Execution Data Collectors and Transformers accept inbound connections from the web browser when you complete the following tasks:
  • Capture and view snapshots in an active Data Collector job.
  • Monitor real-time statistics on the Realtime Summary tab for an active Data Collector or Transformer job.
  • Monitor error records encountered by a pipeline stage in an active Data Collector job.
  • View the execution engine log when monitoring an active Data Collector or Transformer job.
  • View configuration properties, active Java threads, metric charts, logs, and directories when monitoring a Data Collector or Transformer from the Execute view.

Configure network routes and firewalls so that the Control Hub web browser can reach the URLs of registered Data Collectors and Transformers.

If registered Data Collectors and Transformers are installed on a cloud computing platform such as Amazon Elastic Compute Cloud (EC2), configure them to use a publicly accessible URL as described in Publicly Accessible URL for Data Collector or Publicly Accessible URL for Transformer.

If Data Collector containers are provisioned on Kubernetes, you must expose the container outside the cluster using a Kubernetes service as described in Defining a Deployment YAML Specification.
Tip: To ensure that the Control Hub web browser has access to registered engines, click Execute > Data Collectors or Execute > Transformers in the Navigation panel, and then click the URL for each registered Data Collector or Transformer.

Control Hub IP Addresses

By default, Control Hub does not directly send requests to any systems or machines in your corporate network, including registered Data Collectors and Transformers and your origin and destination systems.

However, you can configure a Control Hub subscription to directly send a webhook action to a system in your corporate network. If the receiving system resides behind a firewall or in a system that limits access to specific IP addresses, you must add all Control Hub IP addresses to the allow list for the firewall or system.

Control Hub currently uses the following IP addresses to send webhook actions:
  • 54.69.142.167
  • 52.0.167.89
  • 52.37.153.87
  • 44.233.192.54