Firewall Configuration
When you deploy StreamSets engines to on-premise or cloud computing machines that reside behind a firewall or in a system that limits access to specific IP addresses, allow the required inbound and outbound traffic to each machine.
Inbound Connections
Control Hub does not directly send requests to machines running StreamSets engines. However, you must configure your firewall to allow the following inbound connections to the machines, depending on the engine type and configuration:
Engine Type | Port | Protocol | Usage |
---|---|---|---|
Transformer | Transformer port - 19630 by default | TCP | The Apache Spark cluster must be able to access Transformer at this port number to send the status, metrics, and offsets for running pipelines. |
Data Collector and Transformer | HTTPS port defined in the engine advanced configuration properties of the deployment | TCP | When using direct engine REST APIs for browser to engine
communication, web browsers must be able to reach engines on the
configured HTTPS port number. In most cases, you can use the default WebSocket tunneling communication method and do not need to allow an inbound connection to the HTTPS port number. For more information, see Engine Communication. |
In addition, if you want to use SSH to connect to machines running StreamSets engines, configure your firewall to allow the following inbound connection to the machines. Control Hub does not require SSH access to the machines. However, you might want to enable access for troubleshooting purposes.
Engine Type | Port | Protocol | Usage |
---|---|---|---|
Data Collector and Transformer | 22 | TCP | Optionally connect to the machine using SSH. |
Outbound Connections
StreamSets engines make outbound connections to the following systems. Ensure that your firewall allows outbound connectivity to these systems.
System | DNS and IP Address | Port | Protocol | Usage |
---|---|---|---|---|
StreamSets authentication service | cloud.login.streamsets.com Allow all of the following:
|
443 | TCP, TLS 1.2 or later | User authentication with StreamSets. |
StreamSets identity provider | Allow all of the following:
|
443 | TCP, TLS 1.2 or later | Identity management for username/password and social
logins. SAML logins use the StreamSets authentication service. |
StreamSets Control Hub | Allow all of the following:
|
443 | TCP, TLS 1.2 or later | Engine communication with Control Hub. |
StreamSets server that hosts engine installation and stage library files | archives.streamsets.com | 443 | TCP, TLS 1.2 | Engine and stage library file downloads. |
StreamSets telemetry server | Allow all of the following:
|
443 | HTTPS | Telemetry data collection. |
External origin and destination systems | Depends on the system | Depends on the system | Depends on the system | External system connections so that pipeline stages can process your data. |