Administer SDC Edge

Administering SDC Edge involves configuring, starting, shutting down, and viewing logs for the agent. When using StreamSets Control Hub, you can also use the SDC Edge command line interface to register SDC Edge with Control Hub.

Configuring SDC Edge

You can customize SDC Edge by editing the SDC Edge configuration file, <SDCEdge_home>/etc/edge.conf.

Use a text editor to edit the configuration file. To enable the changes, shut down and then restart SDC Edge.

The SDC Edge configuration file includes the following log property:
Execution Property Description
log-dir Directory where SDC Edge writes the log file.

Default is <SDCEdge_home>/log/. Be sure to uncomment the property if you change the value.
The SDC Edge configuration file includes the following execution property:
Execution Property Description
max-batch-size Maximum number of records included in a batch when the pipeline runs.

Default is 1000.
The SDC Edge configuration file includes the following process property:
Process Property Description
process-metrics-capture-interval Frequency in milliseconds that metrics about the SDC Edge process, such as memory usage, are refreshed.

Default is -1, which means that the metrics are refreshed when the HTTP REST API is called.
The SDC Edge configuration file includes the following HTTP properties:
HTTP Property Description
enabled Specifies whether a web server runs inside the SDC Edge process. When enabled, you can use the Data Collector UI or REST API to manage edge pipelines deployed to SDC Edge. Managing edge pipelines includes previewing, validating, starting, stopping, and monitoring pipelines as well as resetting the origin for pipelines.

Set to true in a development environment to simplify edge pipeline development.

Set to false in a production environment to secure the edge pipelines deployed to SDC Edge. When false, you manage edge pipelines using Control Hub or by starting pipelines when you start SDC Edge.

Default is true.
bind-address Address that SDC Edge binds to. The address is localhost and the defined port number.

Default port number is 18633.
base-http-url SDC Edge URL used to communicate with Control Hub.
Default is http://<hostname>:<port> where:
  • <hostname> is determined by SDC Edge using the following command: hostname -f.
  • <port> is the port number defined in the bind-address property.

You might want to change the default value to more easily identify multiple Edge Data Collectors registered with Control Hub. Be sure to uncomment the property if you change the value.
The SDC Edge configuration file includes the following Control Hub properties that define how SDC Edge works with Control Hub:
Control Hub Property Description
enabled Specifies whether the SDC Edge is enabled to work with Control Hub.

Default is false.
base-url URL to access Control Hub.
app-auth-token Authentication token for this SDC Edge instance when enabled to work with Control Hub.
job-labels Labels to assign to this SDC Edge. Use labels to group Edge Data Collectors registered with Control Hub. To assign multiple labels, enter a comma-separated list of labels.

Default is "all", which you can use to run a job on all registered instances of SDC Edge.

events-recipient Name of the internal Control Hub messaging queue to which SDC Edge sends events.

Do not change this value.
ping-frequency Frequency in milliseconds that SDC Edge notifies Control Hub that it is running.

Default is 6,000.
status-events-interval Frequency in milliseconds that SDC Edge informs Control Hub of the following information:
  • Status of all local and published pipelines that are running on this SDC Edge.
  • Performance information for this SDC Edge - including CPU load and memory usage.

Default is 60,000.

Starting SDC Edge

You can start SDC Edge manually or as a service:

Manual start
To manually start SDC Edge, run the following command from the SDC Edge home directory:
bin/edge
Note: In a test or development environment, you can start SDC Edge so that it skips verifying trusted certificates. To do so, run the following command from the SDC Edge home directory to manually start SDC Edge: bin/edge -insecureSkipVerify. StreamSets highly recommends that you configure SDC Edge to verify trusted certificates in a production environment.
To manually start SDC Edge and an edge pipeline at the same time, see Sending Commands to SDC Edge.
To modify the log level when you manually start SDC Edge, see Logs.
Service start
Before you can start SDC Edge as a service, you must register SDC Edge as a service on the edge device.
To start SDC Edge as a service, run the following command as an administrator from the SDC Edge home directory:
bin/edge -service start
For example, you might run the following command on Linux:
sudo opt/local/streamsets-datacollector-edge/bin/edge -service start

Shutting Down SDC Edge

You can shut down and then manually launch SDC Edge to apply changes to the SDC Edge configuration file.

Shut down SDC Edge based on whether SDC Edge is started manually or as a service:

Manual start
To shut down SDC Edge, type Ctrl+C in the command prompt.
Service start
To shut down SDC Edge started as a service, run the following command as an administrator from the SDC Edge home directory:
bin/edge -service stop
For example, you might run the following command on Linux:
sudo opt/local/streamsets-datacollector-edge/bin/edge -service stop

Restarting SDC Edge

When SDC Edge runs as a service, you can restart SDC Edge to apply changes to the SDC Edge configuration file. During the restart process, SDC Edge shuts down and then automatically restarts.

To restart SDC Edge, run the following command as an administrator from the SDC Edge home directory:
bin/edge -service restart
For example, you might run the following command on Linux:
sudo opt/local/streamsets-datacollector-edge/bin/edge -service restart

Logs

By default, SDC Edge writes log messages at the info severity level to the <SDCEdge_home>/log/edge.log file. To view the logs, simply open the edge.log file in a text editor.

To change the default log file directory, modify the SDC Edge configuration file.

You can modify the log level or send the log messages to the console when you manually start SDC Edge. You cannot modify this log information when SDC Edge runs as a service.

To modify logs, use the appropriate command:

  • To set the log level to debug for troubleshooting, run the following command from the SDC Edge home directory:
    bin/edge -debug
  • To send the log messages to the console instead of to the edge.log file, run the following command from the SDC Edge home directory:
    bin/edge -logToConsole

Register SDC Edge with Control Hub

To use SDC Edge with StreamSets Control Hub, you must register SDC Edge with Control Hub.

When you register SDC Edge to work with Control Hub, you generate an authentication token for that SDC Edge. The SDC Edge uses the authentication token to issue requests to Control Hub. Because all communication between Control Hub and a registered SDC Edge uses HTTPS, the authentication token is kept confidential.

Use the SDC Edge command line interface to register SDC Edge with Control Hub. You can register SDC Edge using a Control Hub password or using a Control Hub user authentication token.

Note: You can also use the Control Hub UI to register SDC Edge. If you download SDC Edge from Control Hub, you automatically register SDC Edge with Control Hub during the installation. If you use another installation method, you manually register SDC Edge with Control Hub after the installation.

Registering with a Password

You can use the SDC Edge command line interface to register SDC Edge with Control Hub using your Control Hub user name and password.

Run the following command from the SDC Edge home directory:
bin/edge -enableControlHub -controlHubUrl=<Control Hub URL> -controlHubUser=<user name> -controlHubPassword=<password> -controlHubLabels=<labels>
For example, the following command registers SDC Edge with the MyCompany organization in Control Hub cloud, assigning two labels to the SDC Edge:
bin/edge -enableControlHub -controlHubUrl=https://cloud.streamsets.com -controlHubUser=admin@MyCompany -controlHubPassword=MyPassword -controlHubLabels=Production,East

After running the command, restart SDC Edge to complete the registration process.

Registering with a User Authentication Token

To avoid exposing your Control Hub password, you can use the Control Hub REST API to log into the Control Hub security application and generate a user authentication token for your user account. Then use the SDC Edge command line interface to register SDC Edge with Control Hub using this user authentication token.

You might want to use a user authentication token when you need to register multiple Edge Data Collectors running on remote edge devices. You can send the user authentication token to the remote edge devices instead of sending your password.

When you generate a user authentication token, the token is valid for 24 hours. If desired, you can run a command to invalidate the token immediately after using it.

  1. Run the following command to log into the Control Hub security application and store the generated user authentication token in a temporary text file: 
    curl -X POST -d '{"userName":"<user name>", "password": "<password>"}' <Control Hub URL>/security/public-rest/v1/authentication/login --header "Content-Type:application/json" --header "X-Requested-By:SDC" -c <text file>

    For example, the following command logs into Control Hub cloud as the admin user of the MyCompany organization and stores the user authentication token in a file named cookie.txt:

    curl -X POST -d '{"userName":"admin@MyCompany", "password": "MyPassword"}' https://cloud.streamsets.com/security/public-rest/v1/authentication/login --header "Content-Type:application/json" --header "X-Requested-By:SDC" -c cookie.txt
  2. Run the following command to read the user authentication token from the temporary text file and assign it to a local environment variable named sessionToken: 
    sessionToken=$(cat cookie.txt | grep SSO | rev | grep -o '^\S*' | rev)
  3. Run the following command to display the user authentication token in the command prompt: 
    echo "Generated session token : $sessionToken"
  4. Copy the user authentication token.
  5. Run the following command from the SDC Edge home directory to use the token to register SDC Edge with Control Hub: 
    bin/edge -enableControlHub -controlHubUrl=<Control Hub URL> -controlHubUser=<user name> -controlHubUserToken=<token> -controlHubLabels=<labels>
    For example, the following command registers SDC Edge with the MyCompany organization in Control Hub cloud, assigning two labels to the SDC Edge:
    bin/edge -enableControlHub -controlHubUrl=https://cloud.streamsets.com -controlHubUser=admin@MyCompany -controlHubUserToken=d2bab2b3-2a1e-4368-9b18-23cdb9c2d97e|-1542046203506 -controlHubLabels=Production,East
  6. Restart SDC Edge to complete the registration process.
  7. Optionally invalidate the user authentication token.
    Control Hub invalidates each user authentication token after 24 hours. If desired, you can run a command to invalidate the token immediately after using it to register SDC Edge.
    Run the following command to invalidate the token:
    curl -X POST <Control Hub URL>/security/_logout --header "Content-Type:application/json" --header "X-Requested-By:SDC" --header "X-SS-REST-CALL:true" --header "X-SS-User-Auth-Token:$sessionToken" -i
    For example, the following command invalidates the user authentication token in Control Hub cloud:
    curl -X POST https://cloud.streamsets.com/security/_logout --header "Content-Type:application/json" --header "X-Requested-By:SDC" --header "X-SS-REST-CALL:true" --header "X-SS-User-Auth-Token:$sessionToken" -i

enableControlHub Command Line Options

Use the -enableControlHub command to register SDC Edge. SDC Edge does not need to be running to use the command.

Use the command from the SDC Edge home directory as follows:
bin/edge -enableControlHub \
(-controlHubUrl <Control Hub URL>) \
(-controlHubUser <Control Hub user>) \
[(-controlHubPassword <password>)] \
[(-controlHubUserToken <user token>)] \
[(-controlHubLabels <labels>)]
The following table describes the -enableControlHub options:
enableControlHub Option Description
-controlHubUrl <Control Hub URL> Required. Set to the appropriate URL:
  • For Control Hub cloud, enter https://cloud.streamsets.com.
  • For Control Hub on-premises, enter the URL provided by your system administrator. For example, https://<hostname>:18631.
-controlHubUser <Control Hub user> Required. Enter your Control Hub user ID using the following format:
<ID>@<organization ID>
-controlHubPassword <password> Required when registering with a password. Enter the password for your Control Hub user account.

Do not use with the -controlHubUserToken option.
-controlHubUserToken <user token> Required when registering with a user authentication token. Enter the user authentication token generated for your Control Hub user account.

Do not use with the -controlHubPassword option.
-controlHubLabels <labels> Optional. Assign a label to this SDC Edge. You can enter multiple labels separated by commas. Labels that you assign here are defined in the SDC Edge configuration file, <SDCEdge_home>/etc/edge.conf. To remove these labels after you register the SDC Edge, you must modify the configuration file.

Use labels to group Edge Data Collectors registered with Control Hub. If you know how you want to group your Edge Data Collectors, you can assign labels now. Or you can assign labels in Control Hub after you register SDC Edge.

Default is "all", which you can use to run a job on all registered Edge Data Collectors.

Disabling SDC Edge from Working with Control Hub

Use the SDC Edge command line interface to temporarily disable SDC Edge from working with Control Hub. SDC Edge does not need to be running to use the command.

Important: Disabling SDC Edge from working with Control Hub does not completely unregister the SDC Edge from Control Hub. To completely unregister SDC Edge, use the Control Hub.
Run the following command from the SDC Edge home directory to temporarily disable SDC Edge from working with Control Hub:
bin/edge -disableControlHub

The command sets the enabled property to false in the SDC Edge configuration file, <SDCEdge_home>/etc/edge.conf.

To enable SDC Edge to work with Control Hub again, simply set the enabled property to true in the edge.conf file.