Manage Pipelines on SDC Edge

After designing edge pipelines in Data Collector and then deploying the edge pipelines to SDC Edge, you can manage the pipelines on SDC Edge. Managing edge pipelines includes previewing, validating, starting, stopping, and monitoring the pipelines as well as resetting the origin for the pipelines.

You can use the following methods to manage edge pipelines on SDC Edge:

Use the Data Collector UI.
Use this method when SDC Edge is accessible by the Data Collector machine.
Send commands to SDC Edge.
Use this method when SDC Edge is not accessible by the Data Collector machine.

Note: When edge pipelines are deployed to a production SDC Edge, you can disable the ability to manage those edge pipelines by setting the enabled HTTP property to false in the SDC Edge configuration file.

Using the Data Collector UI

When SDC Edge is accessible by the Data Collector machine and when the deployed edge pipeline has the Data Collector Edge URL property correctly configured, you can use the Data Collector UI to manage edge pipelines.

Managing edge pipelines includes previewing, validating, starting, stopping, resetting the origin, and monitoring the pipelines.

Note: When you preview, validate, or start an edge pipeline from Data Collector that has not yet been deployed to SDC Edge, Data Collector publishes and then starts the pipeline.

Open the edge pipeline in the pipeline canvas.
On the General tab, set the Data Collector Edge URL property to the URL of the SDC Edge instance that you want to manage the pipeline on.
The default URL is http://localhost:18633.
Use the Data Collector top toolbar to preview, validate, start, stop, and reset the origin for the edge pipeline running on SDC Edge - just as you manage standalone or cluster pipelines running on Data Collector.

As an edge pipeline runs, the Data Collector UI displays the pipeline in Monitor mode so that you can monitor the edge pipeline status and metrics.

Sending Commands to SDC Edge

When SDC Edge is not accessible by the Data Collector machine, you must use the command line and REST API to manage edge pipelines on Data Collector Edge (SDC Edge).

Managing edge pipelines includes previewing, validating, starting, stopping, resetting the offset, and viewing the pipeline status and metrics.

Commands to Start a Pipeline at Startup

You can start an edge pipeline at the same time that you manually start SDC Edge.

Note: You cannot start an edge pipeline at the same time that you start SDC Edge as a service.

On the edge device, run the following command from the SDC Edge home directory:

bin/edge -start=<pipelineID>

For example, use the following command to start SDC Edge and an edge pipeline with the specified ID:

bin/edge -start=FileTailToHTTP74f418f8-c7d3-4d0c-8788-b6474f7160e0

If the edge pipeline uses parameters, use the following command:

bin/edge -start=<pipelineID> -runtimeParameters='{"<parameter_name1>":"<parameter_value1>",
"<parameter_name2":"<parameter_value2>"}'

For example, use the following command if the edge pipeline uses parameters to define the file path used by the File Tail origin:

bin/edge -start=FileTailToHTTP74f418f8-c7d3-4d0c-8788-b6474f7160e0 -runtimeParameters='{"filePath":"/tmp/sds.log"}'

Commands to Manage Pipelines After Start Up

After SDC Edge is started, use the following command to start another edge pipeline:

curl -X POST http://<SDCEdge_hostname>:<port>/rest/v1/pipeline/<pipelineID>/start

For example, use the following command to start a pipeline on a running SDC Edge:

curl -X POST http://localhost:18633/rest/v1/pipeline/fileTailControl74f418f8-cl2f5-4r7k-8790-b9854h1450e8/start

If the edge pipeline uses parameters, use the following command:

curl -X POST http://<SDCEdge_hostname>:<port>/rest/v1/pipeline/<pipelineID>/start -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"<parameter_name1>":"<parameter_value1>","<parameter_name2":"<parameter_value2>"}'

To list all edge pipelines deployed to SDC Edge, use the following command:

curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipelines

To preview edge pipelines, return the status of the preview, return the preview data, and then stop the preview, use the following commands:

curl -X POST http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/preview
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/preview/<previewerID>/status
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/preview/<previewerID>
curl -X DELETE http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/preview/<previewerID>

To validate edge pipelines, use the following commands:

curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/validate

To manage running edge pipelines - including stopping, resetting the offset, and viewing the pipeline status and metrics - use the following commands:

curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/status
curl -X POST http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/stop
curl -X POST http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/resetOffset
curl -X GET http://<SDCEdge_hostname>:18633/rest/v1/pipeline/<pipelineID>/metrics