Meet StreamSets Data Collector EdgeEdge Pipelines Overview
StreamSets Data Collector EdgeTM (SDC Edge) is a lightweight execution agent without a UI that runs pipelines on edge devices with limited resources. Use SDC Edge to read data from an edge device or to receive data from another pipeline and then act on that data to control an edge device.
You design edge pipelines in Data Collector or in the Control Hub. An edge pipeline is a pipeline that runs in edge execution mode on SDC Edge. After designing edge pipelines, you deploy the edge pipelines to SDC Edge installed on an edge device. You run the edge pipelines on SDC Edge.
Edge pipelines work in tandem with standalone execution mode pipelines running on Data Collector.
Edge pipelines are bidirectional - they can both send data to other pipelines and
receive data from other pipelines. To use edge pipelines, you'll work with the following
types of pipelines:
- Edge sending pipeline
- An edge sending pipeline runs on SDC Edge. It uses an origin specific to the edge device to read local data residing on the device. The pipeline can perform minimal processing on the data before sending the data to a Data Collector receiving pipeline.
- Data Collector receiving pipeline
- A Data Collector receiving pipeline runs on Data Collector. It reads data from the edge sending pipeline destination. The Data Collector receiving pipeline performs more complex processing on the data as needed, and then it writes the data to the final destinations.
- Edge receiving pipeline
- An edge receiving pipeline runs on SDC Edge. It listens for data sent by another pipeline running on Data Collector or on SDC Edge and then acts on that data to control the edge device.
Tip: To use edge pipelines, you design and run multiple pipelines
that interact together. You can use StreamSets Control Hub to map multiple dataflows running across multiple instances of Data Collector and
SDC Edge in a single visual topology. For more information, see Meet StreamSets Control Hub.