Design Data Collector Receiving Pipelines

Data Collector receiving pipelines run in the standalone execution mode. You design and run receiving pipelines in Data Collector.

A Data Collector receiving pipeline reads data from the edge sending pipeline destination. The Data Collector receiving pipeline performs more complex processing on the data as needed, and then it writes the data to the final destinations.

Optionally, you can also design a Data Collector receiving pipeline to monitor the data being processed and then send data to an edge receiving pipeline running on SDC Edge. The edge receiving pipeline acts on the data to control the edge device.

A Data Collector receiving pipeline includes the corresponding origin to read from the destination in the edge sending pipeline. For example, if the edge sending pipeline writes to a CoAP Client destination, then the Data Collector receiving pipeline uses a CoAP Server origin to read the data.

You can use the following origins in a Data Collector receiving pipeline:
  • Amazon S3
  • Azure IoT/Event Hub Consumer
  • CoAP Server
  • HTTP Server
  • Kafka Consumer or Kafka Multitopic Consumer
  • Kinesis Consumer
  • MQTT Subscriber
  • WebSocket Server

The MQTT stages require using an intermediary MQTT broker. For example, the edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the Data Collector receiving pipeline reads the data.

The remaining stages can directly communicate with each other. They do not require an intermediary storage location.

Data Collector receiving pipelines can include any type of processor, destination, and executor.