System Metrics

Supported pipeline types:

  • Data Collector Edge
The System Metrics origin reads system metrics from the edge device where StreamSets Data Collector Edge (SDC Edge) is installed. Use the System Metrics origin only in pipelines configured for edge execution mode.

The System Metrics origin reads the metrics from the edge device at regular intervals, based on the amount of delay time between batches that you configure. For example, if you set the delay time to 10 minutes, then the origin creates a new batch containing the selected system metrics every 10 minutes.

Each batch contains a single record that includes the timestamp when the data was read and a map field for each selected system metric type. When you configure the origin, you select the types of system metrics to read - including host information and CPU, memory, disk, network, and process metrics.

For more information about installing SDC Edge, designing edge pipelines, and running and maintaining edge pipelines, see Meet StreamSets Data Collector EdgeEdge Pipelines Overview.

Example

You want to collect, monitor, and analyze the system metrics of all of your edge devices.

You install SDC Edge on each edge device. You use Data Collector to design an edge sending pipeline that includes the System Metrics origin and an HTTP Client destination that posts the system metrics to an HTTP endpoint. You deploy the edge sending pipeline to all of the edge devices and then run the pipeline on each device.

You design a Data Collector receiving pipeline that includes an HTTP Server origin that reads the system metrics posted to the HTTP endpoint. After reading the metrics, the Data Collector receiving pipeline performs additional processing on the data and then writes the data to Elasticsearch for analysis of the metrics. You run the Data Collector receiving pipeline on Data Collector.