SDC RPC Pipeline Overview (deprecated)

Data Collector Remote Protocol Call pipelines, called SDC RPC pipelines, are a set of StreamSets pipelines that pass data from one pipeline to another without writing to an intermediary system.

Important: This functionality is deprecated and may be removed in a future release. You might replace SDC RPC pipelines with pipelines that use the HTTP Server origin to write to the HTTP Client destination.

SDC RPC pipelines can write to each other on the same machine or over a local network or the public internet. You might use SDC RPC pipelines to send data securely between two data centers.

Typically, a pipeline uses a standard origin such as Directory and writes to a standard destination such as HBase. An SDC RPC pipeline includes an SDC RPC destination or an SDC RPC origin to communicate with another SDC RPC pipeline.

To use SDC RPC pipelines, you create an origin pipeline and a destination pipeline. The origin pipeline uses an SDC RPC destination to write directly to an SDC RPC origin in the destination pipeline, as follows:

The SDC RPC destination and SDC RPC origin enable you to pass data securely from one pipeline to another, effectively like creating a single pipeline that spans a network.