Microservice Pipelines

A microservice pipeline is a pipeline that creates a fine-grained service to perform a specific task.

For example, you might use a microservice pipeline to validate changes to user accounts. You can have the pipeline check for matching accounts to validate account creation requests, account updates, and account deletions. The pipeline can tag requests as actionable or not and send all responses back to the originating system for next steps.

You can create REST-based and message-based microservice pipelines that use standalone execution mode. Other microservice types and platforms are planned for future releases.

A microservice pipeline uses a microservice origin to listen for or read JSON-formatted requests. In the pipeline, you can use any available processor to transform and route the records as needed. You can also use destinations and executors to write data and execute additional actions as needed.

Then, use one or more microservice destinations to send responses back to the origin and write the records to a destination system if necessary. The origin then transmits JSON-formatted responses back to the origin system.

For example, you might use the WebSocket Server origin to listen at a WebSocket endpoint and process all authorized requests. In the pipeline, you use stages to process the data and route records to different destinations based on whether the record is considered a success record or bad request record. You use two Send Response to Origin destinations to return different responses, 200 for OK and 400 for Bad Request. The origin then passes the records with the responses back to the WebSocket endpoint. Or, you might use a different microservice destination to write records to a destination system while sending responses back to the origin from the destination system.

For more information and a descriptive video, see our blog post on microservice pipelines. For a full description of a sample microservice pipeline and a tutorial on creating one, see our GitHub tutorial, "Creating a CRUD Microservice Pipeline."

When you configure a microservice pipeline, you use a microservice origin, any available Data Collector processor, destination, and executor, and one or more microservice destination.

To start with a sample microservice pipeline, create a Data Collector pipeline from a sample pipeline and then select the Sample Microservice Pipeline. You can edit the pipeline to suit your needs. Or, you can create a blank pipeline and use the microservice stages in a clean canvas.

The sample pipeline shows how you might use the REST Service origin and several Send Response to Origin destinations in a microservice pipeline:

The sample pipeline uses an HTTP Router processor to route data to different streams based on the REST API request method and path parameter, performs additional processing, and then passes the records to the Send Response to Origin destinations. The destinations pass the records back to the REST Service origin with the specified status codes, in this case, 200 for OK. The pipeline error handling also passes records back to the origin, with the 400 Bad Request status code. The REST Service origin generates a response with the records it receives and sends it to the originating client.

The pipeline description contains instructions on interacting with it using the curl command line HTTP client. For a step-by-step explanation of the sample pipeline, see "Creating a Microservice Pipeline" in the Creating a CRUD Microservice Pipeline tutorial.