Connecting Stages¶
As described in earlier sections and as shown in the first example, connecting stages together to create the flow of your pipeline is essential to its use.
Output Lanes¶
To connect the output lane of one stage to the input lane of another, simply use the >> operator between two
streamsets.sdk.sdc_models.Stage instances:
dev_raw_data_source >> trash
For stages with multiple output paths, the >> operator can be used multiple times:
file_tail = builder.add_stage('File Tail')
file_tail >> trash_1
file_tail >> trash_2
It is also possible to connect a stage with a single output path to the inputs of multiple stages.
To accomplish this, the >> operator expects that the streamsets.sdk.sdc_models.Stage instances, to which
you’ll be connecting the same output, are put into a list:
trash_1 = builder.add_stage('Trash')
trash_2 = builder.add_stage('Trash')
dev_raw_data_source >> [trash_1, trash_2]
Using the above steps creates a pipeline like the one in the image below:
Event Lanes¶
To connect the event lane of one stage to another, use the >= operator:
dev_data_generator >> trash_1
dev_data_generator >= trash_2
