Creating a pipeline¶
Once the authentication step has been handled and you’ve successfully instantiated a streamsets.sdk.DataCollector
object, you’re now ready to build a pipeline.
Instantiating a Pipeline Builder¶
The first step of creating a pipeline is to instantiate a streamsets.sdk.sdc_models.PipelineBuilder instance.
This class handles the majority of the pipeline configuration on your behalf by building the initial JSON representation
of the pipeline, and configuring default values for essential properties (instead of requiring each to be set manually).
The streamsets.sdk.sdc_models.PipelineBuilder instance can be created as follows:
pipeline_builder = sdc.get_pipeline_builder()
Adding Stages to the Pipeline Builder¶
Now that the builder has been instantiated, you can get streamsets.sdk.sdc_models.Stage instances from this
builder for use in the pipeline you’re creating. Adding stages to the pipeline can be done by calling
streamsets.sdk.sdc_models.PipelineBuilder.add_stage(). See the API reference for this method for details on the
arguments it takes.
As shown in the first example, the simplest type of pipeline directs one origin into one
destination. For this example, you can do this with Dev Raw Data Source origin and Trash destination,
respectively:
dev_raw_data_source = pipeline_builder.add_stage('Dev Raw Data Source')
trash = pipeline_builder.add_stage('Trash')
Connecting the Stages¶
With streamsets.sdk.sdc_models.Stage instances in hand, you can connect them by using the >> operator.
Once the stages are connected, you can build the streamsets.sdk.sdc_models.Pipeline instance with the
streamsets.sdk.sdc_models.PipelineBuilder.build() method:
dev_raw_data_source >> trash
pipeline = pipeline_builder.build('My first pipeline')
Add the Pipeline to Data Collector¶
Finally, to add this pipeline to your Data Collector instance, pass it to the
streamsets.sdk.DataCollector.add_pipeline() method:
sdc.add_pipeline(pipeline)