Preview and test run

Previewing a pipeline

To preview a Control Hub pipeline, you can use the streamsets.sdk.ControlHub.run_pipeline_preview() method which will return the command’s output. This provides a list of streamsets.sdk.sdc_models.Record objects that can be further introspected upon.

preview_command = sch.run_pipeline_preview(pipeline)
preview = preview_command.preview
preview[dev_data_generator].output

Output:

[<Record (field={'': '5e9c7f3f-b553-4604-b92b-770fc016cd70'})>,
 <Record (field={'': 'af9eeb58-ff4e-4558-923b-29fbbf94ae8d'})>,
 <Record (field={'': '2b83fada-1eff-45af-bf5c-4c811c5bcc89'})>,
 <Record (field={'': 'aa8c5944-4b95-4aec-9fe3-536d8f0d05f4'})>,
 <Record (field={'': 'a37e6d42-87ab-4736-8a72-107210e05267'})>,
 <Record (field={'': 'dfc4f1f5-854c-4c9e-8324-21505412f4f0'})>,
 <Record (field={'': 'cfd42fc9-4399-44ec-8caf-f2e1d31cd36e'})>,
 <Record (field={'': 'd03ff7c6-0a70-438d-aaeb-01736bddaf52'})>,
 <Record (field={'': 'a4077ac0-4a38-4ef8-8914-ac7f34321ecd'})>,
 <Record (field={'': '708067df-f6ea-4fdb-bfb6-e74a1c002bfc'})>]

Test running a pipeline

To test-run a Control Hub pipeline, use the streamsets.sdk.ControlHub.test_pipeline_run() method. This will execute the pipeline on its Authoring Data Collector instance:

pipeline = sch.pipelines.get(name='Test pipeline')
test_run_command = sch.test_pipeline_run(pipeline)
test_run_command.wait_for_status('RUNNING')
test_run_command.executor_pipeline
test_run_command.executor_instance
sch.stop_test_pipeline_run(test_run_command)

Output:

# test_run_command.executor_pipeline
<Pipeline (id=testRun__48d74ed5-af3c-4196-8696-fba1c6e38673__admin, title=Test pipeline)>

# test_run_command.executor_instance
<streamsets.sdk.sdc.DataCollector at 0x10e872fd0>

# sch.stop_test_pipeline_run(test_run_command)
<sdk.sdc_api.StopPipelineCommand at 0x10e4cd050>