Configuring stages

In practice, it’s rare to have stages in your pipeline that haven’t had some configurations changed from their default values. When using the SDK, the names to use when referring to these configuration properties can generally be inferred from the StreamSets Data Collector UI (e.g. Data Format becomes data_format), but they can also be directly inspected in a Python interpreter using the dir() built-in function on an instance of the streamsets.sdk.sdc_models.Stage class:

dir(dev_raw_data_source)

or by using Python’s built-in help() function:

help(dev_raw_data_source)

With the attribute name in hand, you can read the value of the configuration:

dev_raw_data_source.max_line_length

Output:

1024

As for setting the value of the configuration, this can be done in one of two ways depending on your use case:

Single configurations

If you only have one or two configurations to update, you can set them using attributes of the streamsets.sdk.sdc_models.Stage instance. Continuing in the vein of our example above:

dev_raw_data_source.data_format = 'TEXT'
dev_raw_data_source.raw_data = 'hi\nhello\nhow are you?'

Multiple configurations

For readability, it’s sometimes better to set all attributes simultaneously with one call to the streamsets.sdk.sdc_models.Stage.set_attributes() method:

dev_raw_data_source.set_attributes(data_format='TEXT', raw_data='hi\nhello\nhow are you?')