In practice, it’s rare to have stages in your pipeline that haven’t had some configurations
changed from their default values. When using the SDK, the names to use when referring
to these configuration properties can generally be inferred from the StreamSets Data Collector UI (e.g.
Data Format becomes
data_format), but they can also be directly inspected in a Python
interpreter using the
dir() built-in function on an instance of the
or by using Python’s built-in
With the attribute name in hand, you can read the value of the configuration:
As for setting the value of the configuration, this can be done in one of two ways depending on your use case:
If you only have one or two configurations to update, you can set them using attributes of the
streamsets.sdk.sdc_models.Stage instance. Continuing in the vein of our example above:
dev_raw_data_source.data_format = 'TEXT' dev_raw_data_source.raw_data = 'hi\nhello\nhow are you?'