Multithreaded Processing Modes
The JDBC Multitable Consumer origin performs parallel processing and enables the creation of a multithreaded pipeline. The origin can use multiple threads to process entire tables or partitions within tables.
By default, the origin performs multithreaded partition processing for the tables that fulfill the partition processing requirements, while performing multithreaded table processing for all other tables. When using the default behavior, the origin notes the tables that allow partition processing in the Data Collector log. When needed, you can configure the origin to require partition processing for all tables or to perform only table processing. You can also allow the single-threaded non-incremental processing of tables when needed.
- Multithreaded table processing - The origin can use up to one thread per table. Can process tables with multiple offset columns.
- Multithreaded partition processing - The origin can use up to one
thread per table partition. Use to process larger volumes of data than
multithreaded table processing.
Multithreaded partition processing requires a single primary key or user-defined offset column of a supported data type, and additional details for partition creation. Tables with composite keys or a key or user-defined offset column of an unsupported data type cannot be partitioned.
- Off - Use to perform multithreaded table processing.
Can be used to perform non-incremental loads of tables without key or offset columns, when enabled.
- On (Best Effort) - Use to perform partition processing where possible and allow
multithreaded table processing for tables with multiple key or offset columns.
Can be used to perform non-incremental loads of tables without key or offset columns, when enabled.
- On (Required) - Use to perform partition processing for all specified tables.
Does not allow performing other types of processing for tables that do not meet the partition processing requirements.