JDBC Producer

Supported pipeline types:

  • Data Collector
The JDBC Producer destination uses a JDBC connection to write data to a database table. You can also use the JDBC Producer destination to write change capture data from a Microsoft SQL Server change log. For information about supported versions, see Supported Systems and VersionsSupported Systems and Versions in the Data Collector documentation.
Data Collector provides database-specific destinations, such as SQL Server 2019 BDC Bulk Loader. When available, StreamSets recommends using a database-specific destination.
Important: This stage does not support connecting to non-RDBMS systems, including Hive, Impala, Kudu, or Snowflake. Support for untested systems is not guaranteed. For a list of tested systems, see "Database Vendors and Drivers".

When you configure JDBC Producer, you specify connection information, table name, and optionally define field mappings.

By default, JDBC Producer writes data to the table based on the matching field names. You can override the default field mappings by defining specific mappings. To determine which table rows to update or delete, the destination detects the list of primary key columns for the table, and then uses the fields mapped to those columns to match rows.

You can configure the stage to rollback an entire batch if an error occurs while writing part of the batch. You can also configure custom properties that your driver requires.

The JDBC Producer can use CRUD operations defined in the sdc.operation.type record header attribute to write data. You can define a default operation for records without the header attribute or value. You can also configure whether to use multi-row operations for inserts and deletes, and how to handle records with unsupported operations.

When processing data from a CDC-enabled origin, you can specify the origin change log to aid record processing. For information about Data Collector change data processing and a list of CDC-enabled origins, see Processing Changed Data.

To use a JDBC version older than 4.0, you can specify the driver class name and define a health check query.

You can also use a connectionconnection to configure the destination.

You can use the JDBC Producer as part of the Drift Synchronization Solution for PostgreSQL.