Databricks Delta Lake

Available when using an authoring Data Collector version 4.0.0 or later.

To create a Databricks Delta Lake connection, version 1.2.0 or later of the Databricks Enterprise stage library, streamsets-datacollector-databricks-lib, must be installed on the selected authoring Data Collector.

For a description of the Databricks Delta Lake connection properties, see Databricks Delta Lake Connection Properties.

After you create a Databricks Delta Lake connection, you can use the connection in the following stage:
Engine Stage
Data Collector 4.0.0 or later Databricks Delta Lake destination

Databricks Delta Lake Connection Properties

When creating a Databricks Delta Lake connection, configure the following properties on the Databricks Delta Lake tab:
Databricks Delta Lake Property Description
JDBC URL JDBC URL used to connect to the Databricks cluster.

For example: jdbc:spark://<server_hostname>:443/default;transportMode=http :ssl=1;httpPath=sql/protocolv1/o/0/xxxx-xxxxxx-xxxxxxxx;AuthMech=3;

Tip: In Databricks, you can locate the JDBC URL for your cluster on the JDBC/ODBC tab in the cluster configuration details. As a best practice, remove the PWD parameter from the URL, and then enter the personal access token value in the Token property below.
Token Personal access token used to connect to the Databricks cluster.
Tip: To secure sensitive information, you can use credential stores or runtime resources.