Accessing Databricks Job Details

When you run a Databricks pipeline, Transformer submits an ephemeral job to the Databricks cluster. An ephemeral job is one that runs only once and does not count towards the Databricks job limit. However, job details do not display in the Databricks job menu.

Databricks retains details for ephemeral jobs for 60 days. Use one of the following methods to access details about a Databricks job:

  • Use the Databricks Job link that displays in the Runtime Statistics details when you monitor the pipeline:

  • Use the Databricks Job URL link that displays on the Realtime Summary tab as the job starts:

  • Use the jobs/runs/get Databricks API to check the run state of the workloads.