Cluster and Spark URLs
In monitor mode, the Monitoring panel provides URLs for the cluster or the Spark application that runs the pipeline.
Use the URL to access additional information about the cluster or Spark application. For example, the Spark web UI can include information such as completed jobs, memory usage, running executors, and the Spark driver log.
Cluster and Spark URLs display in the Runtime Statistics section of the Monitoring panel. For example, when you monitor a Databricks pipeline, the Databricks Job URL displays with the other runtime statistics, as follows:
The following table lists the URLs that display for each
cluster manager type:
| Cluster Manager Type | URL |
|---|---|
| Amazon EMR |
|
| Apache Spark for HD Insight |
|
| Databricks |
|
| Dataproc |
|
| Hadoop YARN |
|
| Spark Standalone |
|
| SQL Server 2019 Big Data Cluster |
|