Log Files

The Transformer log, $TRANSFORMER_LOG/transformer.log, provides information about the Transformer application, such as start-up messages, user logins, or pipeline display in the canvas. You can open the log file on the Transformer machine, or you can view the contents of the log file from the Transformer UI, as described in Viewing the Transformer Log.

The Transformer log can also include some information about local pipelines or cluster pipelines run on Hadoop YARN in client deployment mode. For these types of pipelines, the Spark driver program is launched on the local Transformer machine. As a result, some pipeline processing messages are included in the Transformer log.

A Spark driver log provides information about how Spark runs, previews, and validates pipelines.

By default, messages in the Spark driver log are logged at the ERROR severity level. To modify the log level, change the Log Level property on the Cluster tab for the pipeline.

You can view and download the Spark driver log from the Transformer UI for the following types of pipelines:

Local pipelines
Cluster pipelines run in Spark standalone mode
Cluster pipelines run on Amazon EMR
Cluster pipelines run on Hadoop YARN in client deployment mode

For local pipelines or cluster pipelines run on Hadoop YARN in client deployment mode, you can also open the Spark driver log file written to the following location on the Transformer machine for each pipeline: $TRANSFORMER_DATA/runInfo/<pipelineID>/run<timestamp>/driver-all.log

For all other cluster pipelines, the Spark driver program is launched remotely on one of the worker nodes inside the cluster. To view the Spark driver logs for these pipelines, access the Spark web UI for the application launched for the pipeline. Transformer provides easy access to the Spark web UI for many cluster types.