Spark Evaluator (deprecated)

Supported pipeline types:

  • Data Collector
The Spark Evaluator performs custom processing within a pipeline based on a Spark application that you develop. For information about supported versions, see Supported Systems and VersionsSupported Systems and Versions in the Data Collector documentation.
Important: This stage is deprecated and may be removed in a future release.

The Spark Evaluator processor is especially useful when you need to perform heavy custom processing within a pipeline, such as image classification.

Complete the following general tasks to use the Spark Evaluator:
  1. Write the Spark application using Java or Scala. Then, package a JAR file containing the application.
  2. Install the application and its dependencies.
  3. Configure the Spark Evaluator processor to submit the Spark application.

    When you configure the processor, you define the name of the custom Spark class that you developed and define the arguments to pass to the init method in the custom Spark class.

Note: The stage libraries that include the Spark Evaluator include all dependencies required for the processor. The Spark Evaluator does not require any installation prerequisites.