Executors

An executor stage triggers a task when it receives an event.

Use executors as part of a dataflow trigger in an event stream to perform event-driven, pipeline-related tasks, such as moving a fully-written file when a destination closes it.

For more information about the event framework, see Dataflow Triggers Overview.

You can use different executors based on the execution mode of the pipeline: standalone or cluster.

Standalone Pipelines Only

In standalone pipelines, you can use the following executors:
  • MapReduce - Starts the specified MapReduce job upon receiving an event record.
  • Spark - Starts the specified Spark application upon receiving an event record.

Standalone or Cluster Pipelines

In standalone or cluster pipelines, you can use the following executors:
  • Amazon S3 - Creates new Amazon S3 objects for the specified content, copies objects within a bucket, or adds tags to existing Amazon S3 objects.
  • ADLS Gen1 File Metadata (deprecated) - Changes file metadata, creates an empty file, or removes a file or directory in Azure Data Lake Storage Gen1 upon receiving an event.
  • ADLS Gen2 File Metadata - Changes file metadata, creates an empty file, or removes a file or directory in Azure Data Lake Storage Gen2 upon receiving an event.
  • Databricks Job Launcher - Starts the specified Databricks job upon receiving an event record.
  • Databricks Query - Runs Spark SQL queries on Databricks upon receiving an event.
  • Email - Sends custom email to the configured recipients upon receiving an event.
  • Google Cloud Storage - Creates new Google Cloud Storage objects for the specified content, copies or moves objects, or adds metadata to existing Google Cloud Storage objects.
  • HDFS File Metadata - Changes file metadata, creates an empty file, or removes a file or directory in HDFS or a local file system upon receiving an event.
  • Hive Query - Runs user-defined Hive or Impala queries upon receiving an event record.
  • JDBC Query - Runs user-defined SQL queries upon receiving an event record.
  • MapR FS File Metadata - Changes file metadata, creates an empty file, or removes a file or directory in MapR FS upon receiving an event.
  • Pipeline Finisher - Stops and transitions the pipeline to a Finished state upon receiving an event record.
  • SFTP/FTP/FTPS Client - Moves or removes files from an SFTP, FTP, or FTPS server.
  • Shell - Executes a shell script upon receiving an event record.
  • Snowflake - Loads whole files staged by the Snowflake File Uploader destination to Snowflake tables.
  • Spark - Starts a Spark application upon receiving an event record.