Executors
An executor stage triggers a task when it receives an event.
Use executors as part of a dataflow trigger in an event stream to perform event-driven, pipeline-related tasks, such as moving a fully-written file when a destination closes it.
For more information about the event framework, see Dataflow Triggers Overview.
You can use different executors based on the execution mode of the pipeline: standalone or cluster.
Standalone Pipelines Only
Standalone or Cluster Pipelines
In standalone or cluster pipelines, you can use the following executors:
- ADLS Gen1 File Metadata (deprecated) - Changes file metadata, creates an empty file, or removes a file or directory in Azure Data Lake Storage Gen1 upon receiving an event.
- ADLS Gen2 File Metadata - Changes file metadata, creates an empty file, or removes a file or directory in Azure Data Lake Storage Gen2 upon receiving an event.
- Amazon S3 - Creates new Amazon S3 objects for the specified content, copies objects within a bucket, or adds tags to existing Amazon S3 objects.
- Databricks Job Launcher - Starts the specified Databricks job upon receiving an event record.
- Databricks Query - Runs Spark SQL queries on Databricks upon receiving an event.
- Email - Sends custom email to the configured recipients upon receiving an event.
- Google BigQuery - Runs SQL queries on Google BigQuery upon receiving an event.
- Google Cloud Storage - Creates new Google Cloud Storage objects for the specified content, copies or moves objects, or adds metadata to existing Google Cloud Storage objects.
- HDFS File Metadata - Changes file metadata, creates an empty file, or removes a file or directory in HDFS or a local file system upon receiving an event.
- Hive Query - Runs user-defined Hive or Impala queries upon receiving an event record.
- JDBC Query - Runs user-defined SQL queries upon receiving an event record.
- MapR FS File Metadata - Changes file metadata, creates an empty file, or removes a file or directory in MapR FS upon receiving an event.
- Pipeline Finisher - Stops and transitions the pipeline to a Finished state upon receiving an event record.
- SFTP/FTP/FTPS Client - Moves or removes files from an SFTP, FTP, or FTPS server.
- Shell - Executes a shell script upon receiving an event record.
- Snowflake - Loads whole files staged by the Snowflake File Uploader destination to Snowflake tables.
- Spark - Starts a Spark application upon receiving an event record.