Stage Library and Cluster Type Requirements
To assume another role, the Amazon S3 stage must use an Amazon Web Services library for Apache Hadoop 2.8.0 or later.
Some cluster types use an earlier version of the library. As a result, Amazon S3 stages
cannot assume another role when included in pipelines that run on the following cluster
types:
- Google Dataproc cluster
- Databricks cluster
- Hadoop YARN cluster using Cloudera CDH version 5.x.x
Amazon S3 stages can assume another role for the remaining cluster types when you
configure the stage to use one of the following stage libraries:
- AWS cluster-provided libraries when the cluster has the Amazon Web Services library for Apache Hadoop 2.8.0 or later installed.
- AWS Transformer-provided libraries for Hadoop 3.2.0