Amazon Security
When you use an Amazon S3 stage or an EMR or EMR Serverless pipeline, you can configure Transformer to use one of the following authentication methods to connect securely to Amazon Web Services (AWS):
- Instance profile
- When Transformer runs on an Amazon EC2 instance that has an associated instance profile, Transformer uses the instance profile credentials to automatically authenticate with AWS.
- AWS access keys
- When Transformer does not run on an Amazon EC2 instance or when the EC2 instance doesn’t have an instance profile, you can authenticate using an AWS access key pair. When using an AWS access key pair, you specify the access key ID and secret access key to use.
- None
- When accessing a public bucket, you can connect anonymously using no authentication.
Assume Another Role
When using instance profile or AWS access keys authentication, you can configure an Amazon stage or an EMR or EMR Serverless pipeline to assume another IAM role.
When Transformer assumes a role, it temporarily gives up the instance profile or IAM user permissions and uses the permissions assigned to the assumed role. To assume a role, Transformer calls the AWS STS AssumeRole API operation and passes the role to use. The operation creates a new session with the temporary credentials, as long as the following conditions are true:
- The IAM policy attached to the current principal - the IAM role or user - grants permission to assume the specified role.
- The IAM trust policy attached to the role to be assumed permits the current principal to assume it.
Stage Library and Cluster Type Requirements
To assume another role, Transformer must use an Amazon Web Services library for Apache Hadoop 2.8.0 or later.
- Google Dataproc cluster
- Databricks cluster
- Hadoop YARN cluster using Cloudera CDH version 5.x.x
- AWS cluster-provided libraries when the cluster has the Amazon Web Services library for Apache Hadoop 2.8.0 or later installed.
- AWS Transformer-provided libraries for Hadoop 3.2.0
Session Tags to Restrict Role Access
- Assume a role with no restrictions
-
When configured to assume a role with no restrictions, any StreamSets user account that starts the pipeline can assume the role specified in Transformer, as long as the IAM policies attached to the current principal and to the role to be assumed allow it.
For example, any Control Hub user who starts the job for the pipeline can assume the
finance
role when the IAM trust policy attached to thefinance
role allows the role to be assumed by the IAM role or user identified by the selected authentication method. - Assume a role using session tags to restrict role access
- For increased security, you can configure an Amazon S3 stage or an EMR or
EMR Serverless pipeline to assume a role and set session tags to restrict
the Transformer or Control Hub user accounts allowed to assume the role. When configured to set session
tags, the stage or the EMR or EMR Serverless pipeline passes the following
session tag to the AWS STS AssumeRole API operation:
streamsets/principal=<user>
Where
<user>
is the name of the currently logged in Transformer user that starts the pipeline or the Control Hub user that starts the job for the pipeline.AWS IAM verifies that the user account set in the session tag can assume the specified role. The IAM trust policy attached to the role to be assumed must allow the current principal permission to assume the role and must have constraints using IAM condition keys to limit the AssumeRole action based on the requested session tags.
For example, when the Control Hub user Joe starts the job for the pipeline, he can assume the
finance
role when the IAM trust policy attached to thefinance
role allows the userjoe
to assume the role. The Control Hub user Emily cannot assume thefinance
role because the trust policy attached to thefinance
role does not allow the useremily
to assume the role.
To enable Transformer to assume a role, you first must create the trust policy in AWS that allows the role to be assumed. Then, you configure the required stage properties in Control Hub.
Create the Trust Policy
In AWS, create and attach a trust policy to the role to be assumed. The policy must allow other principals - IAM roles or users - to assume the role.
The trust policy that you create for the role to be assumed depends on whether you want to allow Amazon S3 stages or EMR or EMR Serverless pipelines to assume the role with or without restrictions:
- Trust policy to assume the role with no restrictions
- Create and attach a trust policy to the role to be assumed that allows another IAM role or user to assume the role.
- Trust policy to assume a role using session tags to restrict role access
- Create and attach a trust policy to the role to be assumed that allows the IAM role or user to assume the role, uses session tags, and restricts the session tag values to specific StreamSets user accounts.
For more information about creating an IAM trust policy, see the AWS IAM documentation.
Configure Transformer to Assume a Role
After you create and attach a trust policy to the role to be assumed, you can configure Amazon S3 stages or EMR or EMR Serverless pipelines to assume the role.
-
Select AWS Keys or Instance
Profile for the Authentication Method
property on one of the following tabs:
- Amazon S3 tab of the Amazon S3 stage.
- Cluster tab of the pipeline running on an Amazon EMR cluster or an Amazon EMR Serverless application.
- Select Assume Role.
-
Configure the following properties:
Assume Role Property Description Role ARN Amazon resource name (ARN) of the role to assume, entered in the following format:
arn:aws:iam::<account_id>:role/<role_name>
Where
<account_id>
is the ID of your AWS account and<role_name>
is the name of the role to assume. You must create and attach an IAM trust policy to this role that allows the role to be assumed.Role Session Name Optional name for the session created by assuming a role. Overrides the default unique identifier for the session.
Session Timeout Maximum number of seconds for each session created by assuming a role. The session is refreshed if the pipeline continues to run for longer than this amount of time.
Set to a value between 3,600 seconds and 43,200 seconds.
Set Session Tags Sets a session tag to record the name of the currently logged in Data Collector or Transformer user that starts the pipeline or the Control Hub user that starts the job for the pipeline. AWS IAM verifies that the user account set in the session tag can assume the specified role.
Select only when the IAM trust policy attached to the role to be assumed uses session tags and restricts the session tag values to specific user accounts.
When cleared, the connection does not set a session tag.