Amazon Security

When you use an Amazon S3 stage or an Amazon EMR or EMR Serverless pipeline, you can configure Transformer to use one of the following authentication methods to connect securely to Amazon Web Services (AWS):

Instance profile
When Transformer runs on an Amazon EC2 instance that has an associated instance profile, Transformer uses the instance profile credentials to automatically authenticate with AWS.
For more information about associating an instance profile with an EC2 instance, see the Amazon EC2 documentation.
AWS access keys
When Transformer does not run on an Amazon EC2 instance or when the EC2 instance doesn’t have an instance profile, you can authenticate using an AWS access key pair. When using an AWS access key pair, you specify the access key ID and secret access key to use.
Tip: To secure sensitive information, you can use credential stores or runtime resources.
None
When accessing a public bucket, you can connect anonymously using no authentication.

Assume Another Role

When using instance profile or AWS access keys authentication, you can configure an Amazon S3 stage or an Amazon EMR or EMR Serverless pipeline to assume another IAM role.

For example, if the instance profile or the IAM user permissions do not grant access to write to Amazon S3 resources, you can configure the Amazon S3 origin to assume another role that does grant write access.
Important: Transformer supports assuming another role when the pipeline meets the stage library and cluster type requirements.

When Transformer assumes a role, it temporarily gives up the instance profile or IAM user permissions and uses the permissions assigned to the assumed role. To assume a role, Transformer calls the AWS STS AssumeRole API operation and passes the role to use. The operation creates a new session with the temporary credentials, as long as the following conditions are true:

  • The IAM policy attached to the current principal - the IAM role or user - grants permission to assume the specified role.
  • The IAM trust policy attached to the role to be assumed permits the current principal to assume it.

Stage Library and Cluster Type Requirements

To assume another role, Transformer must use an Amazon Web Services library for Apache Hadoop 2.8.0 or later.

Some cluster types use an earlier version of the library. As a result, Amazon S3 stages cannot assume another role when included in pipelines that run on the following cluster types:
Amazon S3 stages can assume another role for the remaining cluster types when you configure the stage to use one of the following stage libraries:
  • AWS cluster-provided libraries when the cluster has the Amazon Web Services library for Apache Hadoop 2.8.0 or later installed.
  • AWS Transformer-provided libraries for Hadoop 3.2.0

Assume Role Methods

You can configure an Amazon S3 stage or an Amazon EMR or EMR Serverless pipeline to assume a role in the following ways:
Assume a role with no restrictions

When configured to assume a role with no restrictions, any StreamSets user account that starts the pipeline can assume the role specified stage or pipeline cluster properties, as long as the IAM policies attached to the current principal and to the role to be assumed allow it.

For example, any StreamSets user who starts the job for the pipeline can assume the finance role when the IAM trust policy attached to the finance role allows the role to be assumed by the IAM role or user identified by the selected authentication method.

Assume a role using an external ID condition
You can use an external ID condition when you configure an Amazon S3 stage or an Amazon EMR or EMR Serverless pipeline.
When you use an external ID condition, any StreamSets user account that starts the pipeline can assume the role specified in the stage or pipeline cluster properties when the IAM policies attached to the current principal and to the role to be assumed allow it. However, the IAM policies attached to the role to be assumed must include an external ID condition, and the specified external ID must be defined in the stage or pipeline.
When configured to use an external ID, the connection passes the following condition to the AWS STS AssumeRole API operation:
"Condition": {"StringEquals": {"sts:ExternalId": "<external id>"}}
Where the <external id> is a unique ID specified in the External ID property of the stage or pipeline and is defined in the IAM trust policy attached to the role to be assumed.
AWS IAM verifies that the StreamSets user account that starts the pipeline can assume the role specified in the stage or pipeline, and that the external ID specified in the stage or pipeline matches the external ID in the IAM trust policy attached to the role to be assumed.
For example, any StreamSets user who starts the job for the pipeline can assume the finance role when both of the following are true:
  • The IAM trust policy attached to the finance role allows the role to be assumed by the IAM role or user identified by the selected authentication method.
  • The IAM trust policy attached to the finance role includes an external ID, such as finance-235A9df84iK, and the connection used in the pipeline includes the same external ID.
For more information about using an external ID, see the AWS documentation.
Tip: To provide additional security, use session tags with an external ID condition.
Assume a role using session tags to restrict role access
For increased security, you can configure an Amazon S3 stage or an Amazon EMR or EMR Serverless pipeline to assume a role and set session tags to restrict the user accounts allowed to assume the role. When configured to set session tags, the stage or pipeline passes the following session tag to the AWS STS AssumeRole API operation:

streamsets/principal=<user>

Where <user> is the name of the currently logged in StreamSets user that starts the pipeline or job for the pipeline.

AWS IAM verifies that the user account set in the session tag can assume the specified role. The IAM trust policy attached to the role to be assumed must allow the current principal permission to assume the role and must have constraints using IAM condition keys to limit the AssumeRole action based on the requested session tags.

For example, when the StreamSets user Joe starts the job for the pipeline, he can assume the finance role when the IAM trust policy attached to the finance role allows the user joe to assume the role. The StreamSets user Emily cannot assume the finance role because the trust policy attached to the finance role does not allow the user emily to assume the role.

To enable Transformer to assume a role, you first must create the trust policy in AWS that allows the role to be assumed. Then, you configure the required cluster or stage properties in Control Hub.

Create the Trust Policy

In AWS, you must create and attach a trust policy to the role to be assumed. The policy must allow other principals - IAM roles or users - to assume the role.

Important: You must also attach a policy to the principal that grants permission to the principal to assume another role. AWS IAM provides several methods of granting a principal access to assume another role. For details, see the AWS IAM documentation.
The trust policy that you create for the role to be assumed depends on whether you want to allow Amazon S3 stages or EMR or EMR Serverless pipelines to assume the role with or without restrictions:
Trust policy to assume the role with no restrictions
Create and attach a trust policy to the role to be assumed that allows another IAM role or user to assume the role.
For example, if using instance profile authentication, you might create the following policy where:
  • <account_id> is the ID of your AWS account.
  • <role_name> is the name of the role permitted to assume this role. Enter the name of the role included in the instance profile associated with the Amazon EC2 instance where the StreamSets engine runs.
{
 "Version": "2022-10-17",
 "Statement": [
   {
     "Sid": "",
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::<account_id>:role/<role_name>"
     },
     "Action": [
       "sts:AssumeRole"
     ]
   }
 ]
}
If using AWS access keys authentication, create a similar trust policy. However, for the principal, specify the ARN of the IAM user permitted to assume this role. Enter the name of the IAM user that owns the access keys used to authenticate with AWS. For example:
...
"Principal": {
       "AWS": "arn:aws:iam::<account_id>:user/<user_name>"
},
...
Trust policy to assume a role with an external ID condition
Create and attach a trust policy to the role to be assumed that allows the IAM role or user to assume the role and that includes an external ID condition.
For example, if using instance profile authentication, you might create the following policy where:
  • <account_id> is the ID of your AWS account.
  • <role_name> is the name of the role permitted to assume this role. Enter the name of the role included in the instance profile associated with the Amazon EC2 instance where Transformer runs.
  • <external_id> is the external ID that must be included in the stage or pipeline cluster properties.
{
 "Version": "2022-10-17",
 "Statement": [
   {
     "Sid": "",
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::<account_id>:role/<role_name>"
     },
     "Action": [
       "sts:AssumeRole"
     ],
     "Condition": {
       "StringEquals": {
         "sts:ExternalId": "<external_id>"
        }
     }
   }
 ]
}
If using AWS access keys authentication, create a similar trust policy.
For more information about using an external ID, see the AWS documentation.
Trust policy to assume a role using session tags to restrict role access
Create and attach a trust policy to the role to be assumed that allows the IAM role or user to assume the role, uses session tags, and restricts the session tag values to specific StreamSets user accounts.
For example, if using instance profile authentication, you might create the following policy where:
  • <account_id> is the ID of your AWS account.
  • <role_name> is the name of the role permitted to assume this role. Enter the name of the role included in the instance profile associated with the Amazon EC2 instance where the StreamSets engine runs.
  • <user1> and <user2> are StreamSets user accounts allowed to assume this role. To specify a Control Hub user account, use the required naming convention: <user ID>@<organization ID>. For example, joe@MyCompany.
{
 "Version": "2022-10-17",
 "Statement": [
   {
     "Sid": "",
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::<account_id>:role/<role_name>"
     },
     "Action": [
       "sts:AssumeRole",
       "sts:TagSession"
     ],
     "Condition": {
       "StringEquals": {
         "aws:RequestTag/streamsets/principal": ["<user1>", "<user2>"]
       },
        "Null": {
          "aws:RequestTag/streamsets/principal": "false"
        }
     }
   }
 ]
}
If using AWS access keys authentication, create a similar trust policy. However, for the principal, specify the ARN of the IAM user permitted to assume this role. Enter the name of the IAM user that owns the access keys used to authenticate with AWS. For example:
...
"Principal": {
       "AWS": "arn:aws:iam::<account_id>:user/<user_name>"
},
...

For more information about creating an IAM trust policy, see the AWS IAM documentation.

Configure Transformer to Assume a Role

After you create and attach a trust policy to the role to be assumed, you can configure an Amazon S3 stage or an Amazon EMR or EMR Serverless pipeline to assume the role.

  1. Select AWS Keys or Instance Profile for the Authentication Method property on one of the following tabs:
    • Amazon S3 tab of the Amazon S3 stage.
    • Cluster tab of the pipeline running on an Amazon EMR cluster or an Amazon EMR Serverless application.
  2. Select Assume Role.
  3. Configure the following properties:
    Assume Role Property Description
    Role ARN

    Amazon resource name (ARN) of the role to assume, entered in the following format:

    arn:aws:iam::<account_id>:role/<role_name>

    Where <account_id> is the ID of your AWS account and <role_name> is the name of the role to assume. You must create and attach an IAM trust policy to this role that allows the role to be assumed.
    Role Session Name

    Optional name for the session created by assuming a role. Overrides the default unique identifier for the session.
    Session Timeout

    Maximum number of seconds for each session created by assuming a role. The session is refreshed if the pipeline continues to run for longer than this amount of time.

    Set to a value between 3,600 seconds and 43,200 seconds.

    Set Session Tags

    Sets a session tag to record the name of the currently logged in StreamSets user that starts the pipeline or the job for the pipeline. AWS IAM verifies that the user account set in the session tag can assume the specified role.

    Select only when the IAM trust policy attached to the role to be assumed uses session tags and restricts the session tag values to specific user accounts.

    When cleared, the connection does not set a session tag.
    External ID External ID included in an IAM trust policy that allows the specified role to be assumed.

    Available when assuming another role.