AWS Environments

An Amazon Web Services (AWS) environment represents the Amazon virtual private cloud (VPC) in your AWS account where engines are deployed.

Your AWS administrator must create a VPC in your AWS account and configure AWS credentials for Control Hub to use. You then create an AWS environment in Control Hub that represents the VPC. When you activate the environment, Control Hub connects to the VPC using the configured credentials, provisions the AWS resources needed to run engines, and deploys engine instances to those resources.

While the environment is in an active state, Control Hub periodically verifies that the Amazon VPC exists and that the credentials are valid. Control Hub does not provision resources in the VPC until you create and start a deployment for this environment.

Before you create an AWS environment, your AWS administrator must complete several prerequisites.

Prerequisites

The prerequisites require logging into StreamSets DataOps Platform to retrieve information generated for the organization.

You first must invite your AWS administrator to join your StreamSets organization. You can invite the administrator using the default role assignments, or you can modify the role assignments to grant the administrator the Environment Manager role only.

After joining the StreamSets organization, your AWS administrator must complete the following prerequisites:
  1. Create an Amazon VPC for the StreamSets AWS environment to use.
  2. Configure instance profiles to associate with the provisioned EC2 instances.
  3. Configure the AWS credentials that Control Hub uses to access and provision resources in your Amazon VPC.

Create an Amazon VPC

Create an Amazon virtual private cloud (VPC) in your AWS account.

You can use an existing VPC. However, StreamSets recommends creating a new VPC for the exclusive use of each StreamSets AWS environment.

You can use private or public subnets within the VPC network, as long as the subnets can send outbound traffic to the internet.

For instructions on creating a VPC and on allowing subnets internet access, see the Amazon VPC documentation.

Security Group

Assign a security group to the VPC that defines the required inbound and outbound rules. You can use an existing security group or create a new group.

Allow the following traffic:
  • Inbound and outbound connections required by StreamSets engines, as described in Firewall Configuration.
  • Outbound connections to AWS Systems Manager. Add the IP address of the ssm.<region>.amazonaws.com host as an allowed destination.

    For the list of AWS IP addresses, see the AWS documentation.

Important: If your VPC uses network access control lists (ACLs) to control inbound and outbound traffic at the subnet level, you must define the same inbound and outbound rules for the network ACLs. StreamSets AWS environments do not require that you set up network ACLs.

Configure Instance Profiles for EC2 Instances

Configure instance profiles for EC2 instances in your AWS account. When Control Hub provisions EC2 instances for an Amazon EC2 deployment belonging to this environment, it associates these instance profiles with the EC2 instances.

Important: Configuring instance profiles requires logging into StreamSets DataOps Platform to retrieve information generated for the organization. If you have not yet joined the StreamSets organization, ask your organization administrator to invite you.
You can configure the instance profile used by a deployment in the following ways:
Configure a default instance profile for the environment
Configure a default instance profile for the parent AWS environment. When you create an Amazon EC2 deployment for this environment, you can simply use the default instance profile configured for the environment.
Configure a unique instance profile for each deployment
Do not configure a default instance profile for the parent AWS environment. When you create an Amazon EC2 deployment for this environment, you must configure the instance profile to use for the deployment.
Configure a default instance profile and override as needed
Configure a default instance profile for the parent AWS environment. When you create an Amazon EC2 deployment for this environment, you can use the default instance profile configured for the environment, or you can override the default and configure a different instance profile for the deployment to use.

Regardless of how you configure the instance profile for the EC2 instances, you must create the IAM policy and attach it to an instance profile as an AWS environment prerequisite. When you create the IAM policy that you attach to the AWS credentials that Control Hub uses to access your AWS account, you must include the Amazon resource name (ARN) of all instance profiles in that policy. This grants Control Hub the ability to associate the instance profiles with the EC2 instances.

Complete the following steps to configure instance profiles for provisioned EC2 instances:
  1. Create an IAM policy that grants limited access to the provisioned EC2 instances.
  2. Use the IAM policy with an instance profile.

Create an IAM Policy

Create an IAM policy in AWS that grants limited access to the provisioned EC2 instances. Use the sample policy that StreamSets provides. You can make compatible changes to the policy as needed.

Note: AWS provides multiple methods for creating IAM policies. These steps provide brief instructions to create a new policy using the AWS Management Console. For instructions on using other methods, such as using the AWS CLI, see the AWS IAM documentation.
  1. Retrieve the sample IAM policy provided by StreamSets. Alternatively, you can copy the contents of the sample policy below, but be aware that this can introduce formatting errors.
    1. Log into StreamSets DataOps Platform.
    2. In the Control Hub Navigation panel, click Set Up > Environments and then click the Create Environment icon: .
    3. Enter a name for the environment, and then select Amazon Web Services (AWS) for the Environment Type.
    4. Click Save & Next.
    5. Copy or download the sample IAM Policy for Profile.

      You can leave this page open in the browser, cancel the environment creation, or click Save & Exit to save an incomplete environment that you finish configuring after completing the prerequisites.

  2. In the AWS Console, go to the IAM service.
  3. In the navigation pane, click Policies, and then click Create policy.
  4. In the policy editor, click the JSON tab and paste the contents of the sample policy that you retrieved from Control Hub.

    The contents of the sample IAM policy for instance profiles are as follows:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "0",
            "Effect": "Allow",
            "Action": "ssm:GetParameter",
            "Resource": "arn:aws:ssm:*:<ACCOUNT_ID>:parameter/StreamSets-Deployment-Token-*"
        }
    ]
}
  5. Replace the <ACCOUNT_ID> parameter in the sample policy with your 12-digit AWS account ID.

    You can make compatible changes to the policy as needed.

  6. Click Next: Tags.
  7. Click Review policy.
  8. On the Review page, enter a policy name.

    For example, to designate that this policy is for the instance profiles used by StreamSets, you might enter the name StreamSets-Instance-Profile-Policy.

  9. Click Create policy.

Use the Policy with an Instance Profile

In AWS, create an instance profile to associate with the provisioned EC2 instances. You can create a single default instance profile to use for all deployments belonging to the parent environment, or you can create a unique instance profile for each deployment.

To create the instance profile, use the AWS Management Console to create a role for Amazon EC2 and attach the IAM policy that you created to this role. When you use the AWS Management Console to create a role for Amazon EC2, the console automatically creates an instance profile and gives it the same name as the role.
Note: AWS provides multiple methods for creating IAM roles. These steps provide brief instructions to create a new role using the AWS Management Console. For instructions on using other methods, such as using the AWS CLI, see the AWS IAM documentation.
  1. In the AWS Console, go to the IAM service.
  2. In the navigation pane, click Roles, and then click Create role.
  3. Select AWS service as the type of trusted entity, and then choose EC2 as the use case.
  4. Click Next: Permissions.
  5. Search for and then select the policy that you created for StreamSets instance profiles.

    For example, select StreamSets-Instance-Profile-Policy.

  6. Attach any additional policies required by the StreamSets engines running on the EC2 instances.

    For example, if your pipelines process Amazon S3 data, attach an S3 policy to the role as well.

  7. Click Next: Tags.
  8. Click Next: Review.
  9. On the Review page, enter a role name.

    For example, to designate that this role is for StreamSets EC2 deployments, you might enter the name StreamSets-Deployment-EC2-Instance-Profile.

  10. Click Create role.
  11. Click the name of the role to view its details, and then copy the instance profile ARN.

    Be sure to copy the instance profile ARN, and not the role ARN. You will enter this instance profile ARN when you create the AWS environment or Amazon EC2 deployment in Control Hub.

  12. To use a unique instance profile for each deployment, simply repeat these steps to create another instance profile and attach the same IAM policy to it.

Configure AWS Credentials

You can grant Control Hub access to your AWS account using a cross-account role or access keys. Control Hub uses the credentials to access and provision resources in your Amazon VPC. StreamSets recommends that you use a cross-account role for production.

Important: Configuring AWS credentials requires logging into StreamSets DataOps Platform to retrieve information generated for the organization. If you have not yet joined the StreamSets organization, ask your organization administrator to invite you.

Complete the following steps to configure AWS credentials for Control Hub:

  1. Create an IAM policy that delegates limited access to Control Hub. Create the same policy when using either authentication method.
  2. Use the IAM policy with a cross-account role or with access keys.

Create an IAM Policy

For either authentication method, create an IAM policy in AWS that delegates limited access to Control Hub. Use the sample policy that StreamSets provides. You can make compatible changes to the policy as needed.

Note: AWS provides multiple methods for creating IAM policies. These steps provide brief instructions to create a new policy using the AWS Management Console. For instructions on using other methods, such as using the AWS CLI, see the AWS IAM documentation.
  1. Retrieve the sample IAM policy provided by StreamSets. Alternatively, you can copy the contents of the sample policy below, but be aware that this can introduce formatting errors.
    1. Log into StreamSets DataOps Platform.
    2. In the Control Hub Navigation panel, click Set Up > Environments and then click the Create Environment icon: .
    3. Enter a name for the environment, and then select Amazon Web Services (AWS) for the Environment Type.
    4. Click Save & Next.
    5. Copy or download the sample IAM Policy for Credentials.

      You can leave this page open in the browser, cancel the environment creation, or click Save & Exit to save an incomplete environment that you finish configuring after completing the prerequisites.

  2. In the AWS Console, go to the IAM service.
  3. In the navigation pane, click Policies, and then click Create policy.
  4. In the policy editor, click the JSON tab and paste the contents of the sample policy that you retrieved from Control Hub.

    The contents of the sample IAM policy for credentials are as follows:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeImages",
                "autoscaling:DescribeScalingActivities",
                "ec2:DescribeVpcs",
                "autoscaling:DescribeAutoScalingGroups",
                "ec2:DescribeRegions",
                "autoscaling:DescribeLaunchConfigurations",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeInstanceTypeOfferings",
                "ec2:DescribeSubnets",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeInstances",
                "autoscaling:DescribeScheduledActions",
                "autoscaling:DescribeWarmPool"
            ],
            "Resource": "*"
        },
        {
            "Sid": "1",
            "Effect": "Allow",
            "Action": [
                "cloudformation:DescribeStacks",
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:UpdateStack",
                "cloudformation:DescribeStackEvents"
            ],
            "Resource": "arn:aws:cloudformation:*:<ACCOUNT_ID>:stack/StreamSets-*/*"
        },
        {
            "Sid": "2",
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:ListTagsForResource",
                "ssm:AddTagsToResource",
                "ssm:RemoveTagsFromResource"
            ],
            "Resource": "arn:aws:ssm:*:<ACCOUNT_ID>:parameter/*"
        },
        {
            "Sid": "3",
            "Effect": "Allow",
            "Action": [
                "autoscaling:CreateLaunchConfiguration",
                "autoscaling:DeleteLaunchConfiguration",
                "autoscaling:UpdateAutoScalingGroup",
                "autoscaling:DeleteAutoScalingGroup",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "autoscaling:CreateAutoScalingGroup",
                "autoscaling:CreateOrUpdateTags",
                "autoscaling:DescribeTags",
                "autoscaling:DeleteTags",
                "autoscaling:SetDesiredCapacity"
            ],
            "Resource": [
                "arn:aws:autoscaling:*:<ACCOUNT_ID>:autoScalingGroup:*:autoScalingGroupName/StreamSets-*",
                "arn:aws:autoscaling:*:<ACCOUNT_ID>:launchConfiguration:*:launchConfigurationName/StreamSets-*"
            ]
        },
        {
            "Sid": "4",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/<INSTANCE_PROFILE_NAME>"
        },
        {
            "Sid": "5",
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
        }
    ]
}
  5. Replace the following parameters in the sample policy:
    Parameter Replacement Value
    <ACCOUNT_ID> Your 12-digit AWS account ID.
    <INSTANCE_PROFILE_NAME> Name of the instance profile that you created for EC2 instances provisioned by StreamSets.

    For example, enter StreamSets-Deployment-EC2-Instance-Profile.

    You can make compatible changes to the policy as needed.

  6. If you created multiple instance profiles, add each instance profile to the sample policy as follows, replacing the <INSTANCE_PROFILE_NAME_n> parameters with the name of each profile that you created: 
    ...
       {
            "Sid": "4",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": [
                "arn:aws:iam::<ACCOUNT_ID>:role/<INSTANCE_PROFILE_NAME_1>",
                "arn:aws:iam::<ACCOUNT_ID>:role/<INSTANCE_PROFILE_NAME_2>"
             ]
        },
...

    For example, if you created a default instance profile for the environment and then created a second instance profile that one deployment will use to override the default profile, enter both profile names here.

  7. Click Next: Tags.
  8. Click Review policy.
  9. On the Review page, enter a policy name.

    For example, to designate that this policy is for StreamSets environments, you might enter the name StreamSets-Environment-Policy.

  10. Click Create policy.

Use the Policy with a Cross-Account Role

For cross-account role authentication, create a cross-account role and attach the IAM policy that you created to this role. Control Hub assumes this role to perform tasks in your AWS account.

Note: AWS provides multiple methods for creating cross-account roles. These steps provide brief instructions to create a new role using the AWS Management Console. For instructions on using other methods, such as using the AWS CLI, see the AWS IAM documentation.
  1. To create a cross-account role, first retrieve the unique external ID generated for your Control Hub organization.
    Important: Using the generated external ID prevents the confused deputy problem and ensures that Control Hub can assume this role only when acting on behalf of your organization.
    1. Log into StreamSets DataOps Platform.
    2. In the Control Hub Navigation panel, click Set Up > Environments and then click the Create Environment icon: .
    3. Enter a name for the environment, and then select Amazon Web Services (AWS) for the Environment Type.
    4. Click Save & Next.
    5. Select Cross-Account Role for the Credential Type.
    6. Copy the External ID value.

      You can leave this page open in the browser, cancel the environment creation, or click Save & Exit to save an incomplete environment that you finish configuring after completing the prerequisites.

  2. In the AWS Console, go to the IAM service.
  3. In the navigation pane, click Roles, and then click Create role.
  4. Select Another AWS account as the type of trusted entity.

  5. In the Account ID property, enter the StreamSets AWS account ID: 632637830840.
  6. Select Require external ID and paste the external ID that you retrieved from Control Hub.
  7. Click Next: Permissions.
  8. Search for and then select the policy that you created for StreamSets environments.

    For example, select StreamSets-Environment-Policy.

  9. Click Next: Tags.
  10. Click Next: Review.
  11. On the Review page, enter a role name.

    For example, to designate that this role is for StreamSets environments, you might enter the name StreamSets-Environment-Role.

  12. Click Create role.
  13. Click the name of the role to view its details, and then copy the role ARN.

    You will enter this ARN when you create the AWS environment in Control Hub.

Use the Policy with Access Keys

For access keys authentication, create an IAM user with programmatic access that uses access keys, and attach the IAM policy that you created to this user. Control Hub uses these access keys to perform tasks in your AWS account.

Note: AWS provides multiple methods for creating users. These steps provide brief instructions to create a new user using the AWS Management Console. You can also attach the policy to an existing user. For instructions on using other methods, such as using the AWS CLI, see the AWS IAM documentation.
  1. In the AWS Console, go to the IAM service.
  2. In the navigation pane, click Users, and then click Add user.
  3. Enter a user name.

    For example, to designate that this user is for StreamSets environments, you might enter the name StreamSets-Environment-User.

  4. For Access type, select Programmatic access.
  5. Click Next: Permissions.
  6. Select Attach existing policies directly.
  7. Search for and then select the policy that you created for StreamSets environments.

    For example, select StreamSets-Environment-Policy.

  8. Click Next: Tags.
  9. Click Next: Review.
  10. Click Create User.
  11. Click Download .csv to download a CSV file that contains the access key ID and secret access key.

    You will enter these access keys when you create the AWS environment in Control Hub.

Creating an AWS Environment

Create an AWS environment to define where to deploy StreamSets engines in your AWS account.

Important: Before creating an environment, your AWS administrator must complete the required prerequisites.

To create a new AWS environment, click Set Up > Environments in the Navigation panel, and then click the Create Environment icon: . Or, if you saved an incomplete environment when you retrieved the information required by the prerequisites, simply edit that environment.

Then, complete the following steps in the environment wizard:
  1. Define the Environment
  2. Configure AWS Credentials
  3. Select the AWS Region
  4. Configure the AWS VPC
  5. Configure AWS Subnets
  6. Share the Environment
  7. Review and Activate the Environment

Define the Environment

Define the environment essentials, including the environment name and type, and optional tags to identify similar environments.

  1. Configure the following properties:
    Define Environment Property Description
    Environment Name Name of the environment.

    Use a brief name that informs your team of the environment use case.
    Environment Type Select one of the following types:

    Once saved, you cannot change the environment type.
    Environment Tags Optional tags that identify similar environments within Control Hub. Use environment tags to easily search and filter environments.

    Enter nested tags using the following format:

    <tag1>/<tag2>/<tag3>
  2. Optionally, click Show Advanced Options and configure the following advanced property:
    Define Environment Advanced Property Description
    Allow Nightly Builds Allows deployments for this environment to use nightly engine builds in addition to released engine versions. Also allows Kubernetes environments to use nightly StreamSets Kubernetes agent builds.

    Nightly builds are for testing features under development and should not be used in production systems.

    The version number of a nightly build includes a -SNAPSHOT suffix and the build number. For example, 5.2.0-SNAPSHOT (Build 1013).
  3. Click one of the following buttons:
    • Cancel - Cancels creating the environment and exits the wizard.
    • Save & Next - Saves the environment and continues.
    • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

Configure AWS Credentials

Configure the credentials that Control Hub uses to access and provision resources in your AWS account.
Important: Before you configure AWS credentials, your AWS administrator must complete the required prerequisites.
  1. Configure the following properties:
    Credentials Property Description
    Credential Type Type of credentials to authenticate with AWS:
    • Cross-Account Role
    • Access Keys
    Important: StreamSets recommends that you use a cross-account role for production.
    IAM Policy for Credentials Sample IAM policy to copy or download so that your AWS administrator can configure AWS credentials as an environment prerequisite.

    The parameters in the sample policy must be replaced with the correct values for your AWS account.
    External ID Unique ID generated for your organization. Copy the ID so that your AWS administrator can create the cross-account role as an environment prerequisite.

    Applicable when using a cross-account role to authenticate with AWS.
    Role ARN Amazon resource name (ARN) of the cross-account role created as an environment prerequisite by your AWS administrator. Enter using the following format:

    arn:aws:iam::<account_id>:role/<role_name>

    Required when using a cross-account role to authenticate with AWS.
    Access Key ID AWS access key ID downloaded as an environment prerequisite by your AWS administrator.

    Required when using access keys to authenticate with AWS.
    Secret Access Key AWS secret access key downloaded as an environment prerequisite by your AWS administrator.

    Required when using access keys to authenticate with AWS.
    IAM Policy for Profile Sample IAM policy to copy or download so that your AWS administrator can configure instance profiles for EC2 instances as an environment prerequisite.

    The parameters in the sample policy must be replaced with the correct values for your AWS account.
    Default Instance Profile Optional instance profile to associate with the EC2 instances provisioned for all deployments belonging to this environment. Enter the instance profile ARN of the instance profile created as a prerequisite by your AWS administrator. Use the following format:

    arn:aws:iam::<account_id>:instance-profile/<role_name>

    If you do not define a default instance profile, then you must define an instance profile when you create a deployment for this environment.

  2. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the environment and continues.
    • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

Select the AWS Region

Select the AWS region for the Amazon VPC created as a prerequisite by your AWS administrator.

  1. Select the AWS region where the VPC is located.
  2. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the environment and continues.
    • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

Configure the AWS VPC

Select the Amazon VPC created as a prerequisite by your AWS administrator, and optionally define AWS tags to apply to provisioned AWS resources.

  1. Configure the following properties:
    VPC Property Description
    VPC ID ID of the Amazon VPC created as an environment prerequisite by your AWS administrator.
    AWS Tags AWS tags to apply to all AWS resources provisioned for this environment.

    Enter the tags as key-value pairs. For tag naming requirements, see the AWS documentation.

    You can define the tags using simple or bulk edit mode. In simple edit mode, click Add Another to define additional tags. In bulk edit mode, configure tags in JSON format.

    Important: These tags are applied to AWS resources, not to Control Hub environments.
  2. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the environment and continues.
    • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

Configure AWS Subnets

Select the subnets and security group to use within the Amazon VPC created as a prerequisite by your AWS administrator.

  1. Configure the following properties:
    Subnet Property Description
    Subnet IDs Subnet IDs within the VPC to use to provision AWS resources.

    Select one or more available subnet IDs.
    Security Group ID Security group ID assigned to the VPC.

    The security group must meet the requirements as described in Security Group.
  2. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the environment and continues.
    • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

Share the Environment

By default, the environment can only be seen by you. Share the environment with other users and groups to grant them access to it.

  1. In the Select Users and Groups field, type a user email address or a group name.
  2. Select users or groups from the list, and then click Add.

    The added users and groups display in the User / Group table.

  3. Modify permissions as needed. By default, each added user or group is granted the following permissions:
    • Read - View the details of the environment. Create and edit a deployment for the environment.
    • Write - Edit, activate, deactivate, and delete the environment.

    For more information, see Environment Permissions.

  4. Click one of the following buttons:
    • Back - Returns to the previous step in the wizard.
    • Save & Next - Saves the environment and continues.
    • Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.

Review and Activate the Environment

You've successfully finished creating the environment. Activate the environment so that you can create deployments for the environment.

Click one of the following buttons:
  • Exit - Saves the environment and exits the wizard, displaying the Deactivated environment in the Environments view.
  • Activate & Add Deployment - Activates the environment and opens the deployment wizard so that you can create a deployment for the environment.
  • Activate & Exit - Activates the environment and exits the wizard, displaying the Active environment in the Environments view.