AWS Environments
An Amazon Web Services (AWS) environment represents the Amazon virtual private cloud (VPC) in your AWS account where engines are deployed.
Your AWS administrator must create a VPC in your AWS account and configure AWS credentials for Control Hub to use. You then create an AWS environment in Control Hub that represents the VPC. When you activate the environment, Control Hub connects to the VPC using the configured credentials, provisions the AWS resources needed to run engines, and deploys engine instances to those resources.
While the environment is in an active state, Control Hub periodically verifies that the Amazon VPC exists and that the credentials are valid. Control Hub does not provision resources in the VPC until you create and start a deployment for this environment.
Before you create an AWS environment, your AWS administrator must complete several prerequisites.
Prerequisites
The prerequisites require logging into StreamSets DataOps Platform to retrieve information generated for the organization.
You first must invite your AWS administrator to join your StreamSets organization. You can invite the administrator using the default role assignments, or you can modify the role assignments to grant the administrator the Environment Manager role only.
- Create an Amazon VPC for the StreamSets AWS environment to use.
- Configure instance profiles to associate with the provisioned EC2 instances.
- Configure the AWS credentials that Control Hub uses to access and provision resources in your Amazon VPC.
Create an Amazon VPC
Create an Amazon virtual private cloud (VPC) in your AWS account.
You can use an existing VPC. However, StreamSets recommends creating a new VPC for the exclusive use of each StreamSets AWS environment.
You can use private or public subnets within the VPC network, as long as the subnets can send outbound traffic to the internet.
For instructions on creating a VPC and on allowing subnets internet access, see the Amazon VPC documentation.
Security Group
Assign a security group to the VPC that defines the required inbound and outbound rules. You can use an existing security group or create a new group.
- Inbound and outbound connections required by StreamSets engines, as described in Firewall Configuration.
- Outbound connections to AWS Systems Manager. Add the IP address of the
ssm.<region>.amazonaws.com
host as an allowed destination.For the list of AWS IP addresses, see the AWS documentation.
Configure Instance Profiles for EC2 Instances
Configure instance profiles for EC2 instances in your AWS account. When Control Hub provisions EC2 instances for an Amazon EC2 deployment belonging to this environment, it associates these instance profiles with the EC2 instances.
- Configure a default instance profile for the environment
- Configure a default instance profile for the parent AWS environment. When you create an Amazon EC2 deployment for this environment, you can simply use the default instance profile configured for the environment.
- Configure a unique instance profile for each deployment
- Do not configure a default instance profile for the parent AWS environment. When you create an Amazon EC2 deployment for this environment, you must configure the instance profile to use for the deployment.
- Configure a default instance profile and override as needed
- Configure a default instance profile for the parent AWS environment. When you create an Amazon EC2 deployment for this environment, you can use the default instance profile configured for the environment, or you can override the default and configure a different instance profile for the deployment to use.
Regardless of how you configure the instance profile for the EC2 instances, you must create the IAM policy and attach it to an instance profile as an AWS environment prerequisite. When you create the IAM policy that you attach to the AWS credentials that Control Hub uses to access your AWS account, you must include the Amazon resource name (ARN) of all instance profiles in that policy. This grants Control Hub the ability to associate the instance profiles with the EC2 instances.
- Create an IAM policy that grants limited access to the provisioned EC2 instances.
- Use the IAM policy with an instance profile.
Create an IAM Policy
Create an IAM policy in AWS that grants limited access to the provisioned EC2 instances. Use the sample policy that StreamSets provides. You can make compatible changes to the policy as needed.
Use the Policy with an Instance Profile
In AWS, create an instance profile to associate with the provisioned EC2 instances. You can create a single default instance profile to use for all deployments belonging to the parent environment, or you can create a unique instance profile for each deployment.
Configure AWS Credentials
You can grant Control Hub access to your AWS account using a cross-account role or access keys. Control Hub uses the credentials to access and provision resources in your Amazon VPC. StreamSets recommends that you use a cross-account role for production.
Complete the following steps to configure AWS credentials for Control Hub:
- Create an IAM policy that delegates limited access to Control Hub. Create the same policy when using either authentication method.
- Use the IAM policy with a cross-account role or with access keys.
Create an IAM Policy
For either authentication method, create an IAM policy in AWS that delegates limited access to Control Hub. Use the sample policy that StreamSets provides. You can make compatible changes to the policy as needed.
Use the Policy with a Cross-Account Role
For cross-account role authentication, create a cross-account role and attach the IAM policy that you created to this role. Control Hub assumes this role to perform tasks in your AWS account.
Use the Policy with Access Keys
For access keys authentication, create an IAM user with programmatic access that uses access keys, and attach the IAM policy that you created to this user. Control Hub uses these access keys to perform tasks in your AWS account.
Creating an AWS Environment
Create an AWS environment to define where to deploy StreamSets engines in your AWS account.
To create a new AWS environment, click Create
Environment icon: . Or, if you saved an incomplete environment
when you retrieved the information required by the prerequisites, simply edit that
environment.
Define the Environment
Define the environment essentials, including the environment name and type, and optional tags to identify similar environments.
Configure AWS Credentials
Select the AWS Region
Select the AWS region for the Amazon VPC created as a prerequisite by your AWS administrator.
- Select the AWS region where the VPC is located.
-
Click one of the following buttons:
- Back - Returns to the previous step in the wizard.
- Save & Next - Saves the environment and continues.
- Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.
Configure the AWS VPC
Select the Amazon VPC created as a prerequisite by your AWS administrator, and optionally define AWS tags to apply to provisioned AWS resources.
Configure AWS Subnets
Select the subnets and security group to use within the Amazon VPC created as a prerequisite by your AWS administrator.
Share the Environment
By default, the environment can only be seen by you. Share the environment with other users and groups to grant them access to it.
Review and Activate the Environment
You've successfully finished creating the environment. Activate the environment so that you can create deployments for the environment.
- Exit - Saves the environment and exits the wizard, displaying the Deactivated environment in the Environments view.
- Activate & Add Deployment - Activates the environment and opens the deployment wizard so that you can create a deployment for the environment.
- Activate & Exit - Activates the environment and exits the wizard, displaying the Active environment in the Environments view.