GCP Environments
A Google Cloud Platform (GCP) environment represents the Google virtual private cloud (VPC) network in your Google Cloud account where engines are deployed.
Your GCP administrator must designate a project for the resources, create a VPC network in the project, and configure Google Cloud credentials for Control Hub to use. You then create a GCP environment in Control Hub that represents the VPC network. When you activate the environment, Control Hub connects to the project and VPC network using the configured credentials, provisions the Google Cloud resources needed to run engines, and deploys engine instances to those resources.
While the environment is in an active state, Control Hub periodically verifies that the project and VPC network exist and that the credentials are valid. Control Hub does not provision resources in the VPC network until you create and start a deployment for this environment.
Prerequisites
The prerequisites require logging into StreamSets DataOps Platform to retrieve information generated for the organization.
You first must invite your Google Cloud administrator to join your StreamSets organization. You can invite the administrator using the default role assignments, or you can modify the role assignments to grant the administrator the Environment Manager role only.
- Designate a Google Cloud project for the Google Cloud resources that Control Hub provisions. Then, enable the required Google APIs on the project.
- Create a Google VPC network in the designated project for the StreamSets GCP environment to use.
- Configure the Google Clouds credentials that Control Hub uses to access and provision resources in your project.
- Create instance service accounts to associate with the provisioned VM instances.
Designate a Project and Enable Google APIs
Designate a Google Cloud project for the resources that Control Hub provisions. You can use an existing project or create a new project. You'll select this project when you create the StreamSets GCP environment.
For instructions on creating or managing Google Cloud projects, see the Google Cloud Resource Manager documentation.
- Cloud Resource Manager API
- Compute Engine API
- Identity and Access Management (IAM) API
- Cloud Deployment Manager V2 API
- Secret Manager API
gcloud
command line tool. For instructions
on using other methods, such as using the Google Cloud Console, see the Google Cloud Endpoints documentation.- Run the following
gcloud
command to set your designated project as the current project:gcloud config set project <PROJECT_ID>
- Run the following command to view the list of Google APIs currently enabled on the
project:
gcloud services list
- Run the following command to enable the required APIs on the
project:
gcloud services enable cloudresourcemanager.googleapis.com compute.googleapis.com iam.googleapis.com deploymentmanager.googleapis.com secretmanager.googleapis.com
Create a Google VPC Network
Create a Google virtual private cloud (VPC) network in your designated Google Cloud project. Or, create a shared VPC network in a host project and then attach your designated project to the host project. For more information on shared VPC networks, see the Google Cloud VPC documentation.
You can use an existing VPC network. However, StreamSets recommends creating a new VPC network for the exclusive use of each StreamSets GCP environment.
You can use private or public subnets within the VPC network, as long as the subnets can send outbound traffic to the internet.
For instructions on creating a VPC network and on allowing subnets internet access, see the Google Cloud VPC documentation.
Firewall Rules
Define the required firewall rules for the VPC network.
- Inbound and outbound connections required by StreamSets engines, as described in Firewall Configuration.
- Outbound connections to Google Secret Manager. Add the IP address of the
https://secretmanager.googleapis.com
host as an allowed destination.For the list of Google Cloud IP addresses, see this Google support article.
You can define the firewall rules for the entire VPC network. Or, you can apply network tags to the firewall rules. Network tags allow you to apply firewall rules to specific VM instances within the network.
When you configure a GCE deployment for the environment, you specify the network tags to use for the provisioned VM instances. For more information on configuring network tags, see the Google Cloud VPC documentation.
Configure Google Cloud Credentials
You can grant Control Hub access to your Google Cloud project using service account impersonation or a service account key. Control Hub uses the credentials to access and provision resources in the project. StreamSets recommends that you use service account impersonation for production.
- Create an IAM service account and add IAM roles to the account to delegate limited access to Control Hub. Create the service account with the same roles when using either credential type.
- Allow Control Hub to impersonate the service account, or create the service account key that Control Hub uses.
Create a Service Account
For either credential type, create a service account in Google Cloud that delegates limited access to Control Hub.
The service account requires the following IAM roles:
- Compute Network Viewer role
- Deployment Manager Editor role, with a condition that limits access to resources provisioned by Control Hub
- Service Account User role
- Secret Manager Admin role, with a condition that limits access to resources provisioned by Control Hub
gcloud
command line tool. For instructions on
using other methods, such as using the Google Cloud Console, see the Google Cloud IAM documentation.Use the Service Account for Impersonation
To use service account impersonation as the credential type, add Control Hub as a member of the service account that you created. This allows Control Hub to impersonate the service account to perform tasks in your Google Cloud project.
gcloud
command line tool. For instructions on using other
methods, such as using the Google Cloud Console, see the Google Cloud IAM documentation.gcloud
command to add the Control Hub service account as a member of this service account:
gcloud iam service-accounts add-iam-policy-binding <SA_NAME>@<PROJECT_ID>.iam.gserviceaccount.com --member=serviceAccount:streamsets@streamsets-gcp-bridge.iam.gserviceaccount.com --role=roles/iam.serviceAccountTokenCreator
Replace the following parameters in the command:
Parameter | Replacement Value |
---|---|
<SA_NAME> | Service account name that you retrieved from Control Hub. |
<PROJECT_ID> | ID of the project designated for the StreamSets GCP environment. |
<SA_NAME>@<PROJECT_ID>.iam.gserviceaccount.com
, when you
create the GCP environment in Control Hub.Use the Service Account Key
To use service account key as the credential type, create a key for the service account that you created. Control Hub uses the key to perform tasks in your Google Cloud project.
gcloud
command line tool,
see the Google Cloud IAM documentation. Create Instance Service Accounts for VM Instances
Create instance service accounts for Google Compute Engine VM instances. When Control Hub provisions VM instances for a GCE deployment belonging to this environment, it associates these instance service accounts with the VM instances.
Instance service accounts require the Secret Manager Secret Accessor role with a condition that limits access to resources provisioned by Control Hub.
- Configure a default instance service account for the environment
- Configure a default instance service account for the parent GCP environment. When you create a GCE deployment for this environment, you can simply use the default instance service account configured for the environment.
- Configure a unique instance service account for each deployment
- Do not configure a default instance service account for the parent GCP environment. When you create a GCE deployment for this environment, you must configure the instance service account to use for the deployment.
- Configure a default instance service account and override as needed
- Configure a default instance service account for the parent GCP environment. When you create a GCE deployment for this environment, you can use the default instance service account configured for the environment, or you can override the default and configure a different instance service account for the deployment to use.
gcloud
command line tool. For instructions on using
other methods, such as using the Google Cloud Console, see the Google Cloud Compute Engine
documentation.Creating a GCP Environment
Create a Google Cloud Platform (GCP) environment to define where to deploy StreamSets engines in your Google Cloud project.
To create a new GCP environment, click Create
Environment icon: . Or, if you saved an incomplete environment
when you retrieved the information required by the prerequisites, simply edit that
environment.
Define the Environment
Define the environment essentials, including the environment name and type, and optional tags to identify similar environments.
Configure GCP Credentials
Configure the GCP Project
Select the GCP project prepared as a prerequisite by your Google Cloud administrator.
- Select the GCP project designated for the StreamSets environment.
-
Click one of the following buttons:
- Back - Returns to the previous step in the wizard.
- Save & Next - Saves the environment and continues.
- Save & Exit - Saves the environment and exits the wizard, displaying the incomplete environment in the Environments view.
Select the GCP VPC
Select the VPC network created as a prerequisite by your Google Cloud administrator, and optionally define GCP labels to apply to provisioned GCP resources.
Share the Environment
By default, the environment can only be seen by you. Share the environment with other users and groups to grant them access to it.
Review and Activate the Environment
You've successfully finished creating the environment. Activate the environment so that you can create deployments for the environment.
- Exit - Saves the environment and exits the wizard, displaying the Deactivated environment in the Environments view.
- Activate & Add Deployment - Activates the environment and opens the deployment wizard so that you can create a deployment for the environment.
- Activate & Exit - Activates the environment and exits the wizard, displaying the Active environment in the Environments view.