Azure VM Deployments
You can create an Azure Virtual Machine (Azure VM) deployment for an active Azure environment.
When you create an Azure VM deployment, you define the engine type, version, and configuration to deploy to the Azure virtual network (VNet) specified in the environment. You also specify the number of engine instances to deploy. Each engine instance runs on a dedicated VM instance.
When you start an Azure VM deployment, Control Hub connects to the Azure VNet specified in the environment and then uses an Azure Resource Manager template to create an Azure Deployment. Azure Resource Manager provisions the group of VM instances in the VNet and then deploys and launches one StreamSets engine instance on each VM instance.
Azure Resource Manager manages the provisioning and monitoring of the VM instances. Control Hub simply receives the status of the deployed StreamSets engine instances and sends any updates to Resource Manager.
When you stop an Azure VM deployment, Resource Manager deletes the existing VM instances.
For more information about Azure Resource Manager, see the Azure Resource Manager documentation.
Before you create an Azure VM deployment, you must complete several prerequisites.
VM Instance Details
Engine Type | Software |
---|---|
Data Collector |
|
Transformer |
|
Prerequisites
- Create a Microsoft Azure (Azure) environment
- Create and activate an Azure environment in Control Hub, as described in Azure Environments.
- Configure a managed identity
- Ask your Azure administrator to configure a managed identity in Azure to associate with the provisioned VM instances. If a default managed identity is defined for the parent Azure environment, you can skip this prerequisite and simply use the default. If a default is not set or if you'd like to override the default for the deployment, see Configure Managed Identities for VM Instances.
- Create a resource group
- Ask your Azure administrator to create a resource group in Azure that the provisioned VM instances are assigned to. If a default resource group is defined for the parent Azure environment, you can skip this prerequisite and simply use the default. If a default is not set or if you'd like to override the default for the deployment, see Configure Resource Groups for VM Instances.
- Create an SSH key pair
- Control Hub does not use a secure shell (SSH) key pair to access the VM instances.
However, Azure requires that an SSH key be assigned to all VM instances.
Designate the SSH key to use in one of the following ways:
- Create a local key pair. When you create the deployment, enter the full contents of the public key to assign to the provisioned VM instances.
- Ask your Azure administrator to create a new key pair or to designate an existing key pair in Azure. When you create the deployment, select the key pair name to assign to the provisioned VM instances. For more information on using the Azure portal to create SSH keys to access Linux VM instances, see the Azure Virtual Machines documentation.
- Optionally, set up an external resource archive
- When your pipelines require external resources and when you plan to deploy multiple engine instances, you must set up an external resource archive that all engine instances can access. When your pipelines do not require external resources or when using a single engine instance to get started with StreamSets, you do not need to complete this prerequisite.
Init Script for Custom DNS Servers
When the Azure VNet uses custom DNS servers, you must define the following initialization script in the Azure deployment so that a StreamSets engine can detect the hostname of the provisioned VM instance.
#!/bin/sh
requireddomain=reddog.microsoft.com
new_ip_address=$(ip -f inet a show eth0| grep inet| awk '{ print $2}' | cut -d/ -f1)
host=`hostname`
nsupdatecmds=/var/tmp/nsupdatecmds
echo "update delete $host.$requireddomain a" > $nsupdatecmds
echo "update add $host.$requireddomain 3600 a $new_ip_address" >> $nsupdatecmds
echo "send" >> $nsupdatecmds
nsupdate $nsupdatecmds
If needed, you can add additional commands to the end of this init script.
For more details about defining an init script, see Init Script.
Creating an Azure VM Deployment
To create a new Azure VM deployment, click Create
Deployment icon: .
Define the Deployment
Define the deployment essentials, including the deployment name and type, the environment that the deployment belongs to, and the engine type and version to deploy.
Once saved, you cannot change the deployment type, the engine version, or the environment.
Configure the Engine
Define the configuration of the engine to deploy. You can use the defaults to get started.
Configure Azure VM Zones
Select the zones to provision the VM instances in. If the Azure region selected for the parent environment does not support zones, the deployment wizard skips this step.
- Select the zones to provision the VM instances in.
-
Click one of the following buttons:
- Back - Returns to the previous step in the wizard.
- Save & Next - Saves the deployment and continues.
- Save & Exit - Saves the deployment and exits the wizard, displaying the incomplete deployment in the Deployments view.
Configure the Azure VM Autoscaling Group
Configure details about the Azure VM instances that will be provisioned.
Configure Azure VM SSH Access
Configure the SSH key to assigned to all provisioned VM instances.
Share the Deployment
By default, the deployment can only be seen by you. Share the deployment with other users and groups to grant them access to it.
Review and Launch the Deployment
You've successfully finished creating the deployment.
Editing an Azure VM Deployment
You can edit an Azure VM deployment while it is deactivated or active.
When you stop a deployment, all existing VM instances are deleted. After you edit properties and then restart the deployment, Control Hub uses Azure Resource Manager to provision a new group of VM instances and launch a new StreamSets engine instance on each VM instance.
When you edit a deployment while it is active, existing VM instances might be deleted, depending on the following types of edited properties:
- General deployment or engine properties
- When you edit general deployment or engine properties while the deployment is active, Azure Resource Manager continues running the existing VM instances. Changes are replicated to all StreamSets engine instances on the next restart of the engines.
- Azure VM properties
- When you edit Azure VM properties while the deployment is active, Azure Resource Manager might replace all of the existing VM instances, depending on the change. If a replacement is needed, Resource Manager replaces all of the existing VM instances. This results in engine downtime while the new instances are being provisioned.
To edit a deployment, locate the deployment in the Deployments
view. In the Actions column, click the
More icon () and then click Edit.
Tracking URL
When you view the details of an active Azure VM deployment, you can access a tracking URL to the Azure portal. Use the URL to view additional information about the Azure resources automatically provisioned for the deployment.
To access the tracking URL, click an Azure VM deployment name in the Deployments view and then locate the Tracking URL property in the deployment details.
Click the URL to open the Azure portal. The portal displays the overview page of the virtual machine scale set created for your StreamSets deployment. The overview page includes links to the VM instances and the resource group provisioned for the deployment, as follows:
The following topics provide brief tips on finding the most useful information about the provisioned resources. For more details about monitoring an Azure virtual machine scale set, see the Microsoft Azure documentation.
Instances
In the overview page of the virtual machine scale set, click
. The Azure portal lists all VM instances provisioned for the deployment.Click an instance name to view specific details about the VM instance, including the public and private IP addresses. For example, the following image displays a sample VM instance details page:
Resource Group
In the overview page of the virtual machine scale set, click the name of the resource group. The Azure portal displays the overview page for the resource group.
To view all deployments created for the resource group, click
. The Azure portal lists each deployment with a Succeeded or Failed status, as follows:Click the name of the first deployment in the list to view the deployment overview page. Expand the deployment details to access the public IP addresses, as follows: