Azure Driver

Considerations & Limitations

You should take into account the following technical considerations when using the Microsoft Azure (AZ) cloud with OpenNebula:

  • There is no direct access to the hypervisor, so it cannot be monitored. (We don’t know where the VM is running on the Azure cloud.)
  • The usual OpenNebula functionality for snapshotting, hot-plugging, or migration is not available with Azure.
  • By default, OpenNebula will always launch Small (1 CPU, 1792 MB RAM) instances, unless otherwise specified. The following table is an excerpt of all the instance types available in Azure. A more exhaustive list can be found (and edited) in /etc/one/az_driver.conf.
Name CPU Capacity Memory Capacity
ExtraSmall 0.1 Cores 768 MB
Small 1 Cores 1792 MB
Medium 2 Cores 3584 MB
Large 4 Cores 7168 MB
ExtraLarge 8 Cores 14336 MB
A5 2 Cores 14336 MB
A6 4 Cores 28672 MB
A7 8 Cores 57344 MB
A8 8 Cores 57344 MB
A9 16 Cores 114688 MB

Prerequisites

  • You must have a working account for Azure.
  • First, the Subscription ID, that can be uploaded and retrieved from All services -> Subscriptions.
  • Second, the Management Certificate file, that can be created with the following steps. We need the .pem file (for the ruby gem) and the .cer file (to upload to Azure):
## Install openssl

## CentOS
$ sudo yum install openssl

## Ubuntu
$ sudo apt-get install openssl

# Move to a folder (wherever you prefer, but it's best to choose a private folder to store all your keys)
$ mkdir ~/.ssh/azure && cd ~/.ssh/azure

## Create certificate
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout myPrivateKey.key -out myCert.pem
$ chmod 600 myPrivateKey.key

## Generate .cer file for Azure
$ openssl x509 -outform der -in myCert.pem -out myCert.cer

## You should have now your .pem certificate and your private key
$ find .
==>
    ./myCert.pem
    ./myPrivateKey.key
  • Third, the certificate file (.cer) has to be uploaded to All services -> Subscriptions -> Management certificates
  • In order to allow the Azure driver to authenticate properly with our Azure account, you need to sign your .pem file:
## Concatenate key and pem certificate (sign with private key)
$ cat myCert.pem myPrivateKey.key > azureOne.pem

azureOne.pem is the resulting signed certificate. OpenNebula does not need you to store this in any specific location of the OpenNebula front-end filesystem, but please keep it safe. Remember that you will need to read it in the host creation process. We’ll talk about how to perform this action later.

  • The following gem is required: azure. This gem is automatically installed as part of the installation process. Otherwise, run the install_gems script as root:
/usr/share/one/install_gems cloud

OpenNebula Configuration

Uncomment the Azure AZ IM and VMM drivers from the /etc/one/oned.conf file in order to use the driver.

IM_MAD = [
      name       = "az",
      executable = "one_im_sh",
      arguments  = "-c -t 1 -r 0 az" ]

VM_MAD = [
    name       = "az",
    executable = "one_vmm_sh",
    arguments  = "-t 15 -r 0 az",
    type       = "xml" ]

Driver flags are the same as other drivers:

FLAG SETs
-t Number of threads, i.e. number of actions performed at the same time
-r Number of retries when contacting Azure service

The Azure driver has its own configuration file with a few options ready to customize. Edit /etc/one/az_driver.conf:

proxy_uri:
instance_types:
    ExtraSmall:
        cpu: 1
        memory: 0.768
    Small:
        cpu: 1
        memory: 1.75
    Medium:
        cpu: 2
        memory: 3.5
    Large:
        cpu: 4
        memory: 7.0
    ExtraLarge:
        cpu: 8
        memory: 14.0
    ...

In the above file, each instance_type represents the physical resources that Azure will serve.

If the OpenNebula frontend needs to use a proxy to connect to the public Internet you also need to configure the proxy in that file. The parameter is called proxy_uri. Authenticated proxies are not supported, that is ones that require user name and password. For example, if the proxy is on 10.0.0.1 and its port is 8080 the configuration line should read:

proxy_uri: http://10.0.0.1:8080

Warning

The instance_types section lists the machines that Azure is able to provide. The Azure driver will retrieve this kind of information, so it’s better not to change it unless you know what you are doing.

Warning

If you were using OpenNebula before 5.4 you may have noticed that there are no Microsoft credentials in the configuration file anymore. This is for security reasons. In 5.4 there is a new secure credentials storage for Microsoft accounts, so you do not need to store sensitive credential data on disk. The OpenNebula daemon stores the data in an encrypted format.

Once the file is saved, OpenNebula needs to be restarted. Create a new Host with Microsoft credentials that uses the AZ drivers:

onehost create azure_host -t az -i az -v az

Note

-t is needed to specify what type of remote provider host we want to set up. If you’ve followed all the instructions properly, your default editor should appear, asking for the credentials and other mandatory data that will allow you to communicate with Azure.

Once you have opened your editor you can look for additional help at the top of your screen. There is more information in the Azure Auth template Attributes section. The basic three variables you have to set are: AZ_ID, AZ_CERT and REGION_NAME.

Azure Specific Template Attributes

In order to deploy an instance in Azure through OpenNebula, you must include an Azure PUBLIC_CLOUD section in the virtual machine template. This is an example of a virtual machine template that can be deployed in our local resources or in Azure.

CPU      = 0.5
MEMORY   = 128

# KVM template machine, this will be use when submitting this VM to local resources
DISK     = [ IMAGE_ID = 3 ]
NIC      = [ NETWORK_ID = 7 ]

# Azure template machine, this will be use wen submitting this VM to Azure
PUBLIC_CLOUD = [
  TYPE=AZURE,
  INSTANCE_TYPE=ExtraSmall,
  IMAGE=b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140606.1-en-us-30GB,
  VM_USER="azuser",
  VM_PASSWORD="myr@nd0mPass9",
  WIN_RM="https",
  TCP_ENDPOINTS="80",
  SSHPORT=2222
]

#Add this if you want this VM to only go to the West EuropeAzure cloud
#SCHED_REQUIREMENTS = 'HOSTNAME = "west-europe"'

These are the attributes that can be used in the PUBLIC_CLOUD section of the template for TYPE “AZURE”, There is an exhaustive list of attributes in the Virtual Machine Definition File Reference Section.

Note

The PUBLIC_CLOUD sections allow for substitutions from template and virtual network variables, the same way as the CONTEXT section allows.

Default values for all these attributes can be defined in the /etc/one/az_driver.default file.

<!--
 Default configuration attributes for the Azure driver
 (all domains will use these values as defaults)
 Valid attributes are: INSTANCE_TYPE, IMAGE, VM_USER, VM_PASSWORD, LOCATION,
 STORAGE_ACCOUNT, WIN_RM, CLOUD_SERVICE, TCP_ENDPOINTS, SSHPORT, AFFINITY_GROUP,
 VIRTUAL_NETWORK_NAME, SUBNET and AVAILABILITY_SET
 Use XML syntax to specify defaults, note elements are UPCASE
 Example:
 <TEMPLATE>
   <AZURE>
     <LOCATION>west-europe</LOCATION>
     <INSTANCE_TYPE>Small</INSTANCE_TYPE>
     <CLOUD_SERVICE>MyDefaultCloudService</CLOUD_SERVICE>
     <IMAGE>0b11de9248dd4d87b18621318e037d37__RightImage-Ubuntu-12.04-x64-v13.4</IMAGE>
     <VM_USER>MyUser</VM_USER>
     <VM_PASSWORD>MyPassword</VM_PASSWORD>
     <STORAGE_ACCOUNT>MyStorageAccountName</STORAGE_ACCOUNT>
     <WIN_RM>http</WIN_RM>
     <CLOUD_SERVICE>MyCloudServiceName</CLOUD_SERVICE>
     <TCP_ENDPOINTS>80,3389:3390</TCP_ENDPOINTS>
     <SSHPORT>2222</SSHPORT>
     <AFFINITY_GROUP>MyAffinityGroup</AFFINITY_GROUP>
     <VIRTUAL_NETWORK_NAME>MyVirtualNetwork</VIRTUAL_NETWORK_NAME>
     <SUBNET>MySubNet<SUBNET>
     <AVAILABILITY_SET>MyAvailabilitySetName<AVAILABILITY_SET>
   </AZURE>
 </TEMPLATE>
-->

<TEMPLATE>
  <AZURE>
     <LOCATION>west-europe</LOCATION>
     <INSTANCE_TYPE>Small</INSTANCE_TYPE>
  </AZURE>
</TEMPLATE>

Note

Valid Azure images to set in the IMAGE atribute of the PUBLIC_CLOUD section can be extracted with the following ruby snippet:

#!/usr/bin/env ruby

require "azure"

# Get a list of available virtual machine images
def get_image_names
    vm_image_management = Azure.vm_image_management
    vm_image_management.list_os_images.each do |image|
        puts "#{image.os_type}"
        puts "      locations: #{image.locations}"
        puts "      name     : #{image.name}"
        puts
    end
end

Azure.configure do |config|
          config.management_certificate = '/path-to/azureOne.pem'
          config.subscription_id        = 'XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'
end

get_image_names

Azure Auth Attributes

After successfully executing onehost create with the -t option, your default editor will open. An example follows of how you can complete the information:

AZ_ID = "this-is-my-azure-identifier"
AZ_CERT = "-----BEGIN CERTIFICATE-----
          xxxxxxxxxxxxxxxxxxxxxxxxxxxx
          -----END CERTIFICATE-----
          -----BEGIN PRIVATE KEY-----
          xxxxxxxxxxxxxxxxxxxxxxxxxxxx
          -----END PRIVATE KEY-----"

REGION_NAME = "West Europe"
CAPACITY = [
    SMALL = "3",
    MEDIUM = "1" ]

The first two attributes have the authentication info required by Azure:

  • AZ_ID: Your Microsoft Azure account identifier, found in All services -> Subscriptions.
  • AZ_CERT: The certificate that you signed before. In our example this file is called ‘azureOne.pem’. You only need to read this file once to set the attribute and start using Azure:
cat ~/.ssh/azure/azureOne.pem
  • Copy the content into your system clipboard without any mistake selecting all the text (ctrl + Shift + c if you are using a typical desktop).
  • Paste that into the AZ_CERT value. Make sure it is inside quotes, without leaving any blank space.

This information will be encrypted as soon as the host is created. In the host template the values of the AZ_ID and AZ_CERT attributes will be encrypted to maintain your future communication with Azure securely.

  • REGION_NAME: The name of the Azure region that your account uses to deploy machines. You can check Microsoft’s Regions Azure page to find more about the region availability.

In the example the region is set to West Europe.

  • CAPACITY: This attribute sets the size and number of Azure machines that your OpenNebula host will handle. See the instance_types section in the azure_driver.conf file for the supported names. Remember that it is mandatory to capitalize the names (Small => SMALL).

Multi Azure Location/Account Support

It is possible to define various Azure hosts to allow OpenNebula to manage different Azure locations or different Azure accounts. OpenNebula chooses the datacenter in which to launch the VM in the following way:

  • If the VM description contains the LOCATION attribute, then OpenNebula knows that the VM needs to be launched in this Azure location.
  • If the name of the host matches the region name (remember, this is the same as an Azure location), then OpenNebula knows that the VMs sent to this host needs to be launched in that Azure datacenter.
  • If the VM doesn’t have a LOCATION attribute, and the host name doesn’t match any of the defined regions, then the default region is picked.

When you create a new host, the credentials and endpoint for that host are retrieved from the /etc/one/az_driver.conf file using the host name. Therefore, if you want to add a new host to manage a different datacenter, i.e. west-europe, just add your credentials and the capacity limits to the west-europe section in the configuration file, and specify that name (west-europe) when creating the new host.

regions:
    ...
    west-europe:
        region_name: "West Europe"
        pem_management_cert: "<path-to-your-vonecloud-pem-certificate-here>"
        subscription_id: "your-subscription-id"
        management_endpoint:
        capacity:
            Small: 5
            Medium: 1
            Large: 0

After that, create a new Host with the west-europe name:

onehost create west-europe -i az -v az

If the Host name does not match any region key, the default will be used.

You can define a different Azure section in your template for each Azure host, so with one template you can define different VMs depending on which host it is scheduled. Just include a LOCATION attribute in each PUBLIC_CLOUD section:

PUBLIC_CLOUD = [ TYPE=AZURE,
                 INSTANCE_TYPE=Small,
                 IMAGE=b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140606.1-en-us-30GB,
                 VM_USER="MyUserName",
                 VM_PASSWORD="MyPassword",
                 LOCATION="brazil-south"
]

PUBLIC_CLOUD = [ TYPE=AZURE,
                 INSTANCE_TYPE=Medium,
                 IMAGE=0b11de9248dd4d87b18621318e037d37__RightImage-Ubuntu-12.04-x64-v13.4,
                 VM_USER="MyUserName",
                 VM_PASSWORD="MyPassword",
                 LOCATION="west-europe"
]

You will have a small Ubuntu 14.04 VM launched when this VM template is sent to host brazil-south and a medium Ubuntu 13.04 VM launched whenever the VM template is sent to host west-europe.

Warning

If only one Azure host is defined, the Azure driver will deploy all Azure templates onto it, not paying attention to the LOCATION attribute.

Hybrid VM Templates

A powerful use of cloud bursting in OpenNebula is the ability to use hybrid templates, defining a VM if OpenNebula decides to launch it locally, and also defining it if it is going to be outsourced to Azure. The idea behind this is to reference the same kind of VM even if it is incarnated by different images (the local image and the Azure image).

An example of a hybrid template:

## Local Template section
NAME=MNyWebServer

CPU=1
MEMORY=256

DISK=[IMAGE="nginx-golden"]
NIC=[NETWORK="public"]

PUBLIC_CLOUD = [ TYPE=AZURE,
                 INSTANCE_TYPE=Medium,
                 IMAGE=0b11de9248dd4d87b18621318e037d37__RightImage-Ubuntu-12.04-x64-v13.4,
                 VM_USER="MyUserName",
                 VM_PASSWORD="MyPassword",
                 LOCATION="west-europe"
]

OpenNebula will use the first portion (from NAME to NIC) in the above template when the VM is scheduled to a local virtualization node, and the PUBLIC_CLOUD section of TYPE=”AZURE” when the VM is scheduled to an Azure node (i.e. when the VM is going to be launched in Azure).

Testing

You must create a template file containing the information of the VMs you want to launch.

CPU      = 1
MEMORY   = 1700

# KVM template machine, this will be use when submitting this VM to local resources
DISK     = [ IMAGE_ID = 3 ]
NIC      = [ NETWORK_ID = 7 ]

# Azure template machine, this will be use when submitting this VM to Azure

PUBLIC_CLOUD = [ TYPE=AZURE,
                 INSTANCE_TYPE=Medium,
                 IMAGE=0b11de9248dd4d87b18621318e037d37__RightImage-Ubuntu-12.04-x64-v13.4,
                 VM_USER="MyUserName",
                 VM_PASSWORD="MyPassword",
                 LOCATION="west-europe"
]

# Add this if you want to use only Azure cloud
#SCHED_REQUIREMENTS = 'HYPERVISOR = "AZURE"'

You can submit and control the template using the OpenNebula interface:

onetemplate create aztemplate
onetemplate instantiate aztemplate

Now you can monitor the state of the VM with

onevm list
    ID USER     GROUP    NAME         STAT CPU     MEM        HOSTNAME        TIME
     0 oneadmin oneadmin one-0        runn   0      0K     west-europe    0d 07:03

Also you can see information (like IP address) related to the Azure instance launched via the command. The attributes available are:

  • AZ_AVAILABILITY_SET_NAME
  • AZ_CLOUD_SERVICE_NAME,
  • AZ_DATA_DISKS,
  • AZ_DEPLOYMENT_NAME,
  • AZ_DISK_NAME,
  • AZ_HOSTNAME,
  • AZ_IMAGE,
  • AZ_IPADDRESS,
  • AZ_MEDIA_LINK,
  • AZ_OS_TYPE,
  • AZ_ROLE_SIZE,
  • AZ_TCP_ENDPOINTS,
  • AZ_UDP_ENDPOINTS,
  • AZ_VIRTUAL_NETWORK_NAME
onevm show 0
VIRTUAL MACHINE 0 INFORMATION
ID                  : 0
NAME                : one-0
USER                : oneadmin
GROUP               : oneadmin
STATE               : ACTIVE
LCM_STATE           : RUNNING
RESCHED             : No
START TIME          : 06/25 13:05:29
END TIME            : -
HOST                : west-europe
CLUSTER ID          : -1
DEPLOY ID           : one-0_opennebuladefaultcloudservicename-0


VIRTUAL MACHINE MONITORING
USED MEMORY         : 0K
USED CPU            : 0
NET_TX              : 0K
NET_RX              : 0K

PERMISSIONS
OWNER               : um-
GROUP               : ---
OTHER               : ---

VIRTUAL MACHINE HISTORY
SEQ HOST            ACTION             DS           START        TIME     PROLOG
  0 west-europe     none               -1  06/25 13:06:25   0d 00h06m   0h00m00s


USER TEMPLATE
PUBLIC_CLOUD=[
  IMAGE="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140606.1-en-us-30GB",
  INSTANCE_TYPE="ExtraSmall",
  SSH_PORT="2222",
  TCP_ENDPOINTS="80",
  TYPE="AZURE",
  VM_PASSWORD="MyVMPassword",
  VM_USER="MyUserName",
  WIN_RM="https" ]
VIRTUAL MACHINE TEMPLATE
AUTOMATIC_REQUIREMENTS="!(PUBLIC_CLOUD = YES) | (PUBLIC_CLOUD = YES & (HYPERVISOR = AZURE | HYPERVISOR = AZURE))"
AZ_CLOUD_SERVICE_NAME="opennebuladefaultcloudservicename-0"
AZ_DEPLOYMENT_NAME="OpenNebulaDefaultCloudServiceName-0"
AZ_DISK_NAME="OpenNebulaDefaultCloudServiceName-0-one-0_OpenNebulaDefaultCloudServiceName-0-0-201406251107210062"
AZ_HOSTNAME="ubuntu"
AZ_IMAGE="b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04-LTS-amd64-server-20140606.1-en-us-30GB"
AZ_IPADDRESS="191.233.70.93"
AZ_MEDIA_LINK="http://one0opennebuladefaultclo.blob.core.windows.net/vhds/disk_2014_06_25_13_07.vhd"
AZ_OS_TYPE="Linux"
AZ_ROLE_SIZE="ExtraSmall"
AZ_TCP_ENDPOINTS="name=SSH,vip=23.97.101.202,publicport=2222,local_port=22,local_port=tcp;name=TCP-PORT-80,vip=23.97.101.202,publicport=80,local_port=80,local_port=tcp"
CPU="1"
MEMORY="1024"
VMID="0"

Scheduler Configuration

Since Azure Hosts are treated by the scheduler like any other host, VMs will be automatically deployed in them. But you probably want to lower their priority and start using them only when the local infrastructure is full.

Configure the Priority

The Azure drivers return a probe with the value PRIORITY = -1. This can be used by the scheduler, configuring the ‘fixed’ policy in sched.conf:

DEFAULT_SCHED = [
    policy = 4
]

The local hosts will have a priority of 0 by default, but you could set any value manually with the onehost update or onecluster update commands.

There are two other parameters that you may want to adjust in sched.conf:

  • MAX_DISPATCH: Maximum number of Virtual Machines actually dispatched to a host in each scheduling action
  • MAX_HOST: Maximum number of Virtual Machines dispatched to a given host in each scheduling action

In a scheduling cycle, when MAX_HOST VMs have been deployed to a host, the host is discarded for the following pending VMs.

For example, having this configuration:

  • MAX_HOST = 1
  • MAX_DISPATCH = 30
  • 2 Hosts: 1 in the local infrastructure, and 1 using the Azure drivers
  • 2 pending VMs

The first VM will be deployed in the local host. The second VM will have also sort the local host with higher priority, but because 1 VM was already deployed, the second VM will be launched in Azure.

A quick way to ensure that your local infrastructure will always be used before the Azure hosts is to set MAX_DISPATCH to the number of local hosts.

Force a Local or Remote Deployment

The Azure drivers report the host attribute PUBLIC_CLOUD = YES. Knowing this, you can use that attribute in your VM requirements.

To force a VM deployment in a local host, use:

SCHED_REQUIREMENTS = "!(PUBLIC_CLOUD = YES)"

To force a VM deployment in a Azure host, use:

SCHED_REQUIREMENTS = "PUBLIC_CLOUD = YES"

Importing VMs

VMs running on Azure that were not launched through OpenNebula can be imported into OpenNebula.