OneFlow Services Management

OneFlow allows users and administrators to define, execute and manage multi-tiered applications, which we call Services, composed of interconnected Virtual Machines with deployment dependencies between them. Each group of Virtual Machines is deployed and managed as a single entity, and is completely integrated with the advanced OpenNebula user and group management.

What Is a Service

The following diagram represents a multi-tier application. Each node represents a Role, and its cardinality (the number of VMs that will be deployed). The arrows indicate the deployment dependencies: each Role’s VMs are deployed only when all its parent’s VMs are running.

image0

This Service can be represented with the following JSON template:

{
  "name": "my_service",
  "deployment": "straight",
  "ready_status_gate": true|false,
  "roles": [
    {
      "name": "frontend",
      "vm_template": 0
    },
    {
      "name": "db_master",
      "parents": [
        "frontend"
      ],
      "vm_template": 1
    },
    {
      "name": "db_slave",
      "parents": [
        "frontend"
      ],
      "cardinality": 3,
      "vm_template": 2
    },
    {
      "name": "worker",
      "parents": [
        "db_master",
        "db_slave"
      ],
      "cardinality": 10,
      "vm_template": 3
    }
  ]
}

Managing Service Templates

OneFlow allows OpenNebula administrators and users to register Service Templates in OpenNebula, to be instantiated later as Services. These Templates can be instantiated several times, and also shared with other users.

Users can manage the Service Templates using the command oneflow-template, or Sunstone. For each user, the actual list of Service Templates available is determined by the ownership and permissions of the Templates.

Create and List Existing Service Templates

The command oneflow-template create registers a JSON template file. For example, if the previous example template is saved in /tmp/my_service.json, you can execute:

oneflow-template create /tmp/my_service.json
ID: 0

You can also create Service Templates from Sunstone:

image1

To list the available Service Templates, use oneflow-template list/show/top:

$ oneflow-template list
        ID USER            GROUP           NAME
         0 oneadmin        oneadmin        my_service

$ oneflow-template show 0
SERVICE TEMPLATE 0 INFORMATION
ID                  : 0
NAME                : my_service
USER                : oneadmin
GROUP               : oneadmin

PERMISSIONS
OWNER               : um-
GROUP               : ---
OTHER               : ---

TEMPLATE CONTENTS
{
  "name": "my_service",
  "roles": [
    {

....

Templates can be deleted with oneflow-template delete.

Determining when a VM is READY

Depending on the deployment strategy, OneFlow will wait until all the VMs in a specific Role are all in running state before deploying VMs that belong to a child Role. How OneFlow determines the running state of the VMs can be specified with the checkbox Wait for VMs to report that the are READY available in the Service creation dialog in Sunstone, or the attribute in ready_status_gate in the top level of the Service Template JSON.

oneflow-ready-status-checkbox

If ready_status_gate is set to true, a VM will only be considered to be in running state the following points are true:

  • VM is in running state for OpenNebula. Which specifically means that LCM_STATE==3 and STATE>=3
  • The VM has READY=YES in the user template.

The idea is to report via OneGate from inside the VM that it’s running during the boot sequence:

curl -X "PUT" http://<onegate>/vm \
  --header "X-ONEGATE-TOKEN: ..." \
  --header "X-ONEGATE-VMID: ..." \
  -d "READY = YES"

This can also be done directly using OpenNebula’s interfaces: CLI, Sunstone or API.

If ready_status_gate is set to false, a VM will be considered to be in running state when it’s in running state for OpenNebula (LCM_STATE==3 and STATE>=3). Take into account that the VM will be considered RUNNING the very same moment the hypervisor boots the VM (before it loads the OS).

Configure Dynamic Networks

Each Service Role has a Virtual Machine Template assigned. The VM Template will define the capacity, disks, and network interfaces. But instead of using the Virtual Networks set in the VM Template, the Service Template can define a set of dynamic networks.

oneflow-templates-net-1

Each Role can be attached to the dynamic networks individually.

oneflow-templates-net-2

When a Service Template defines dynamic networks, the instantiate dialog will ask the user to select the networks to use for the new Service.

oneflow-templates-net-3

This allows you to create more generic Service Templates. For example, the same Service Template can be used by users of different groups that may have access to different Virtual Networks.

Managing Services

A Service Template can be instantiated as a Service. Each newly created Service will be deployed by OneFlow following its deployment strategy.

Each Service Role creates Virtual Machines in OpenNebula from VM Templates, that must be created beforehand.

Create and List Existing Services

New Services are created from Service Templates, using the oneflow-template instantiate command:

oneflow-template instantiate 0
ID: 1

To list the available Services, use oneflow list/top:

oneflow list
        ID USER            GROUP           NAME                      STATE
         1 oneadmin        oneadmin        my_service                PENDING

image3

The Service will eventually change to DEPLOYING. You can see information for each Role and individual Virtual Machine using oneflow show

$ oneflow show 1
SERVICE 1 INFORMATION
ID                  : 1
NAME                : my_service
USER                : oneadmin
GROUP               : oneadmin
STRATEGY            : straight
SERVICE STATE       : DEPLOYING

PERMISSIONS
OWNER               : um-
GROUP               : ---
OTHER               : ---

ROLE frontend
ROLE STATE          : RUNNING
CARNIDALITY         : 1
VM TEMPLATE         : 0
NODES INFORMATION
 VM_ID NAME                    STAT UCPU    UMEM HOST                       TIME
     0 frontend_0_(service_1)  runn   67  120.3M localhost              0d 00h01

ROLE db_master
ROLE STATE          : DEPLOYING
PARENTS             : frontend
CARNIDALITY         : 1
VM TEMPLATE         : 1
NODES INFORMATION
 VM_ID NAME                    STAT UCPU    UMEM HOST                       TIME
     1                         init           0K                        0d 00h00

ROLE db_slave
ROLE STATE          : DEPLOYING
PARENTS             : frontend
CARNIDALITY         : 3
VM TEMPLATE         : 2
NODES INFORMATION
 VM_ID NAME                    STAT UCPU    UMEM HOST                       TIME
     2                         init           0K                        0d 00h00
     3                         init           0K                        0d 00h00
     4                         init           0K                        0d 00h00

ROLE worker
ROLE STATE          : PENDING
PARENTS             : db_master, db_slave
CARNIDALITY         : 10
VM TEMPLATE         : 3
NODES INFORMATION
 VM_ID NAME                    STAT UCPU    UMEM HOST                       TIME



LOG MESSAGES
09/19/12 14:44 [I] New state: DEPLOYING

Life-cycle

The deployment attribute defines the deployment strategy that the Life Cycle Manager (part of the oneflow-server) will use. These two values can be used:

  • none: All Roles are deployed at the same time.
  • straight: Each Role is deployed when all its parent Roles are RUNNING.

Regardless of the strategy used, the Service will be RUNNING when all of the Roles are also RUNNING. Likewise, a Role will enter this state only when all the VMs are running.

image4

This table describes the Service states:

Service State Meaning
PENDING The Service starts in this state, and will stay in it until the LCM decides to deploy it
DEPLOYING Some Roles are being deployed
RUNNING All Roles are deployed successfully
WARNING A VM was found in a failure state
SCALING A Role is scaling up or down
COOLDOWN A Role is in the cooldown period after a scaling operation
UNDEPLOYING Some Roles are being undeployed
DONE The Service will stay in this state after a successful undeployment. It can be deleted
FAILED_DEPLOYING An error occurred while deploying the Service
FAILED_UNDEPLOYING An error occurred while undeploying the Service
FAILED_SCALING An error occurred while scaling the Service

Each Role has an individual state, described in the following table:

Role State Meaning
PENDING The Role is waiting to be deployed
DEPLOYING The VMs are being created, and will be monitored until all of them are running
RUNNING All the VMs are running
WARNING A VM was found in a failure state
SCALING The Role is waiting for VMs to be deployed or to be shutdown
COOLDOWN The Role is in the cooldown period after a scaling operation
UNDEPLOYING The VMs are being shutdown. The Role will stay in this state until all VMs are done
DONE All the VMs are done
FAILED_DEPLOYING An error occurred while deploying the VMs
FAILED_UNDEPLOYING An error occurred while undeploying the VMs
FAILED_SCALING An error occurred while scaling the Role

Life-Cycle Operations

Services are deployed automatically by the Life Cycle Manager. To undeploy a running Service, users can use the commands oneflow shutdown and oneflow delete.

The command oneflow shutdown will perform a graceful a terminate on all the running VMs (see onevm terminate). If the straight deployment strategy is used, the Roles will be shutdown in the reverse order of the deployment.

After a successful shutdown, the Service will remain in the DONE state. If any of the VM terminate operations cannot be performed, the Service state will show FAILED, to indicate that manual intervention is required to complete the cleanup. In any case, the Service can be completely removed using the command oneflow delete.

If a Service and its VMs must be immediately undeployed, the command oneflow delete can be used from any Service state. This will execute a terminate operation for each VM and delete the Service. Please be aware that this is not recommended, because failed terminate actions may leave VMs in the system.

When a Service fails during a deployment, undeployment or scaling operation, the command oneflow recover can be used to retry the previous action once the problem has been solved.

Elasticity

A Role’s cardinality can be adjusted manually, based on metrics, or based on a schedule. To start the scalability immediately, use the command oneflow scale:

oneflow scale <serviceid> <role_name> <cardinality>

To define automatic elasticity policies, proceed to the elasticity documentation guide.

Sharing Information between VMs

The Virtual Machines of a Service can share information with each other, using the OneGate server. OneGate allows Virtual Machine guests to push information to OpenNebula, and pull information about their own VM or Service.

From any VM, use the PUT ${ONEGATE_ENDPOINT}/vm action to store any information in the VM user template. This information will be in the form of attribute=vale, e.g. ACTIVE_TASK = 13. Other VMs in the Service can request that information using the GET ${ONEGATE_ENDPOINT}/service action.

You can read more details in the OneGate API documentation.

Managing Permissions

Both Services and Template resources are completely integrated with the OpenNebula user and group management. This means that each resource has an owner and group, and permissions. The VMs created by a Service are owned by the Service owner, so he can list and manage them.

For example, to change the owner and group of the Service 1, we can use oneflow chown/chgrp:

$ oneflow list
        ID USER            GROUP           NAME                      STATE
         1 oneadmin        oneadmin        my_service                RUNNING

$ onevm list
    ID USER     GROUP    NAME            STAT UCPU    UMEM HOST             TIME
     0 oneadmin oneadmin frontend_0_(ser runn   17   43.5M localhost    0d 01h06
     1 oneadmin oneadmin db_master_0_(se runn   59  106.2M localhost    0d 01h06
...

$ oneflow chown my_service johndoe apptools

$ oneflow list
        ID USER            GROUP           NAME                      STATE
         1 johndoe         apptools        my_service                RUNNING

$ onevm list
    ID USER     GROUP    NAME            STAT UCPU    UMEM HOST             TIME
     0 johndoe  apptools frontend_0_(ser runn   62   83.2M localhost    0d 01h16
     1 johndoe  apptools db_master_0_(se runn   74  115.2M localhost    0d 01h16
...

Note

The Service’s VM ownership is also changed.

All Services and Templates have associated permissions for the owner, the users in its group, and others. For each one of these groups, there are three rights that can be set: USE, MANAGE and ADMIN. These permissions are very similar to those of UNIX file system, and can be modified with the command chmod.

For example, to allow all users in the apptools group to USE (list, show) and MANAGE (shutdown, delete) the Service 1:

$ oneflow show 1
SERVICE 1 INFORMATION
..

PERMISSIONS
OWNER               : um-
GROUP               : ---
OTHER               : ---
...

$ oneflow chmod my_service 660

$ oneflow show 1
SERVICE 1 INFORMATION
..

PERMISSIONS
OWNER               : um-
GROUP               : um-
OTHER               : ---
...

Another common scenario is having Service Templates created by oneadmin that can be instantiated by any user. To implement this scenario, execute:

$ oneflow-template show 0
SERVICE TEMPLATE 0 INFORMATION
ID                  : 0
NAME                : my_service
USER                : oneadmin
GROUP               : oneadmin

PERMISSIONS
OWNER               : um-
GROUP               : ---
OTHER               : ---
...

$ oneflow-template chmod 0 604

$ oneflow-template show 0
SERVICE TEMPLATE 0 INFORMATION
ID                  : 0
NAME                : my_service
USER                : oneadmin
GROUP               : oneadmin

PERMISSIONS
OWNER               : um-
GROUP               : ---
OTHER               : u--
...

Please refer to the OpenNebula documentation for more information about users & groups, and resource permissions.

Scheduling Actions on the Virtual Machines of a Role

You can use the action command to perform a VM action on all the Virtual Machines belonging to a Role. For example, if you want to suspend the Virtual Machines of the worker Role:

$ oneflow action <service_id> <role_name> <vm_action>

These are the commands that can be performed:

  • terminate
  • terminate-hard
  • undeploy
  • undeploy-hard
  • hold
  • release
  • stop
  • suspend
  • resume
  • reboot
  • reboot-hard
  • poweroff
  • poweroff-hard
  • snapshot-create

Instead of performing the action immediately on all the VMs, you can perform it on small groups of VMs with these options:

  • -p, --period x: Seconds between each group of actions
  • -n, --number x: Number of VMs to apply the action to each period

Let’s say you need to reboot all the VMs of a Role, but you also need to avoid downtime. This command will reboot 2 VMs each 5 minutes:

oneflow action my-service my-role reboot --period 300 --number 2

The /etc/one/oneflow-server.conf file contains default values for period and number that are used if you omit one of them.

Recovering from Failures

Some common failures can be resolved without manual intervention, calling the oneflow recover command. This command has different effects depending on the Service state:

State New State Recover action
FAILED_DEPLOYING DEPLOYING

VMs in DONE or FAILED are terminated.

VMs in UNKNOWN are booted.

FAILED_UNDEPLOYING UNDEPLOYING The undeployment is resumed.
FAILED_SCALING SCALING

VMs in DONE or FAILED are terminated.

VMs in UNKNOWN are booted.

For a scale-down, the shutdown actions are retried.

COOLDOWN RUNNING The Service is simply set to running before the cooldown period is over.
WARNING WARNING

VMs in DONE or FAILED are terminated.

VMs in UNKNOWN are booted.

New VMs are instantiated to maintain the current cardinality.

Service Template Reference

For more information on the resource representation, please check the API guide

Read the elasticity policies documentation for more information.