Upgrading from OpenNebula 5.8.x

This section describes the installation procedure for systems that are already running a 5.8.x OpenNebula. The upgrade to OpenNebula 5.8 can be done directly following this section, you don’t need to perform intermediate version upgrades. The upgrade will preserve all current users, hosts, resources and configurations; for both Sqlite and MySQL backends.

Read the Compatibility Guide and Release Notes to know what is new in OpenNebula 5.8.

Upgrading a Federation and High Availability

You need to perform the following steps in all the HA nodes and all zones. Note that you need to update all the servers at the same time, not one by one.

Preparation

Before proceeding, make sure you don’t have any VMs in a transient state (prolog, migr, epil, save). Wait until these VMs get to a final state (runn, suspended, stopped, done). Check the Managing Virtual Machines guide for more information on the VM life-cycle.

Stop OpenNebula

Stop OpenNebula and any other related services you may have running: OneFlow, EC2, and Sunstone. Use preferably the system tools, like systemctl or service as root in order to stop the services.

Backup

Backup the configuration files located in /etc/one. You don’t need to do a manual backup of your database, the onedb command will perform one automatically.

cp -r /etc/one /etc/one.$(date +'%Y-%m-%d')

Installation of New Version

Follow the Platform Notes and the Installation guide, taking into account that you will already have configured the passwordless ssh access for oneadmin.

Make sure to run the install_gems tool, as the new OpenNebula version may have different gem requirements.

Note

If executing install_gems you get a message asking to overwrite files for aws executables you can safely answer “yes”.

It is highly recommended not to keep your current oned.conf, and update the oned.conf file shipped with OpenNebula 5.8 to your setup. If for any reason you plan to preserve your current oned.conf file, read the Compatibility Guide and the complete oned.conf 5.8 reference.

Configuration Files Upgrade

If you haven’t modified any configuration files, the package managers will replace the configuration files with their newer versions and no manual intervention is required.

If you have customized any configuration files under /etc/one we recommend you to follow these steps regardless of the platform/linux distribution.

  1. Backup /etc/one (already performed)
  2. Install the new packages (already performed)
  3. Compare the old and new configuration files: diff -ur /etc/one.YYYY-MM-DD /etc/one. Or you can use graphical diff-tools like meld to compare both directories, which are very useful in this step.
  4. Edit the new files and port all the customizations from the previous version.
  5. You should never overwrite the configuration files with older versions.

Note

Configuration files from inside the remote scripts directory structure /var/lib/one/remotes/ have been moved into dedicated directory /var/lib/one/remotes/etc/. Check all the files on the new path, and apply any necessary changes to your environment.

Database Upgrade

Perform the Database Upgrade

The database schema and contents are incompatible between versions. The OpenNebula daemon checks the existing DB version, and will fail to start if the version found is not the one expected, with the message ‘Database version mismatch’.

You can upgrade the existing DB with the ‘onedb’ command. You can specify any Sqlite or MySQL database. Check the onedb reference for more information.

Note

Make sure at this point that OpenNebula is not running. If you installed from packages, the service may have been started automatically.

After you install the latest OpenNebula, and fix any possible conflicts in oned.conf, you can issue the ‘onedb upgrade -v’ command. The connection parameters have to be supplied with the command line options, see the onedb manpage for more information. Some examples:

onedb upgrade -v --sqlite /var/lib/one/one.db
onedb upgrade -v -S localhost -u oneadmin -p oneadmin -d opennebula

If everything goes well, you should get an output similar to this one:

$ onedb upgrade -v -u oneadmin -d opennebula
MySQL Password:
Version read:
Shared tables 5.6.0 : Database migrated from 5.5.80 to 5.6.0 (OpenNebula 5.6.0) by onedb command.
Local tables  5.6.0 : Database migrated from 5.5.80 to 5.6.0 (OpenNebula 5.6.0) by onedb command.

Warning: Using a password on the command line interface can be insecure.
MySQL dump stored in /var/lib/one/mysql_localhost_opennebula_2019-2-26_16:18:59.sql
Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file


>>> Running migrators for shared tables
  ...

>>> Running migrators for local tables
  ...
  > Done in 0.00s

Database migrated from 5.6.0 to 5.8.0 (OpenNebula 5.8.0) by onedb command.

Total time: 342.99s

Note

Make sure you keep the backup file. If you face any issues, the onedb command can restore this backup, but it won’t downgrade databases to previous versions.

Check DB Consistency

After the upgrade is completed, you should run the command onedb fsck.

First, move the 5.6.x backup file created by the upgrade command to a safe place.

mv /var/lib/one/mysql_localhost_opennebula.sql /path/for/one-backups/

Then execute the following command:

$ onedb fsck -S localhost -u oneadmin -p oneadmin -d opennebula
MySQL dump stored in /var/lib/one/mysql_localhost_opennebula.sql
Use 'onedb restore' or restore the DB using the mysql command:
mysql -u user -h server -P port db_name < backup_file

Total errors found: 0

Reload Start Scripts

Follow this section if you are using a systemd base distribution, like CentOS 7+, Ubuntu 16.04+, etc.

In order for the system to re-read the configuration files you should issue the following command after the installation of the new packages:

systemctl daemon-reload

Update the Drivers

You should be able now to start OpenNebula as usual, running service opennebula start as root. At this point, as oneadmin user, execute onehost sync to update the new drivers in the hosts.

Warning

Doing onehost sync is important. If the monitorization drivers are not updated, the hosts will behave erratically.

Note

You can skip this step if you are not using KVM hosts, or any hosts that use remove monitoring probes.

Testing

OpenNebula will continue the monitoring and management of your previous Hosts and VMs.

As a measure of caution, look for any error messages in oned.log, and check that all drivers are loaded successfully. After that, keep an eye on oned.log while you issue the onevm, onevnet, oneimage, oneuser, onehost list commands. Try also using the show subcommand for some resources.

Restoring the Previous Version

If for any reason you need to restore your previous OpenNebula, follow these steps:

  • With OpenNebula 5.8 still installed, restore the DB backup using ‘onedb restore -f’
  • Uninstall OpenNebula 5.8, and install again your previous version.
  • Copy back the backup of /etc/one you did to restore your configuration.

Known Issues

If the MySQL database password contains special characters, such as @ or #, the onedb command will fail to connect to it.

The workaround is to temporarily change the oneadmin’s password to an ASCII string. The set password statement can be used for this:

$ mysql -u oneadmin -p

mysql> SET PASSWORD = PASSWORD('newpass');

Bug recovering

If Ceph datastores were used with OpenNebula <= 5.6.2 and any VM have been reverted to a snapshot, it’s needed to follow the next steps for recovering snapshot tree consistency:

Warning

Check history in order to find how many reverts have been done. If the number of reverts are greater than 1 we do not recommend to deleted any snapshot, becase it will cause lose of information. If the number of revert is 1 you can fix it by following the steps below.

  • Use the command onedb update-body vm --id <vm_id> for updating the body of the VM.
  • Set /VM/SNAPSHOTS/CURRENT_BASE to the ID of the current active snapshot.