stoney conductor: VM Backup

Overview

This page describes how the VMs and VM-Templates are backed-up inside the stoney cloud.

Basic idea

The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:

• Snapshot: Save the machines state (CPU, Memory and Disk)
• Merge: Merge the Disk-Image-Snapshot with the Live-Image
• Retain: Export the snapshot files

A more detailed and technical description for these three sub-processes can be found here.

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:

Backup a single machine

The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:

object machine = args[0];
 
if( snapshot( machine ) )
{
 
    if ( merge( machine ) )
    {
 
        if ( retain( machine ) )
        {
            printf("Successfully backed up machine %s\n", machine);
 
        } else
        {
            printf("Error while retaining machine %s: %s\n", machine, error);
        }
 
    } else
    {
        printf("Error while merging machine %s: %s\n", machine, error);
    }
 
} else
{
    printf("Error while snapshotting machine %s: %s\n", machine, error);
}

Backup multiple machines at the same time

When backing up multiple machines at the same time, we need to make sure that the downtime for the machines are as close together as possible. Therefore the control instance should call first the snapshot process for all machines. After every machine has been snapshotted, the control instance can call the merge and retain process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the merge and retain process. So the control instance needs a little bit more logic:

object machines[] = args[0];
object successful_snapshots[];
 
# Snapshot all machines
for( int i = 0; i <  sizeof(machines) / sizeof(object) ; i++ )
{
    # If the snapshot was successful, put the machine into the 
    # successful_snapshots array
    if ( snapshot( machines[i] ) )
    {
        successful_snapshots[machines[i]];
    } else
    {
        printf("Error while snapshotting machine %s: %s\n", machines[i],error);
    }
}
 
# Merge and reatin all successful_snapshot machines
for ( int i = 0; i <  sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
    # Check if the element at this position is not null, then the snapshot 
    # for this machine was successful
    if ( successful_snapshots[i] )
    {
        if ( merge( successful_snapshots[i] ) )
        {
            if ( retain( successful_snapshots[i] ) )
            {
              printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
            } else
            {
                printf("Error while retaining machine %s: %s\n", successful_snapshots[i],error);
            }
 
        } else
        {
            printf("Error while merging machine %s: %s\n", successful_snapshots[i],error);
        }
    }
}

Sub-Processes

Snapshot

1. Create a snapshot with state:
   • If the VM vm-001 is running:
     • Save the state of VM vm-001 to the file vm-001.state (This file can either be created on a RAM-Disk or directly in the retain location. This example however saves the file to a RAM-Disk):

       virsh save vm-001 /path/to/ram-disk/vm-001.state

     • After this command, the VMs CPU and memory state is represented by the file /path/to/ram-disk/vm-001.state and the VM vm-001 is shut down.
   • If the VM vm-001 is shut down:
     • Create a fake state file for the VM:

       echo "Machine is not runnung, no state file" > /path/to/ram-disk/vm-001.state

2. Move the disk image /path/to/images/vm-001.qcow2 to the retain location:

   mv /path/to/images/vm-001.qcow2 /path/to/retain/vm-001.qcow2

   • Please note: The retain directory (/path/to/retain/) has to be on the same partition as the images directory (/path/to/images/). This will make the mv operation very fast (only renaming the inode). So the downtime (remember the VM vm-001 is shut down) is as short as possible.
   • Please note²: If the VM vm-001 has more than just one disk-image, repeat this step for every disk-image
3. Create the new (empty) disk image with the old as backing store file:

   qemu-img create -f qcow2 -b /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2

   • Please note: If the VM vm-001 has more than just one disk-image, repeat this step for every disk-image
4. Set correct ownership and permission to the newly created image:

   • chmod 660 /path/to/images/vm-001.qcow2

   • chown root:vm-storage /path/to/images/vm-001.qcow2

   • Please note: If the VM vm-001 has more than just one disk-image, repeat these steps for every disk-image
5. Save the VMs XML description
   • Save the current XML description of VM vm-001 to a file at the retain location:

     virsh dumpxml vm-001 > /path/to/retain/vm-001.xml

6. Save the backend entry
   • There is no generic command to save the backend entry (since the command depends on the backend). Important here is, that the backend entry of the VM vm-001 is saved to the retain location: /path/to/retain/vm-001.backend
7. Restore the VMs vm-001 from its saved state (this will also start the VM):

   virsh restore /path/to/ram-disk/vm-001.state

   • Please note: After this operation the VM vm-001 is running again (continues where we stopped it), and we have a consistent backup for the VM vm-001:
     • The file /path/to/ram-disk/vm-001.state contains the CPU and memory state of VM vm-001 at time T₁
     • The file /path/to/retain/vm-001.qcow2 contains the disk state of VM vm-001 at time T₁
       • Important: Remember: The live-disk-image /path/to/images/vm-001.qcow2 still contains a reference to this file!! So you cannot delete or move it!!!
     • The file /path/to/retain/vm-001.xml contains the XML description of VM vm-001 at time T₁
     • The file /path/to/retain/vm-001.backend contains the backend entry of VM vm-001 at time T₁
8. Move the state file from the RAM-Disk to the retain location (if you used the RAM-Disk to save the VMs state)

   • mv /path/to/ram-disk/vm-001.state /path/to/retain/vm-001.state

See also: Snapshot workflow

Merge

1. Check if the VM vm-001 is running
   • If not, start the VM in paused state:

     virsh start --paused vm-001

2. Merge the live-disk-image (/path/to/images/vm-001.qcow2) with its backing store file (/path/to/retain/vm-001.qcow2):

   virsh qemu-monitor-command vm-001 --hmp "block_stream drive-virtio-disk0"

   • Please note: If a VM has more than just one disk-image, repeat this step for every image. Just increase the number at the end of the command. So command to merge the second disk image would be:

     virsh qemu-monitor-command vm-001 --hmp "block_stream drive-virtio-disk1"

3. If the machine is running in paused state (means we started it in 1. because it was not running), stop it again:

   • virsh shutdown vm-001

Please note: After these steps, the live-disk-image /path/to/image/vm-001.qcow2 no longer contains a reference to the image at the retain location (/path/to/retain/vm-001.qcow2). This is important for the retain process.

See also: Merge workflow

Retain

1. Move the all the files in from the retain directory (/path/to/retain/) to the backup directory (/path/to/backup/)
   1. Move the VMs state file to the backup directory

      • mv /path/to/retain/vm-001.state /path/to/backup/vm-001.state

   2. Move the VMs disk image to the backup directory

      • mv /path/to/retain/vm-001.qcow2 /path/to/backup/vm-001.qcow2

        • Please note: If the VM vm-001 has more than just one disk image, repeat this step for each disk image
   3. Move the VMs XML description file to the backup directory

      • mv /path/to/retain/vm-001.xml /path/to/backup/vm-001.xml

   4. Move the VMs backend entry file to the backup directory

      • mv /path/to/retain/vm-001.backend /path/to/backup/vm-001.backend

See also Retain workflow

Communication through backend

Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend