stoney-cloud.org - User contributions [en]

File:wrapper-interaction.xmi

2014-06-27T12:59:04Z

Pat: Initial version

Initial version

stoney conductor: VM Backup

2014-06-27T12:58:47Z

Pat: /* Current Implementation (Backup) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (create-, export- and commitSnapshot) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

You can modify/update this workflow by editing [[File:Daemon-communication.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commit command, it sets the '''sstProvisioningMode''' to '''comitted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the comitting of the changes from the overlay disk-images back to the underlying ones is done.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: comitted
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''committed''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the KVMBackupWrapper.pl script with the machines list as a parameter
# Wait for the KVMBackupWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

You can modify/update this workflow by editing [[File:wrapper-interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
'''Attention:''' The restore process is not yet defined / nor implemented. The following documentation is about the old restore process.
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
'''Attention''': The restore process is not yet defined / nor implemented. The following documentation is about the old restore process.

* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#Current_Implementation_.28Backup.29]]), the restore process does not work when clicking the icon in the webinterface.

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

File:Daemon-communication.xmi

2014-06-27T12:58:02Z

Pat: Pat uploaded a new version of "File:Daemon-communication.xmi": Correct workflow

Initial Version

File:Daemon-communication.xmi

2014-06-27T12:57:15Z

Pat: Initial Version

Initial Version

stoney conductor: VM Backup

2014-06-27T12:55:51Z

Pat: /* Communication through backend */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (create-, export- and commitSnapshot) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

You can modify/update this workflow by editing [[File:Daemon-communication.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commit command, it sets the '''sstProvisioningMode''' to '''comitted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the comitting of the changes from the overlay disk-images back to the underlying ones is done.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: comitted
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''committed''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the KVMBackupWrapper.pl script with the machines list as a parameter
# Wait for the KVMBackupWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
'''Attention:''' The restore process is not yet defined / nor implemented. The following documentation is about the old restore process.
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
'''Attention''': The restore process is not yet defined / nor implemented. The following documentation is about the old restore process.

* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#Current_Implementation_.28Backup.29]]), the restore process does not work when clicking the icon in the webinterface.

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

File:node-integration.xmi

2014-06-27T12:54:49Z

Pat: Initial version

Initial version

stoney cloud: Multi-Node Installation

2014-06-27T12:54:34Z

Pat: /* Node Integration */

== Overview ==
The [http://www.stoney-cloud.org/ stoney cloud] builds upon various standard open source components and can be run on commodity hardware. The final Multi-Node Setup consists of the following components:
* One [[Primary-Master-Node]] with an OpenLDAP Directory Server for the storage of the stoney cloud user and service related data with the web based management [[VM-Manager]] interface and the Linux kernel based virtualization technology.
* One [[Secondary-Master-Node]] with the Linux kernel based virtualization technology.
* Two [[Storage-Node | Storage-Nodes]] configured as a replicated and distributed data storage service based on [http://www.gluster.org/ GlusterFS].
The components communicate with each other over a standard Ethernet based IPv4 network.

=== Prerequisites ===
The following items and conditions are required to be able to install and configure a stoney cloud environment:
* Dedicated Hardware (4 Servers) which fulfil the following requirements:
** 64-Bit Intel with VT-Technologie (AMD is not tested at the moment).
** 8 Gigabyte Memory (more is better).
** 147 Gigabyte up to 2 Terabyte disks.
*** For all four Nodes ([[VM-Node | VM-Nodes]] and [[Storage-Node | Storage-Nodes]]) two 147 Gigabyte Disks, configured as RAID1, are enough for the [http://www.gentoo.org/ Gentoo Linux] operating system.
*** Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.
** Two physical Ethernet Interfaces, which support the same bandwidth (for example 1 Gigabit/s).
* Two Gigabit layer-2 switches supporting [http://en.wikipedia.org/wiki/IEEE_802.3ad IEEE 802.3ad] (dynamic link aggregation), [http://en.wikipedia.org/wiki/IEEE_802.1Q IEEE 802.1Q] (VLAN tagging) and stacking (optional, but recommended).
* Experience with Linux environments especially with [http://www.gentoo.org Gentoo Linux].
* Good experience with IP networking, because the Switches need to be configured manually (dynamic link aggregation and VLAN tagging).

=== Limitations ===
During the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system, the first two physical Ethernet Interfaces are automatically configured as a logical interface (bond0) and the four tagged VLANs are set up. If more than two physical Ethernet Interfaces are to be included in to the logical interface (bond0) or the physical Ethernet Interfaces have different bandwidths (for example 1 Gigabit/s and 10 Gigabit/s), the final logical interface (bond0) needs to be configured manually after the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system.
Only the first two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] are set up automatically. More [[Storage-Node | Storage-Nodes]] need to be integrated manually.

=== Network Overview ===
As stated before, a minimal multi node stoney cloud environment consist of two VM- and Storage-Nodes.
It is highly recommended to use IEEE 802.3ad link aggregation (bonding, trunking etc.) over two network cards and attach them as one logical link to the access switch.

It is out of scope of this document on how to configure the switches for stacking, link aggregation and VLAN tagging, consult the respective user manual.

There are two scenarios on how to connect the nodes to the network.

==== Network Overview: Physical Layer Scenario 1 (recommended) ====
The preferred solution is to use two switches which are stackable (supporting link aggregation over two switches). Connect the nodes and switches as illustrated below. Thus eliminating the single point of failure as presented in scenario 2.

<pre>
+----------------------+ +----------------------+
| | | |
| vm-node-01 | | vm-node-02 |
| | | |
+----------------------+ +----------------------+
bond0 | \ / | bond0
| \ / |
| \ / |
| \/ | ___
| /\ | ___( )___
| / \ | __( )__
| / \ | _( )_
+-------------+ +-------------+ _( )_
| switch-01 |===| switch-02 |-------------------(_ Corporate LAN/WAN _)
+-------------+ +-------------+ (_ _)
| \ / | (__ __)
| \ / | (___ ___)
| \ / | (___)
| \/ |
| /\ |
| / \ |
bond0 | / \ | bond0
+-----------------------+ +-----------------------+
| | | |
| tier1-storage-node-01 | | tier1-storage-node-02 |
| | | |
+-----------------------+ +-----------------------+
</pre>

==== Network Overview: Physical Layer Scenario 2 (use at your own risk) ====
If theres only one switch available (or the switches aren't stackable) connect the nodes as illustrated below. As you can see, the switch is as a single point of failure.
<pre>
+----------------------+ +----------------------+
| | | |
| vm-node-01 | | vm-node-02 |
| | | |
+----------------------+ +----------------------+ ___
bond0 \\ // bond0 ___( )___
\\ // __( )__
\\ // _( )_
+-------------+ _( )_
| switch-01 |-----------------------------(_ Corporate LAN/WAN _)
+-------------+ (_ _)
// \\ (__ __)
// \\ (___ ___)
bond0 // \\ bond0 (___)
+-----------------------+ +-----------------------+
| | | |
| tier1-storage-node-01 | | tier1-storage-node-02 |
| | | |
+-----------------------+ +-----------------------+
</pre>

==== Network overview: Logical layer ====
The goal is to achieve the following configuration:
<pre>
+----------------+----------------+----------------+----------------+
| 10.1.110.1X | 10.1.120.1X | 10.1.130.1X | 192.168.140.1X |
+----------------+----------------+----------------+----------------+
| | | | vmbr0 |
| vlan110 | vlan120 | vlan130 +----------------+
| | | | vlan140 |
+----------------+----------------+----------------+----------------+
+-------------------------------------------------------------------+
| bond0 (bonding.mode=802.3ad) |
+-------------------------------------------------------------------+
+----------------+ +----------------+
| eth0 | | eth1 |
+----------------+ +----------------+
</pre>

The ideal stoney cloud environment is based on four logical separated VLANs (virtual LANs):
* '''admin''': Administrative network, used for administration and monitoring purposes.
* '''data''': Data network, used for GlusterFS traffic.
* '''int''': Internal network, used for internal traffic such as LDAP, libvirt and more.
* '''pub''': Public network, used for accessing the VM-Manager webinterface, Spice traffic and internet access.

Each of the above VLANs hold dedicated services and separates them from each other. This documentation assumes, that the four VLANs are present and the following IP networks are available:

{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|VLAN name
!align="left"|VLAN ID
!align="left"|Network prefix
!align="left"|Default Gateway address
!align="left"|Broadcast address
!align="left"|Domain name
!align="left"|VIP
|-

|admin
|110
|10.1.110.0/24
| --
|10.1.110.255
|admin.example.com
|
|-

|data
|120
|10.1.120.0/24
| --
|10.1.120.255
|data.example.com
|
|-

|int
|130
|10.1.130.0/24
| --
|10.1.130.255
|int.example.com
|10.1.130.10
|-

|pub
|140
|192.168.140.0/24
|192.168.140.1
|192.168.140.255
|example.com
|192.168.140.10
|-

|}

The IP allocation of the nodes will be assumed as stated in the table below:
{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|Node name
!align="left"|Admin address (VLAN 110)
!align="left"|Data address (VLAN 120)
!align="left"|Int address (VLAN 130)
!align="left"|Pub address (VLAN 140)
|-

|-
|tier1-storage-node-01
|10.1.110.11
|10.1.120.11
|10.1.130.11
|192.168.140.11
|-

|-
|tier1-storage-node-02
|10.1.110.12
|10.1.120.12
|10.1.130.12
|192.168.140.12
|-

|-
|vm-node-01
|10.1.110.13
|10.1.120.13
|10.1.130.13
|192.168.140.13
|-

|-
|vm-node-02
|10.1.110.14
|10.1.120.14
|10.1.130.14
|192.168.140.14
|-

|}

We'll also presume, we have the following Domain Name Servers:
* Domain Name Server 1: '''192.168.140.1'''

== Base Installation ==
All nodes are based on the [http://www.gentoo.org/ Gentoo Linux] operating system.

=== BIOS Set Up Checklist ===
* Enable the "reboot-after-power-loss" option (if your BIOS supports it).
* Make sure, that you have the newest BIOS (BMC and perhaps SCSI firmware) version.
* Make sure, you've disabled halt on post errors (or similar) or enable keyboard-less operation (if your BIOS supports it).

=== RAID Set Up ===
Create a RAID1 volume. This RAID-Set is used for the Operating System. Please be aware, the the current stoney cloud only supports 147 Gigabyte up to 2 Terabyte disks for this first RAID-Set.

Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.

=== Node Installation ===
The first step of Semi-Automatic Multi-Node Set Up is the same for all four Nodes. In this documentation we presume, that you stick to the naming convention mentioned above. After the Base Installation of the Nodes, the following daemons will be running:
* '''crond''': Crond, to execute scheduled commands
* '''ntpd''': Network Time Protocol daemon, keeps the time synced with the Time from Servers in the LAN or WAN.
* '''sshd''': OpenSSH SSH daemon, used for remote access and remote administration.
* '''syslogd''': System Logging, keeps track of messages from the System and the Applications.
* '''udevd''': Linux dynamic device management, that manage events, symlinks and permissions of devices.

==== First Storage-Node (tier1-storage-node-01) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions as follows (the bold values are examples can be set through the administrator and are variable, according to the local setup):
## Global Section
### Confirm that you want to start? '''yes'''
### Choose a Node-Type: '''Storage-Node'''
### Choose a Block-Device: '''sda'''
### Confirm to erase all (from a previous installation): '''yes'''
### Confirm to continue with the given the Partition-Scheme: '''yes'''
### Choose the network interfaces to bond together.
#### Device #0: '''eth0'''
#### Device #1: '''eth1'''
### Node-Name: '''tier1-storage-node-01'''
## pub-VLAN Section
### VLAN ID: '''140'''
### Domain Name: '''example.com'''
### IP Address: '''192.168.140.11'''
### Netmask: '''24'''
### Broadcast: '''192.168.140.255'''
### Confirm the pub-VLAN Section with '''yes'''
## admin-VLAN Section
### VLAN ID: '''110'''
### Domain Name: '''admin.example.com'''
### IP Address: '''10.1.110.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.110.255'''
### Confirm the admin-VLAN Section with '''yes'''
## data-VLAN Section
### VLAN ID: '''120'''
### Domain Name: '''data.example.com'''
### IP Address: '''10.1.120.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.120.255'''
### Confirm the data-VLAN Section with '''yes'''
## int-VLAN Section
### VLAN ID: '''130'''
### Domain Name: '''int.example.com'''
### IP Address: '''10.1.130.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.130.255'''
### Confirm the int-VLAN Section with '''yes'''
### Enter the default Gateway: '''192.168.140.1'''
### Enter the primary DNS-Server: '''192.168.140.1'''
### Omit configuring a second DNS-Server with '''no'''
## Confirm the listed configuration with '''yes'''
## Enter your very secret root password
## Confirm to reboot with '''yes'''
# Make sure, that you boot from the first harddisk and not from the installation medium again.
# Continue with [[Multi-Node Installation#Specialized_Installation|specializing your Node]]

==== Second Storage-Node (tier1-storage-node-02) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

==== Primary-Master-Node (vm-node-01) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

==== Secondary-Master-Node (vm-node-02) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

== Skipping Checks ==
To skip checks, type '''no''' when asked:
Do you want to start the installation?
yes or no?: '''no'''

Then manually restart the stoney cloud installer with the desired options. For example:
/mnt/cdrom/foss-cloud-installer -c

Options:
-c: Skip CPU requirement checks
-m: Skip memory requirement checks
-s: Skip CPU and memory requirement checks

== Specialized Installation ==
==== First Storage-Node (tier1-storage-node-01) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].

Log into the first [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-storage-node

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

==== Second Storage-Node (tier1-storage-node-02) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].

Log into the second [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-storage-node

For more information about the script and what it does, please visit the [[node-configuration]] script page.

==== Primary-Master-Node (vm-node-01) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the VM-Node]].

Log into the [[Primary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

The stoney cloud uses virtual ip addresses (VIPs) for fail over purposes. Therefore you need to configure [http://www.pureftpd.org/project/ucarp ucarp].

Confirm that you want to run the script.
Do you really want to proceed with configuration of the primary-master-node?
yes or no (default: no): '''yes'''

Enter the VIP (virtual IP) for the public interface. The apache will listen on this VIP because it is listening on the public interface (if you followed this documentation the VIP for the public interface is 192.168.140.10).
Please enter the VIP for the pub-interface (VLAN 140)
(default: 192.168.140.10): '''192.168.140.10'''

Enter the VIP (virtual IP) for the internal interace. The LDAP will listen on this VIP because it is listening on the internal interface (if you followed this documentation the VIP for the internal interface is 10.1.130.10).
Please enter the VIP for the int-interface (VLAN 130)
(default: 10.1.130.10): '''10.1.130.10'''

Enter the RIP (real IP) for the secondary-master-node on the internal interface. The RIP is needed to keep the LDAP directories synchronized (if you followed this documentation the RIP for the secondary-master-node on the internal interface is 10.1.130.14)
Please enter the IP for the int-interface (VLAN 130) of the
secondary-master-node (default: 10.1.130.14): '''10.1.130.14'''

The script now tests the network configuration and

In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node. Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the primary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':

IP address (default: 10.1.110.11): '''10.1.110.11'''
Username (default: root): '''root'''
Password for root: '''**********'''

In order to mount the gluster-filesystem, you need to connect via ssh to the secondary-storage-node. Enter the IP for the secondary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.12). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the secondary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':

IP address (default: 10.1.110.12): '''10.1.110.12'''
Username (default: root): '''root'''
Password for root: '''**********'''

* Configure the LDAP directory:
** Define the password for the LDAP-Superuser (cn=Manager,dc=stoney-cloud,dc=org)
** Currently the user for the prov-backup-kvm daemon is the LDAP-Superuser so enter the same password again
** Define the password for the LDAP-dhcp user (cn=dhcp,ou=services,ou=administration,dc=stoney-cloud,dc=org)
** Enter all necessary information for the stoney cloud administrator (User1)
*** Given name
*** Surname
*** Gender
*** E-mail
*** Language
*** Password

* Finally enter the domain name which will correspond to the public VIP (default is stoney-cloud.example.org)

* Due to [https://github.com/stepping-stone/node-integration/issues/9 bug #9], you need to manually finish the configuration of the libvirthook scripts:
** You mainly have to fill in the following variables:
*** '''libvirtHookFirewallSvnUser'''
*** '''libvirtHookFirewallSvnPassword'''
** See also [https://int.stepping-stone.ch/wiki/libvirt_Hooks#Config_for_test-environment this test configuration]
* Due to [https://github.com/stepping-stone/node-integration/issues/12 bug #12], you need to manually configure the LDAPKVMWrapper.pl script:
** Fill in the <code>/etc/Provisioning/Backup/LDAPKVMWrapper.conf</code> file
** Create a cronjob entry which runs the script <code>/usr/bin/LDAPKVMWrapper.pl</code> once a day:
*** <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

==== Secondary-Master-Node (vm-node-02) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the Secondary-Master-Node]].

Log into the [[Secondary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-master-node

* In order to get the configuration from the primary-master-node, we need to access it via ssh
** Enter the IP for the primary-master-node on the admin interface (if you followed this documentation it is 10.1.130.13)
** Enter the username (if you followd the default setup it is root)
** Enter the users password.

* In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node, so enter the following information:
** Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11)
** Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root)
** Enter the users password
* Repeat the same procedure for the secondary-storage-node (if you followed this documentation the IP is 10.1.110.12)

* Enter the LDAP-Superuser password you defined during the [[Multi-Node Installation#Primary-Master-Node (stoney-cloud-node-01)_2 | primary-master-node]] installation

* Due tu [https://github.com/stepping-stone/node-integration/issues/9 bug #9], you need to manually finish the configuration of the libvirthook scripts:
** You mainly have to fill in the following variables:
*** '''libvirtHookFirewallSvnUser'''
*** '''libvirtHookFirewallSvnPassword'''
** See also [https://int.stepping-stone.ch/wiki/libvirt_Hooks#Config_for_test-environment this test configuration]
* Due to [https://github.com/stepping-stone/node-integration/issues/12 bug #12], you need to manually configure the LDAPKVMWrapper.pl script:
** Fill in the <code>/etc/Provisioning/Backup/LDAPKVMWrapper.conf</code> file
** Create a cronjob entry which runs the script <code>/usr/bin/LDAPKVMWrapper.pl</code> once a day:
*** <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

== Links ==
= Node Integration =
The following figure gives an overview what the node-integration script does for the different node types

[[File:node-integration.png|500px|thumbnail|none|What does the node-integration script for the different node types]]

You can modify/update these steps by editing [[File:node-integration.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

= Old Documentation =
== Specialized Installation ==
=== Primary-Master-Node (vm-node-01) ===
If you configured a additional Backup Volume on the Storage Nodes, you want to mount them now in the VM-Node.

Log into the Primary-Master-Node and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

==== Manual Steps ====
In order to be able to migrate a VM from a carrier, a special user called transfer will be created.
<source lang="bash">
lvcreate -L 60G -n transfer local0

mkfs.xfs -L "OSBD_transfe" /dev/local0/transfer

cat << EOF >> /etc/fstab

LABEL=OSBD_transfe /home/transfer xfs noatime,nodev,nosuid,noexec 0 2
EOF

mount /home/transfer

useradd --comment "User which is used for VM disk file transfer between carriers" \
--create-home \
--system \
--user-group \
transfer

passwd transfer
</source>

Allow password authentication for the transfer user:
<source lang="bash">
$EDITOR /etc/ssh/sshd_config
</source>

<pre>
[...]

Match User transfer
PasswordAuthentication yes

[...]
</pre>
To apply the changes above, restart the SSH daemon:
<source lang="bash">
/etc/init.d/sshd restart
</source>

[[Category:stoney cloud]][[Category:Installation]]

File:node-integration.png

2014-06-27T12:53:09Z

Pat: Initial verison

Initial verison

stoney cloud: Multi-Node Installation

2014-06-27T12:52:49Z

Pat: /* Node Integration */

== Overview ==
The [http://www.stoney-cloud.org/ stoney cloud] builds upon various standard open source components and can be run on commodity hardware. The final Multi-Node Setup consists of the following components:
* One [[Primary-Master-Node]] with an OpenLDAP Directory Server for the storage of the stoney cloud user and service related data with the web based management [[VM-Manager]] interface and the Linux kernel based virtualization technology.
* One [[Secondary-Master-Node]] with the Linux kernel based virtualization technology.
* Two [[Storage-Node | Storage-Nodes]] configured as a replicated and distributed data storage service based on [http://www.gluster.org/ GlusterFS].
The components communicate with each other over a standard Ethernet based IPv4 network.

=== Prerequisites ===
The following items and conditions are required to be able to install and configure a stoney cloud environment:
* Dedicated Hardware (4 Servers) which fulfil the following requirements:
** 64-Bit Intel with VT-Technologie (AMD is not tested at the moment).
** 8 Gigabyte Memory (more is better).
** 147 Gigabyte up to 2 Terabyte disks.
*** For all four Nodes ([[VM-Node | VM-Nodes]] and [[Storage-Node | Storage-Nodes]]) two 147 Gigabyte Disks, configured as RAID1, are enough for the [http://www.gentoo.org/ Gentoo Linux] operating system.
*** Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.
** Two physical Ethernet Interfaces, which support the same bandwidth (for example 1 Gigabit/s).
* Two Gigabit layer-2 switches supporting [http://en.wikipedia.org/wiki/IEEE_802.3ad IEEE 802.3ad] (dynamic link aggregation), [http://en.wikipedia.org/wiki/IEEE_802.1Q IEEE 802.1Q] (VLAN tagging) and stacking (optional, but recommended).
* Experience with Linux environments especially with [http://www.gentoo.org Gentoo Linux].
* Good experience with IP networking, because the Switches need to be configured manually (dynamic link aggregation and VLAN tagging).

=== Limitations ===
During the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system, the first two physical Ethernet Interfaces are automatically configured as a logical interface (bond0) and the four tagged VLANs are set up. If more than two physical Ethernet Interfaces are to be included in to the logical interface (bond0) or the physical Ethernet Interfaces have different bandwidths (for example 1 Gigabit/s and 10 Gigabit/s), the final logical interface (bond0) needs to be configured manually after the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system.
Only the first two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] are set up automatically. More [[Storage-Node | Storage-Nodes]] need to be integrated manually.

=== Network Overview ===
As stated before, a minimal multi node stoney cloud environment consist of two VM- and Storage-Nodes.
It is highly recommended to use IEEE 802.3ad link aggregation (bonding, trunking etc.) over two network cards and attach them as one logical link to the access switch.

It is out of scope of this document on how to configure the switches for stacking, link aggregation and VLAN tagging, consult the respective user manual.

There are two scenarios on how to connect the nodes to the network.

==== Network Overview: Physical Layer Scenario 1 (recommended) ====
The preferred solution is to use two switches which are stackable (supporting link aggregation over two switches). Connect the nodes and switches as illustrated below. Thus eliminating the single point of failure as presented in scenario 2.

<pre>
+----------------------+ +----------------------+
| | | |
| vm-node-01 | | vm-node-02 |
| | | |
+----------------------+ +----------------------+
bond0 | \ / | bond0
| \ / |
| \ / |
| \/ | ___
| /\ | ___( )___
| / \ | __( )__
| / \ | _( )_
+-------------+ +-------------+ _( )_
| switch-01 |===| switch-02 |-------------------(_ Corporate LAN/WAN _)
+-------------+ +-------------+ (_ _)
| \ / | (__ __)
| \ / | (___ ___)
| \ / | (___)
| \/ |
| /\ |
| / \ |
bond0 | / \ | bond0
+-----------------------+ +-----------------------+
| | | |
| tier1-storage-node-01 | | tier1-storage-node-02 |
| | | |
+-----------------------+ +-----------------------+
</pre>

==== Network Overview: Physical Layer Scenario 2 (use at your own risk) ====
If theres only one switch available (or the switches aren't stackable) connect the nodes as illustrated below. As you can see, the switch is as a single point of failure.
<pre>
+----------------------+ +----------------------+
| | | |
| vm-node-01 | | vm-node-02 |
| | | |
+----------------------+ +----------------------+ ___
bond0 \\ // bond0 ___( )___
\\ // __( )__
\\ // _( )_
+-------------+ _( )_
| switch-01 |-----------------------------(_ Corporate LAN/WAN _)
+-------------+ (_ _)
// \\ (__ __)
// \\ (___ ___)
bond0 // \\ bond0 (___)
+-----------------------+ +-----------------------+
| | | |
| tier1-storage-node-01 | | tier1-storage-node-02 |
| | | |
+-----------------------+ +-----------------------+
</pre>

==== Network overview: Logical layer ====
The goal is to achieve the following configuration:
<pre>
+----------------+----------------+----------------+----------------+
| 10.1.110.1X | 10.1.120.1X | 10.1.130.1X | 192.168.140.1X |
+----------------+----------------+----------------+----------------+
| | | | vmbr0 |
| vlan110 | vlan120 | vlan130 +----------------+
| | | | vlan140 |
+----------------+----------------+----------------+----------------+
+-------------------------------------------------------------------+
| bond0 (bonding.mode=802.3ad) |
+-------------------------------------------------------------------+
+----------------+ +----------------+
| eth0 | | eth1 |
+----------------+ +----------------+
</pre>

The ideal stoney cloud environment is based on four logical separated VLANs (virtual LANs):
* '''admin''': Administrative network, used for administration and monitoring purposes.
* '''data''': Data network, used for GlusterFS traffic.
* '''int''': Internal network, used for internal traffic such as LDAP, libvirt and more.
* '''pub''': Public network, used for accessing the VM-Manager webinterface, Spice traffic and internet access.

Each of the above VLANs hold dedicated services and separates them from each other. This documentation assumes, that the four VLANs are present and the following IP networks are available:

{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|VLAN name
!align="left"|VLAN ID
!align="left"|Network prefix
!align="left"|Default Gateway address
!align="left"|Broadcast address
!align="left"|Domain name
!align="left"|VIP
|-

|admin
|110
|10.1.110.0/24
| --
|10.1.110.255
|admin.example.com
|
|-

|data
|120
|10.1.120.0/24
| --
|10.1.120.255
|data.example.com
|
|-

|int
|130
|10.1.130.0/24
| --
|10.1.130.255
|int.example.com
|10.1.130.10
|-

|pub
|140
|192.168.140.0/24
|192.168.140.1
|192.168.140.255
|example.com
|192.168.140.10
|-

|}

The IP allocation of the nodes will be assumed as stated in the table below:
{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|Node name
!align="left"|Admin address (VLAN 110)
!align="left"|Data address (VLAN 120)
!align="left"|Int address (VLAN 130)
!align="left"|Pub address (VLAN 140)
|-

|-
|tier1-storage-node-01
|10.1.110.11
|10.1.120.11
|10.1.130.11
|192.168.140.11
|-

|-
|tier1-storage-node-02
|10.1.110.12
|10.1.120.12
|10.1.130.12
|192.168.140.12
|-

|-
|vm-node-01
|10.1.110.13
|10.1.120.13
|10.1.130.13
|192.168.140.13
|-

|-
|vm-node-02
|10.1.110.14
|10.1.120.14
|10.1.130.14
|192.168.140.14
|-

|}

We'll also presume, we have the following Domain Name Servers:
* Domain Name Server 1: '''192.168.140.1'''

== Base Installation ==
All nodes are based on the [http://www.gentoo.org/ Gentoo Linux] operating system.

=== BIOS Set Up Checklist ===
* Enable the "reboot-after-power-loss" option (if your BIOS supports it).
* Make sure, that you have the newest BIOS (BMC and perhaps SCSI firmware) version.
* Make sure, you've disabled halt on post errors (or similar) or enable keyboard-less operation (if your BIOS supports it).

=== RAID Set Up ===
Create a RAID1 volume. This RAID-Set is used for the Operating System. Please be aware, the the current stoney cloud only supports 147 Gigabyte up to 2 Terabyte disks for this first RAID-Set.

Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.

=== Node Installation ===
The first step of Semi-Automatic Multi-Node Set Up is the same for all four Nodes. In this documentation we presume, that you stick to the naming convention mentioned above. After the Base Installation of the Nodes, the following daemons will be running:
* '''crond''': Crond, to execute scheduled commands
* '''ntpd''': Network Time Protocol daemon, keeps the time synced with the Time from Servers in the LAN or WAN.
* '''sshd''': OpenSSH SSH daemon, used for remote access and remote administration.
* '''syslogd''': System Logging, keeps track of messages from the System and the Applications.
* '''udevd''': Linux dynamic device management, that manage events, symlinks and permissions of devices.

==== First Storage-Node (tier1-storage-node-01) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions as follows (the bold values are examples can be set through the administrator and are variable, according to the local setup):
## Global Section
### Confirm that you want to start? '''yes'''
### Choose a Node-Type: '''Storage-Node'''
### Choose a Block-Device: '''sda'''
### Confirm to erase all (from a previous installation): '''yes'''
### Confirm to continue with the given the Partition-Scheme: '''yes'''
### Choose the network interfaces to bond together.
#### Device #0: '''eth0'''
#### Device #1: '''eth1'''
### Node-Name: '''tier1-storage-node-01'''
## pub-VLAN Section
### VLAN ID: '''140'''
### Domain Name: '''example.com'''
### IP Address: '''192.168.140.11'''
### Netmask: '''24'''
### Broadcast: '''192.168.140.255'''
### Confirm the pub-VLAN Section with '''yes'''
## admin-VLAN Section
### VLAN ID: '''110'''
### Domain Name: '''admin.example.com'''
### IP Address: '''10.1.110.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.110.255'''
### Confirm the admin-VLAN Section with '''yes'''
## data-VLAN Section
### VLAN ID: '''120'''
### Domain Name: '''data.example.com'''
### IP Address: '''10.1.120.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.120.255'''
### Confirm the data-VLAN Section with '''yes'''
## int-VLAN Section
### VLAN ID: '''130'''
### Domain Name: '''int.example.com'''
### IP Address: '''10.1.130.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.130.255'''
### Confirm the int-VLAN Section with '''yes'''
### Enter the default Gateway: '''192.168.140.1'''
### Enter the primary DNS-Server: '''192.168.140.1'''
### Omit configuring a second DNS-Server with '''no'''
## Confirm the listed configuration with '''yes'''
## Enter your very secret root password
## Confirm to reboot with '''yes'''
# Make sure, that you boot from the first harddisk and not from the installation medium again.
# Continue with [[Multi-Node Installation#Specialized_Installation|specializing your Node]]

==== Second Storage-Node (tier1-storage-node-02) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

==== Primary-Master-Node (vm-node-01) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

==== Secondary-Master-Node (vm-node-02) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

== Skipping Checks ==
To skip checks, type '''no''' when asked:
Do you want to start the installation?
yes or no?: '''no'''

Then manually restart the stoney cloud installer with the desired options. For example:
/mnt/cdrom/foss-cloud-installer -c

Options:
-c: Skip CPU requirement checks
-m: Skip memory requirement checks
-s: Skip CPU and memory requirement checks

== Specialized Installation ==
==== First Storage-Node (tier1-storage-node-01) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].

Log into the first [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-storage-node

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

==== Second Storage-Node (tier1-storage-node-02) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].

Log into the second [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-storage-node

For more information about the script and what it does, please visit the [[node-configuration]] script page.

==== Primary-Master-Node (vm-node-01) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the VM-Node]].

Log into the [[Primary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

The stoney cloud uses virtual ip addresses (VIPs) for fail over purposes. Therefore you need to configure [http://www.pureftpd.org/project/ucarp ucarp].

Confirm that you want to run the script.
Do you really want to proceed with configuration of the primary-master-node?
yes or no (default: no): '''yes'''

Enter the VIP (virtual IP) for the public interface. The apache will listen on this VIP because it is listening on the public interface (if you followed this documentation the VIP for the public interface is 192.168.140.10).
Please enter the VIP for the pub-interface (VLAN 140)
(default: 192.168.140.10): '''192.168.140.10'''

Enter the VIP (virtual IP) for the internal interace. The LDAP will listen on this VIP because it is listening on the internal interface (if you followed this documentation the VIP for the internal interface is 10.1.130.10).
Please enter the VIP for the int-interface (VLAN 130)
(default: 10.1.130.10): '''10.1.130.10'''

Enter the RIP (real IP) for the secondary-master-node on the internal interface. The RIP is needed to keep the LDAP directories synchronized (if you followed this documentation the RIP for the secondary-master-node on the internal interface is 10.1.130.14)
Please enter the IP for the int-interface (VLAN 130) of the
secondary-master-node (default: 10.1.130.14): '''10.1.130.14'''

The script now tests the network configuration and

In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node. Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the primary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':

IP address (default: 10.1.110.11): '''10.1.110.11'''
Username (default: root): '''root'''
Password for root: '''**********'''

In order to mount the gluster-filesystem, you need to connect via ssh to the secondary-storage-node. Enter the IP for the secondary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.12). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the secondary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':

IP address (default: 10.1.110.12): '''10.1.110.12'''
Username (default: root): '''root'''
Password for root: '''**********'''

* Configure the LDAP directory:
** Define the password for the LDAP-Superuser (cn=Manager,dc=stoney-cloud,dc=org)
** Currently the user for the prov-backup-kvm daemon is the LDAP-Superuser so enter the same password again
** Define the password for the LDAP-dhcp user (cn=dhcp,ou=services,ou=administration,dc=stoney-cloud,dc=org)
** Enter all necessary information for the stoney cloud administrator (User1)
*** Given name
*** Surname
*** Gender
*** E-mail
*** Language
*** Password

* Finally enter the domain name which will correspond to the public VIP (default is stoney-cloud.example.org)

* Due to [https://github.com/stepping-stone/node-integration/issues/9 bug #9], you need to manually finish the configuration of the libvirthook scripts:
** You mainly have to fill in the following variables:
*** '''libvirtHookFirewallSvnUser'''
*** '''libvirtHookFirewallSvnPassword'''
** See also [https://int.stepping-stone.ch/wiki/libvirt_Hooks#Config_for_test-environment this test configuration]
* Due to [https://github.com/stepping-stone/node-integration/issues/12 bug #12], you need to manually configure the LDAPKVMWrapper.pl script:
** Fill in the <code>/etc/Provisioning/Backup/LDAPKVMWrapper.conf</code> file
** Create a cronjob entry which runs the script <code>/usr/bin/LDAPKVMWrapper.pl</code> once a day:
*** <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

==== Secondary-Master-Node (vm-node-02) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the Secondary-Master-Node]].

Log into the [[Secondary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-master-node

* In order to get the configuration from the primary-master-node, we need to access it via ssh
** Enter the IP for the primary-master-node on the admin interface (if you followed this documentation it is 10.1.130.13)
** Enter the username (if you followd the default setup it is root)
** Enter the users password.

* In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node, so enter the following information:
** Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11)
** Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root)
** Enter the users password
* Repeat the same procedure for the secondary-storage-node (if you followed this documentation the IP is 10.1.110.12)

* Enter the LDAP-Superuser password you defined during the [[Multi-Node Installation#Primary-Master-Node (stoney-cloud-node-01)_2 | primary-master-node]] installation

* Due tu [https://github.com/stepping-stone/node-integration/issues/9 bug #9], you need to manually finish the configuration of the libvirthook scripts:
** You mainly have to fill in the following variables:
*** '''libvirtHookFirewallSvnUser'''
*** '''libvirtHookFirewallSvnPassword'''
** See also [https://int.stepping-stone.ch/wiki/libvirt_Hooks#Config_for_test-environment this test configuration]
* Due to [https://github.com/stepping-stone/node-integration/issues/12 bug #12], you need to manually configure the LDAPKVMWrapper.pl script:
** Fill in the <code>/etc/Provisioning/Backup/LDAPKVMWrapper.conf</code> file
** Create a cronjob entry which runs the script <code>/usr/bin/LDAPKVMWrapper.pl</code> once a day:
*** <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

== Links ==
= Node Integration =
The following figure gives an overview what the node-integration script does for the different node types

[[File:node-integration.png|500px|thumbnail|none|What does the node-integration script for the different node types]]

= Old Documentation =
== Specialized Installation ==
=== Primary-Master-Node (vm-node-01) ===
If you configured a additional Backup Volume on the Storage Nodes, you want to mount them now in the VM-Node.

Log into the Primary-Master-Node and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

==== Manual Steps ====
In order to be able to migrate a VM from a carrier, a special user called transfer will be created.
<source lang="bash">
lvcreate -L 60G -n transfer local0

mkfs.xfs -L "OSBD_transfe" /dev/local0/transfer

cat << EOF >> /etc/fstab

LABEL=OSBD_transfe /home/transfer xfs noatime,nodev,nosuid,noexec 0 2
EOF

mount /home/transfer

useradd --comment "User which is used for VM disk file transfer between carriers" \
--create-home \
--system \
--user-group \
transfer

passwd transfer
</source>

Allow password authentication for the transfer user:
<source lang="bash">
$EDITOR /etc/ssh/sshd_config
</source>

<pre>
[...]

Match User transfer
PasswordAuthentication yes

[...]
</pre>
To apply the changes above, restart the SSH daemon:
<source lang="bash">
/etc/init.d/sshd restart
</source>

[[Category:stoney cloud]][[Category:Installation]]

stoney cloud: Multi-Node Installation

2014-06-27T12:51:33Z

Pat: /* Node Integration */

== Overview ==
The [http://www.stoney-cloud.org/ stoney cloud] builds upon various standard open source components and can be run on commodity hardware. The final Multi-Node Setup consists of the following components:
* One [[Primary-Master-Node]] with an OpenLDAP Directory Server for the storage of the stoney cloud user and service related data with the web based management [[VM-Manager]] interface and the Linux kernel based virtualization technology.
* One [[Secondary-Master-Node]] with the Linux kernel based virtualization technology.
* Two [[Storage-Node | Storage-Nodes]] configured as a replicated and distributed data storage service based on [http://www.gluster.org/ GlusterFS].
The components communicate with each other over a standard Ethernet based IPv4 network.

=== Prerequisites ===
The following items and conditions are required to be able to install and configure a stoney cloud environment:
* Dedicated Hardware (4 Servers) which fulfil the following requirements:
** 64-Bit Intel with VT-Technologie (AMD is not tested at the moment).
** 8 Gigabyte Memory (more is better).
** 147 Gigabyte up to 2 Terabyte disks.
*** For all four Nodes ([[VM-Node | VM-Nodes]] and [[Storage-Node | Storage-Nodes]]) two 147 Gigabyte Disks, configured as RAID1, are enough for the [http://www.gentoo.org/ Gentoo Linux] operating system.
*** Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.
** Two physical Ethernet Interfaces, which support the same bandwidth (for example 1 Gigabit/s).
* Two Gigabit layer-2 switches supporting [http://en.wikipedia.org/wiki/IEEE_802.3ad IEEE 802.3ad] (dynamic link aggregation), [http://en.wikipedia.org/wiki/IEEE_802.1Q IEEE 802.1Q] (VLAN tagging) and stacking (optional, but recommended).
* Experience with Linux environments especially with [http://www.gentoo.org Gentoo Linux].
* Good experience with IP networking, because the Switches need to be configured manually (dynamic link aggregation and VLAN tagging).

=== Limitations ===
During the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system, the first two physical Ethernet Interfaces are automatically configured as a logical interface (bond0) and the four tagged VLANs are set up. If more than two physical Ethernet Interfaces are to be included in to the logical interface (bond0) or the physical Ethernet Interfaces have different bandwidths (for example 1 Gigabit/s and 10 Gigabit/s), the final logical interface (bond0) needs to be configured manually after the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system.
Only the first two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] are set up automatically. More [[Storage-Node | Storage-Nodes]] need to be integrated manually.

=== Network Overview ===
As stated before, a minimal multi node stoney cloud environment consist of two VM- and Storage-Nodes.
It is highly recommended to use IEEE 802.3ad link aggregation (bonding, trunking etc.) over two network cards and attach them as one logical link to the access switch.

It is out of scope of this document on how to configure the switches for stacking, link aggregation and VLAN tagging, consult the respective user manual.

There are two scenarios on how to connect the nodes to the network.

==== Network Overview: Physical Layer Scenario 1 (recommended) ====
The preferred solution is to use two switches which are stackable (supporting link aggregation over two switches). Connect the nodes and switches as illustrated below. Thus eliminating the single point of failure as presented in scenario 2.

<pre>
+----------------------+ +----------------------+
| | | |
| vm-node-01 | | vm-node-02 |
| | | |
+----------------------+ +----------------------+
bond0 | \ / | bond0
| \ / |
| \ / |
| \/ | ___
| /\ | ___( )___
| / \ | __( )__
| / \ | _( )_
+-------------+ +-------------+ _( )_
| switch-01 |===| switch-02 |-------------------(_ Corporate LAN/WAN _)
+-------------+ +-------------+ (_ _)
| \ / | (__ __)
| \ / | (___ ___)
| \ / | (___)
| \/ |
| /\ |
| / \ |
bond0 | / \ | bond0
+-----------------------+ +-----------------------+
| | | |
| tier1-storage-node-01 | | tier1-storage-node-02 |
| | | |
+-----------------------+ +-----------------------+
</pre>

==== Network Overview: Physical Layer Scenario 2 (use at your own risk) ====
If theres only one switch available (or the switches aren't stackable) connect the nodes as illustrated below. As you can see, the switch is as a single point of failure.
<pre>
+----------------------+ +----------------------+
| | | |
| vm-node-01 | | vm-node-02 |
| | | |
+----------------------+ +----------------------+ ___
bond0 \\ // bond0 ___( )___
\\ // __( )__
\\ // _( )_
+-------------+ _( )_
| switch-01 |-----------------------------(_ Corporate LAN/WAN _)
+-------------+ (_ _)
// \\ (__ __)
// \\ (___ ___)
bond0 // \\ bond0 (___)
+-----------------------+ +-----------------------+
| | | |
| tier1-storage-node-01 | | tier1-storage-node-02 |
| | | |
+-----------------------+ +-----------------------+
</pre>

==== Network overview: Logical layer ====
The goal is to achieve the following configuration:
<pre>
+----------------+----------------+----------------+----------------+
| 10.1.110.1X | 10.1.120.1X | 10.1.130.1X | 192.168.140.1X |
+----------------+----------------+----------------+----------------+
| | | | vmbr0 |
| vlan110 | vlan120 | vlan130 +----------------+
| | | | vlan140 |
+----------------+----------------+----------------+----------------+
+-------------------------------------------------------------------+
| bond0 (bonding.mode=802.3ad) |
+-------------------------------------------------------------------+
+----------------+ +----------------+
| eth0 | | eth1 |
+----------------+ +----------------+
</pre>

The ideal stoney cloud environment is based on four logical separated VLANs (virtual LANs):
* '''admin''': Administrative network, used for administration and monitoring purposes.
* '''data''': Data network, used for GlusterFS traffic.
* '''int''': Internal network, used for internal traffic such as LDAP, libvirt and more.
* '''pub''': Public network, used for accessing the VM-Manager webinterface, Spice traffic and internet access.

Each of the above VLANs hold dedicated services and separates them from each other. This documentation assumes, that the four VLANs are present and the following IP networks are available:

{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|VLAN name
!align="left"|VLAN ID
!align="left"|Network prefix
!align="left"|Default Gateway address
!align="left"|Broadcast address
!align="left"|Domain name
!align="left"|VIP
|-

|admin
|110
|10.1.110.0/24
| --
|10.1.110.255
|admin.example.com
|
|-

|data
|120
|10.1.120.0/24
| --
|10.1.120.255
|data.example.com
|
|-

|int
|130
|10.1.130.0/24
| --
|10.1.130.255
|int.example.com
|10.1.130.10
|-

|pub
|140
|192.168.140.0/24
|192.168.140.1
|192.168.140.255
|example.com
|192.168.140.10
|-

|}

The IP allocation of the nodes will be assumed as stated in the table below:
{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|Node name
!align="left"|Admin address (VLAN 110)
!align="left"|Data address (VLAN 120)
!align="left"|Int address (VLAN 130)
!align="left"|Pub address (VLAN 140)
|-

|-
|tier1-storage-node-01
|10.1.110.11
|10.1.120.11
|10.1.130.11
|192.168.140.11
|-

|-
|tier1-storage-node-02
|10.1.110.12
|10.1.120.12
|10.1.130.12
|192.168.140.12
|-

|-
|vm-node-01
|10.1.110.13
|10.1.120.13
|10.1.130.13
|192.168.140.13
|-

|-
|vm-node-02
|10.1.110.14
|10.1.120.14
|10.1.130.14
|192.168.140.14
|-

|}

We'll also presume, we have the following Domain Name Servers:
* Domain Name Server 1: '''192.168.140.1'''

== Base Installation ==
All nodes are based on the [http://www.gentoo.org/ Gentoo Linux] operating system.

=== BIOS Set Up Checklist ===
* Enable the "reboot-after-power-loss" option (if your BIOS supports it).
* Make sure, that you have the newest BIOS (BMC and perhaps SCSI firmware) version.
* Make sure, you've disabled halt on post errors (or similar) or enable keyboard-less operation (if your BIOS supports it).

=== RAID Set Up ===
Create a RAID1 volume. This RAID-Set is used for the Operating System. Please be aware, the the current stoney cloud only supports 147 Gigabyte up to 2 Terabyte disks for this first RAID-Set.

Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.

=== Node Installation ===
The first step of Semi-Automatic Multi-Node Set Up is the same for all four Nodes. In this documentation we presume, that you stick to the naming convention mentioned above. After the Base Installation of the Nodes, the following daemons will be running:
* '''crond''': Crond, to execute scheduled commands
* '''ntpd''': Network Time Protocol daemon, keeps the time synced with the Time from Servers in the LAN or WAN.
* '''sshd''': OpenSSH SSH daemon, used for remote access and remote administration.
* '''syslogd''': System Logging, keeps track of messages from the System and the Applications.
* '''udevd''': Linux dynamic device management, that manage events, symlinks and permissions of devices.

==== First Storage-Node (tier1-storage-node-01) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions as follows (the bold values are examples can be set through the administrator and are variable, according to the local setup):
## Global Section
### Confirm that you want to start? '''yes'''
### Choose a Node-Type: '''Storage-Node'''
### Choose a Block-Device: '''sda'''
### Confirm to erase all (from a previous installation): '''yes'''
### Confirm to continue with the given the Partition-Scheme: '''yes'''
### Choose the network interfaces to bond together.
#### Device #0: '''eth0'''
#### Device #1: '''eth1'''
### Node-Name: '''tier1-storage-node-01'''
## pub-VLAN Section
### VLAN ID: '''140'''
### Domain Name: '''example.com'''
### IP Address: '''192.168.140.11'''
### Netmask: '''24'''
### Broadcast: '''192.168.140.255'''
### Confirm the pub-VLAN Section with '''yes'''
## admin-VLAN Section
### VLAN ID: '''110'''
### Domain Name: '''admin.example.com'''
### IP Address: '''10.1.110.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.110.255'''
### Confirm the admin-VLAN Section with '''yes'''
## data-VLAN Section
### VLAN ID: '''120'''
### Domain Name: '''data.example.com'''
### IP Address: '''10.1.120.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.120.255'''
### Confirm the data-VLAN Section with '''yes'''
## int-VLAN Section
### VLAN ID: '''130'''
### Domain Name: '''int.example.com'''
### IP Address: '''10.1.130.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.130.255'''
### Confirm the int-VLAN Section with '''yes'''
### Enter the default Gateway: '''192.168.140.1'''
### Enter the primary DNS-Server: '''192.168.140.1'''
### Omit configuring a second DNS-Server with '''no'''
## Confirm the listed configuration with '''yes'''
## Enter your very secret root password
## Confirm to reboot with '''yes'''
# Make sure, that you boot from the first harddisk and not from the installation medium again.
# Continue with [[Multi-Node Installation#Specialized_Installation|specializing your Node]]

==== Second Storage-Node (tier1-storage-node-02) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

==== Primary-Master-Node (vm-node-01) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

==== Secondary-Master-Node (vm-node-02) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

== Skipping Checks ==
To skip checks, type '''no''' when asked:
Do you want to start the installation?
yes or no?: '''no'''

Then manually restart the stoney cloud installer with the desired options. For example:
/mnt/cdrom/foss-cloud-installer -c

Options:
-c: Skip CPU requirement checks
-m: Skip memory requirement checks
-s: Skip CPU and memory requirement checks

== Specialized Installation ==
==== First Storage-Node (tier1-storage-node-01) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].

Log into the first [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-storage-node

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

==== Second Storage-Node (tier1-storage-node-02) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].

Log into the second [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-storage-node

For more information about the script and what it does, please visit the [[node-configuration]] script page.

==== Primary-Master-Node (vm-node-01) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the VM-Node]].

Log into the [[Primary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

The stoney cloud uses virtual ip addresses (VIPs) for fail over purposes. Therefore you need to configure [http://www.pureftpd.org/project/ucarp ucarp].

Confirm that you want to run the script.
Do you really want to proceed with configuration of the primary-master-node?
yes or no (default: no): '''yes'''

Enter the VIP (virtual IP) for the public interface. The apache will listen on this VIP because it is listening on the public interface (if you followed this documentation the VIP for the public interface is 192.168.140.10).
Please enter the VIP for the pub-interface (VLAN 140)
(default: 192.168.140.10): '''192.168.140.10'''

Enter the VIP (virtual IP) for the internal interace. The LDAP will listen on this VIP because it is listening on the internal interface (if you followed this documentation the VIP for the internal interface is 10.1.130.10).
Please enter the VIP for the int-interface (VLAN 130)
(default: 10.1.130.10): '''10.1.130.10'''

Enter the RIP (real IP) for the secondary-master-node on the internal interface. The RIP is needed to keep the LDAP directories synchronized (if you followed this documentation the RIP for the secondary-master-node on the internal interface is 10.1.130.14)
Please enter the IP for the int-interface (VLAN 130) of the
secondary-master-node (default: 10.1.130.14): '''10.1.130.14'''

The script now tests the network configuration and

In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node. Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the primary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':

IP address (default: 10.1.110.11): '''10.1.110.11'''
Username (default: root): '''root'''
Password for root: '''**********'''

In order to mount the gluster-filesystem, you need to connect via ssh to the secondary-storage-node. Enter the IP for the secondary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.12). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the secondary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':

IP address (default: 10.1.110.12): '''10.1.110.12'''
Username (default: root): '''root'''
Password for root: '''**********'''

* Configure the LDAP directory:
** Define the password for the LDAP-Superuser (cn=Manager,dc=stoney-cloud,dc=org)
** Currently the user for the prov-backup-kvm daemon is the LDAP-Superuser so enter the same password again
** Define the password for the LDAP-dhcp user (cn=dhcp,ou=services,ou=administration,dc=stoney-cloud,dc=org)
** Enter all necessary information for the stoney cloud administrator (User1)
*** Given name
*** Surname
*** Gender
*** E-mail
*** Language
*** Password

* Finally enter the domain name which will correspond to the public VIP (default is stoney-cloud.example.org)

* Due to [https://github.com/stepping-stone/node-integration/issues/9 bug #9], you need to manually finish the configuration of the libvirthook scripts:
** You mainly have to fill in the following variables:
*** '''libvirtHookFirewallSvnUser'''
*** '''libvirtHookFirewallSvnPassword'''
** See also [https://int.stepping-stone.ch/wiki/libvirt_Hooks#Config_for_test-environment this test configuration]
* Due to [https://github.com/stepping-stone/node-integration/issues/12 bug #12], you need to manually configure the LDAPKVMWrapper.pl script:
** Fill in the <code>/etc/Provisioning/Backup/LDAPKVMWrapper.conf</code> file
** Create a cronjob entry which runs the script <code>/usr/bin/LDAPKVMWrapper.pl</code> once a day:
*** <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

==== Secondary-Master-Node (vm-node-02) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the Secondary-Master-Node]].

Log into the [[Secondary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-master-node

* In order to get the configuration from the primary-master-node, we need to access it via ssh
** Enter the IP for the primary-master-node on the admin interface (if you followed this documentation it is 10.1.130.13)
** Enter the username (if you followd the default setup it is root)
** Enter the users password.

* In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node, so enter the following information:
** Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11)
** Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root)
** Enter the users password
* Repeat the same procedure for the secondary-storage-node (if you followed this documentation the IP is 10.1.110.12)

* Enter the LDAP-Superuser password you defined during the [[Multi-Node Installation#Primary-Master-Node (stoney-cloud-node-01)_2 | primary-master-node]] installation

* Due tu [https://github.com/stepping-stone/node-integration/issues/9 bug #9], you need to manually finish the configuration of the libvirthook scripts:
** You mainly have to fill in the following variables:
*** '''libvirtHookFirewallSvnUser'''
*** '''libvirtHookFirewallSvnPassword'''
** See also [https://int.stepping-stone.ch/wiki/libvirt_Hooks#Config_for_test-environment this test configuration]
* Due to [https://github.com/stepping-stone/node-integration/issues/12 bug #12], you need to manually configure the LDAPKVMWrapper.pl script:
** Fill in the <code>/etc/Provisioning/Backup/LDAPKVMWrapper.conf</code> file
** Create a cronjob entry which runs the script <code>/usr/bin/LDAPKVMWrapper.pl</code> once a day:
*** <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

== Links ==
= Node Integration =
The following figure gives an overview what the node-integration script does for the different node types

[[File:node-integration.png | thumb | 500 px]]

= Old Documentation =
== Specialized Installation ==
=== Primary-Master-Node (vm-node-01) ===
If you configured a additional Backup Volume on the Storage Nodes, you want to mount them now in the VM-Node.

Log into the Primary-Master-Node and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

==== Manual Steps ====
In order to be able to migrate a VM from a carrier, a special user called transfer will be created.
<source lang="bash">
lvcreate -L 60G -n transfer local0

mkfs.xfs -L "OSBD_transfe" /dev/local0/transfer

cat << EOF >> /etc/fstab

LABEL=OSBD_transfe /home/transfer xfs noatime,nodev,nosuid,noexec 0 2
EOF

mount /home/transfer

useradd --comment "User which is used for VM disk file transfer between carriers" \
--create-home \
--system \
--user-group \
transfer

passwd transfer
</source>

Allow password authentication for the transfer user:
<source lang="bash">
$EDITOR /etc/ssh/sshd_config
</source>

<pre>
[...]

Match User transfer
PasswordAuthentication yes

[...]
</pre>
To apply the changes above, restart the SSH daemon:
<source lang="bash">
/etc/init.d/sshd restart
</source>

[[Category:stoney cloud]][[Category:Installation]]

stoney cloud: Multi-Node Installation

2014-06-27T12:05:08Z

Pat: /* Links */

== Overview ==
The [http://www.stoney-cloud.org/ stoney cloud] builds upon various standard open source components and can be run on commodity hardware. The final Multi-Node Setup consists of the following components:
* One [[Primary-Master-Node]] with an OpenLDAP Directory Server for the storage of the stoney cloud user and service related data with the web based management [[VM-Manager]] interface and the Linux kernel based virtualization technology.
* One [[Secondary-Master-Node]] with the Linux kernel based virtualization technology.
* Two [[Storage-Node | Storage-Nodes]] configured as a replicated and distributed data storage service based on [http://www.gluster.org/ GlusterFS].
The components communicate with each other over a standard Ethernet based IPv4 network.

=== Prerequisites ===
The following items and conditions are required to be able to install and configure a stoney cloud environment:
* Dedicated Hardware (4 Servers) which fulfil the following requirements:
** 64-Bit Intel with VT-Technologie (AMD is not tested at the moment).
** 8 Gigabyte Memory (more is better).
** 147 Gigabyte up to 2 Terabyte disks.
*** For all four Nodes ([[VM-Node | VM-Nodes]] and [[Storage-Node | Storage-Nodes]]) two 147 Gigabyte Disks, configured as RAID1, are enough for the [http://www.gentoo.org/ Gentoo Linux] operating system.
*** Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.
** Two physical Ethernet Interfaces, which support the same bandwidth (for example 1 Gigabit/s).
* Two Gigabit layer-2 switches supporting [http://en.wikipedia.org/wiki/IEEE_802.3ad IEEE 802.3ad] (dynamic link aggregation), [http://en.wikipedia.org/wiki/IEEE_802.1Q IEEE 802.1Q] (VLAN tagging) and stacking (optional, but recommended).
* Experience with Linux environments especially with [http://www.gentoo.org Gentoo Linux].
* Good experience with IP networking, because the Switches need to be configured manually (dynamic link aggregation and VLAN tagging).

=== Limitations ===
During the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system, the first two physical Ethernet Interfaces are automatically configured as a logical interface (bond0) and the four tagged VLANs are set up. If more than two physical Ethernet Interfaces are to be included in to the logical interface (bond0) or the physical Ethernet Interfaces have different bandwidths (for example 1 Gigabit/s and 10 Gigabit/s), the final logical interface (bond0) needs to be configured manually after the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system.
Only the first two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] are set up automatically. More [[Storage-Node | Storage-Nodes]] need to be integrated manually.

=== Network Overview ===
As stated before, a minimal multi node stoney cloud environment consist of two VM- and Storage-Nodes.
It is highly recommended to use IEEE 802.3ad link aggregation (bonding, trunking etc.) over two network cards and attach them as one logical link to the access switch.

It is out of scope of this document on how to configure the switches for stacking, link aggregation and VLAN tagging, consult the respective user manual.

There are two scenarios on how to connect the nodes to the network.

==== Network Overview: Physical Layer Scenario 1 (recommended) ====
The preferred solution is to use two switches which are stackable (supporting link aggregation over two switches). Connect the nodes and switches as illustrated below. Thus eliminating the single point of failure as presented in scenario 2.

<pre>
+----------------------+ +----------------------+
| | | |
| vm-node-01 | | vm-node-02 |
| | | |
+----------------------+ +----------------------+
bond0 | \ / | bond0
| \ / |
| \ / |
| \/ | ___
| /\ | ___( )___
| / \ | __( )__
| / \ | _( )_
+-------------+ +-------------+ _( )_
| switch-01 |===| switch-02 |-------------------(_ Corporate LAN/WAN _)
+-------------+ +-------------+ (_ _)
| \ / | (__ __)
| \ / | (___ ___)
| \ / | (___)
| \/ |
| /\ |
| / \ |
bond0 | / \ | bond0
+-----------------------+ +-----------------------+
| | | |
| tier1-storage-node-01 | | tier1-storage-node-02 |
| | | |
+-----------------------+ +-----------------------+
</pre>

==== Network Overview: Physical Layer Scenario 2 (use at your own risk) ====
If theres only one switch available (or the switches aren't stackable) connect the nodes as illustrated below. As you can see, the switch is as a single point of failure.
<pre>
+----------------------+ +----------------------+
| | | |
| vm-node-01 | | vm-node-02 |
| | | |
+----------------------+ +----------------------+ ___
bond0 \\ // bond0 ___( )___
\\ // __( )__
\\ // _( )_
+-------------+ _( )_
| switch-01 |-----------------------------(_ Corporate LAN/WAN _)
+-------------+ (_ _)
// \\ (__ __)
// \\ (___ ___)
bond0 // \\ bond0 (___)
+-----------------------+ +-----------------------+
| | | |
| tier1-storage-node-01 | | tier1-storage-node-02 |
| | | |
+-----------------------+ +-----------------------+
</pre>

==== Network overview: Logical layer ====
The goal is to achieve the following configuration:
<pre>
+----------------+----------------+----------------+----------------+
| 10.1.110.1X | 10.1.120.1X | 10.1.130.1X | 192.168.140.1X |
+----------------+----------------+----------------+----------------+
| | | | vmbr0 |
| vlan110 | vlan120 | vlan130 +----------------+
| | | | vlan140 |
+----------------+----------------+----------------+----------------+
+-------------------------------------------------------------------+
| bond0 (bonding.mode=802.3ad) |
+-------------------------------------------------------------------+
+----------------+ +----------------+
| eth0 | | eth1 |
+----------------+ +----------------+
</pre>

The ideal stoney cloud environment is based on four logical separated VLANs (virtual LANs):
* '''admin''': Administrative network, used for administration and monitoring purposes.
* '''data''': Data network, used for GlusterFS traffic.
* '''int''': Internal network, used for internal traffic such as LDAP, libvirt and more.
* '''pub''': Public network, used for accessing the VM-Manager webinterface, Spice traffic and internet access.

Each of the above VLANs hold dedicated services and separates them from each other. This documentation assumes, that the four VLANs are present and the following IP networks are available:

{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|VLAN name
!align="left"|VLAN ID
!align="left"|Network prefix
!align="left"|Default Gateway address
!align="left"|Broadcast address
!align="left"|Domain name
!align="left"|VIP
|-

|admin
|110
|10.1.110.0/24
| --
|10.1.110.255
|admin.example.com
|
|-

|data
|120
|10.1.120.0/24
| --
|10.1.120.255
|data.example.com
|
|-

|int
|130
|10.1.130.0/24
| --
|10.1.130.255
|int.example.com
|10.1.130.10
|-

|pub
|140
|192.168.140.0/24
|192.168.140.1
|192.168.140.255
|example.com
|192.168.140.10
|-

|}

The IP allocation of the nodes will be assumed as stated in the table below:
{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|Node name
!align="left"|Admin address (VLAN 110)
!align="left"|Data address (VLAN 120)
!align="left"|Int address (VLAN 130)
!align="left"|Pub address (VLAN 140)
|-

|-
|tier1-storage-node-01
|10.1.110.11
|10.1.120.11
|10.1.130.11
|192.168.140.11
|-

|-
|tier1-storage-node-02
|10.1.110.12
|10.1.120.12
|10.1.130.12
|192.168.140.12
|-

|-
|vm-node-01
|10.1.110.13
|10.1.120.13
|10.1.130.13
|192.168.140.13
|-

|-
|vm-node-02
|10.1.110.14
|10.1.120.14
|10.1.130.14
|192.168.140.14
|-

|}

We'll also presume, we have the following Domain Name Servers:
* Domain Name Server 1: '''192.168.140.1'''

== Base Installation ==
All nodes are based on the [http://www.gentoo.org/ Gentoo Linux] operating system.

=== BIOS Set Up Checklist ===
* Enable the "reboot-after-power-loss" option (if your BIOS supports it).
* Make sure, that you have the newest BIOS (BMC and perhaps SCSI firmware) version.
* Make sure, you've disabled halt on post errors (or similar) or enable keyboard-less operation (if your BIOS supports it).

=== RAID Set Up ===
Create a RAID1 volume. This RAID-Set is used for the Operating System. Please be aware, the the current stoney cloud only supports 147 Gigabyte up to 2 Terabyte disks for this first RAID-Set.

Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.

=== Node Installation ===
The first step of Semi-Automatic Multi-Node Set Up is the same for all four Nodes. In this documentation we presume, that you stick to the naming convention mentioned above. After the Base Installation of the Nodes, the following daemons will be running:
* '''crond''': Crond, to execute scheduled commands
* '''ntpd''': Network Time Protocol daemon, keeps the time synced with the Time from Servers in the LAN or WAN.
* '''sshd''': OpenSSH SSH daemon, used for remote access and remote administration.
* '''syslogd''': System Logging, keeps track of messages from the System and the Applications.
* '''udevd''': Linux dynamic device management, that manage events, symlinks and permissions of devices.

==== First Storage-Node (tier1-storage-node-01) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions as follows (the bold values are examples can be set through the administrator and are variable, according to the local setup):
## Global Section
### Confirm that you want to start? '''yes'''
### Choose a Node-Type: '''Storage-Node'''
### Choose a Block-Device: '''sda'''
### Confirm to erase all (from a previous installation): '''yes'''
### Confirm to continue with the given the Partition-Scheme: '''yes'''
### Choose the network interfaces to bond together.
#### Device #0: '''eth0'''
#### Device #1: '''eth1'''
### Node-Name: '''tier1-storage-node-01'''
## pub-VLAN Section
### VLAN ID: '''140'''
### Domain Name: '''example.com'''
### IP Address: '''192.168.140.11'''
### Netmask: '''24'''
### Broadcast: '''192.168.140.255'''
### Confirm the pub-VLAN Section with '''yes'''
## admin-VLAN Section
### VLAN ID: '''110'''
### Domain Name: '''admin.example.com'''
### IP Address: '''10.1.110.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.110.255'''
### Confirm the admin-VLAN Section with '''yes'''
## data-VLAN Section
### VLAN ID: '''120'''
### Domain Name: '''data.example.com'''
### IP Address: '''10.1.120.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.120.255'''
### Confirm the data-VLAN Section with '''yes'''
## int-VLAN Section
### VLAN ID: '''130'''
### Domain Name: '''int.example.com'''
### IP Address: '''10.1.130.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.130.255'''
### Confirm the int-VLAN Section with '''yes'''
### Enter the default Gateway: '''192.168.140.1'''
### Enter the primary DNS-Server: '''192.168.140.1'''
### Omit configuring a second DNS-Server with '''no'''
## Confirm the listed configuration with '''yes'''
## Enter your very secret root password
## Confirm to reboot with '''yes'''
# Make sure, that you boot from the first harddisk and not from the installation medium again.
# Continue with [[Multi-Node Installation#Specialized_Installation|specializing your Node]]

==== Second Storage-Node (tier1-storage-node-02) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

==== Primary-Master-Node (vm-node-01) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

==== Secondary-Master-Node (vm-node-02) ====
# Insert the stoney cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.

== Skipping Checks ==
To skip checks, type '''no''' when asked:
Do you want to start the installation?
yes or no?: '''no'''

Then manually restart the stoney cloud installer with the desired options. For example:
/mnt/cdrom/foss-cloud-installer -c

Options:
-c: Skip CPU requirement checks
-m: Skip memory requirement checks
-s: Skip CPU and memory requirement checks

== Specialized Installation ==
==== First Storage-Node (tier1-storage-node-01) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].

Log into the first [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-storage-node

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

==== Second Storage-Node (tier1-storage-node-02) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].

Log into the second [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-storage-node

For more information about the script and what it does, please visit the [[node-configuration]] script page.

==== Primary-Master-Node (vm-node-01) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the VM-Node]].

Log into the [[Primary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

The stoney cloud uses virtual ip addresses (VIPs) for fail over purposes. Therefore you need to configure [http://www.pureftpd.org/project/ucarp ucarp].

Confirm that you want to run the script.
Do you really want to proceed with configuration of the primary-master-node?
yes or no (default: no): '''yes'''

Enter the VIP (virtual IP) for the public interface. The apache will listen on this VIP because it is listening on the public interface (if you followed this documentation the VIP for the public interface is 192.168.140.10).
Please enter the VIP for the pub-interface (VLAN 140)
(default: 192.168.140.10): '''192.168.140.10'''

Enter the VIP (virtual IP) for the internal interace. The LDAP will listen on this VIP because it is listening on the internal interface (if you followed this documentation the VIP for the internal interface is 10.1.130.10).
Please enter the VIP for the int-interface (VLAN 130)
(default: 10.1.130.10): '''10.1.130.10'''

Enter the RIP (real IP) for the secondary-master-node on the internal interface. The RIP is needed to keep the LDAP directories synchronized (if you followed this documentation the RIP for the secondary-master-node on the internal interface is 10.1.130.14)
Please enter the IP for the int-interface (VLAN 130) of the
secondary-master-node (default: 10.1.130.14): '''10.1.130.14'''

The script now tests the network configuration and

In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node. Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the primary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':

IP address (default: 10.1.110.11): '''10.1.110.11'''
Username (default: root): '''root'''
Password for root: '''**********'''

In order to mount the gluster-filesystem, you need to connect via ssh to the secondary-storage-node. Enter the IP for the secondary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.12). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the secondary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':

IP address (default: 10.1.110.12): '''10.1.110.12'''
Username (default: root): '''root'''
Password for root: '''**********'''

* Configure the LDAP directory:
** Define the password for the LDAP-Superuser (cn=Manager,dc=stoney-cloud,dc=org)
** Currently the user for the prov-backup-kvm daemon is the LDAP-Superuser so enter the same password again
** Define the password for the LDAP-dhcp user (cn=dhcp,ou=services,ou=administration,dc=stoney-cloud,dc=org)
** Enter all necessary information for the stoney cloud administrator (User1)
*** Given name
*** Surname
*** Gender
*** E-mail
*** Language
*** Password

* Finally enter the domain name which will correspond to the public VIP (default is stoney-cloud.example.org)

* Due to [https://github.com/stepping-stone/node-integration/issues/9 bug #9], you need to manually finish the configuration of the libvirthook scripts:
** You mainly have to fill in the following variables:
*** '''libvirtHookFirewallSvnUser'''
*** '''libvirtHookFirewallSvnPassword'''
** See also [https://int.stepping-stone.ch/wiki/libvirt_Hooks#Config_for_test-environment this test configuration]
* Due to [https://github.com/stepping-stone/node-integration/issues/12 bug #12], you need to manually configure the LDAPKVMWrapper.pl script:
** Fill in the <code>/etc/Provisioning/Backup/LDAPKVMWrapper.conf</code> file
** Create a cronjob entry which runs the script <code>/usr/bin/LDAPKVMWrapper.pl</code> once a day:
*** <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

==== Secondary-Master-Node (vm-node-02) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the Secondary-Master-Node]].

Log into the [[Secondary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-master-node

* In order to get the configuration from the primary-master-node, we need to access it via ssh
** Enter the IP for the primary-master-node on the admin interface (if you followed this documentation it is 10.1.130.13)
** Enter the username (if you followd the default setup it is root)
** Enter the users password.

* In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node, so enter the following information:
** Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11)
** Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root)
** Enter the users password
* Repeat the same procedure for the secondary-storage-node (if you followed this documentation the IP is 10.1.110.12)

* Enter the LDAP-Superuser password you defined during the [[Multi-Node Installation#Primary-Master-Node (stoney-cloud-node-01)_2 | primary-master-node]] installation

* Due tu [https://github.com/stepping-stone/node-integration/issues/9 bug #9], you need to manually finish the configuration of the libvirthook scripts:
** You mainly have to fill in the following variables:
*** '''libvirtHookFirewallSvnUser'''
*** '''libvirtHookFirewallSvnPassword'''
** See also [https://int.stepping-stone.ch/wiki/libvirt_Hooks#Config_for_test-environment this test configuration]
* Due to [https://github.com/stepping-stone/node-integration/issues/12 bug #12], you need to manually configure the LDAPKVMWrapper.pl script:
** Fill in the <code>/etc/Provisioning/Backup/LDAPKVMWrapper.conf</code> file
** Create a cronjob entry which runs the script <code>/usr/bin/LDAPKVMWrapper.pl</code> once a day:
*** <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>

For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.

== Links ==
= Node Integration =

= Old Documentation =
== Specialized Installation ==
=== Primary-Master-Node (vm-node-01) ===
If you configured a additional Backup Volume on the Storage Nodes, you want to mount them now in the VM-Node.

Log into the Primary-Master-Node and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

==== Manual Steps ====
In order to be able to migrate a VM from a carrier, a special user called transfer will be created.
<source lang="bash">
lvcreate -L 60G -n transfer local0

mkfs.xfs -L "OSBD_transfe" /dev/local0/transfer

cat << EOF >> /etc/fstab

LABEL=OSBD_transfe /home/transfer xfs noatime,nodev,nosuid,noexec 0 2
EOF

mount /home/transfer

useradd --comment "User which is used for VM disk file transfer between carriers" \
--create-home \
--system \
--user-group \
transfer

passwd transfer
</source>

Allow password authentication for the transfer user:
<source lang="bash">
$EDITOR /etc/ssh/sshd_config
</source>

<pre>
[...]

Match User transfer
PasswordAuthentication yes

[...]
</pre>
To apply the changes above, restart the SSH daemon:
<source lang="bash">
/etc/init.d/sshd restart
</source>

[[Category:stoney cloud]][[Category:Installation]]

stoney backup: Server set-up

2014-06-27T07:41:57Z

Pat: /* Provisioning global configuration */

= Abstract =
This document describes server setup for the stoney cloud (Online) Backup service, built upon the [http://www.gentoo.org/ Gentoo] Linux distribution.

= Overview =
After working through this documentation, you will be able to set up and configure your own (Online) Backup service server.

= Software Installation =

== Requirements ==
A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].

== Keywords & USE-Flags ==
For a minimal OpenLDAP directory installation:
echo "net-nds/openldap minimal sasl" >> /etc/portage/package.use
echo "net-nds/openldap ~amd64" >> /etc/portage/package.keywords

NSS and PAM modules for lookups using LDAP:
echo "sys-auth/nss-pam-ldapd sasl" >> /etc/portage/package.use
echo "sys-auth/nss-pam-ldapd ~amd64" >> /etc/portage/package.keywords
echo "sys-fs/quota ldap" >> /etc/portage/package.use

echo "=app-admin/jailkit-2.16 ~amd64" >> /etc/portage/package.keywords

For the prov-backup-rsnapshot daemon:
echo "dev-perl/Net-SMTPS ~amd64" >> /etc/portage/package.keywords
echo "perl-core/Switch ~amd64" >> /etc/portage/package.keywords

To build puttygen only without X11:
echo "net-misc/putty ~amd64" >> /etc/portage/package.keywords
echo "net-misc/putty -gtk" >> /etc/portage/package.use

== Emerge ==
emerge -va nss-pam-ldapd \
quota \
net-misc/putty \
app-admin/jailkit \
sys-apps/haveged \
net-misc/putty \
sys-apps/sst-backup-utils \
sys-apps/sst-prov-backup-rsnapshot

To list the dependencies of ebuilds, you can use <code>equery</code>:
equery depgraph sst-backup-utils
<pre>
* Searching for sst-backup-utils ...

* dependency graph for sys-apps/sst-backup-utils-0.1.0
`-- sys-apps/sst-backup-utils-0.1.0 amd64
`-- dev-perl/PerlUtil-0.1.0 (>=dev-perl/PerlUtil-0.1.0) amd64
`-- virtual/perl-Sys-Syslog-0.320.0 (virtual/perl-Sys-Syslog) amd64
`-- dev-perl/perl-ldap-0.530.0 (dev-perl/perl-ldap) amd64
`-- dev-perl/XML-Simple-2.200.0 (dev-perl/XML-Simple) amd64
`-- dev-perl/Config-IniFiles-2.780.0 (dev-perl/Config-IniFiles) amd64
`-- dev-perl/XML-Validator-Schema-1.100.0 (dev-perl/XML-Validator-Schema) amd64
`-- dev-perl/Date-Calc-6.300.0 (dev-perl/Date-Calc) amd64
`-- dev-perl/DateManip-6.310.0 (dev-perl/DateManip) amd64
`-- dev-perl/Schedule-Cron-Events-1.930.0 (dev-perl/Schedule-Cron-Events) amd64
`-- dev-perl/DateTime-Format-Strptime-1.520.0 (dev-perl/DateTime-Format-Strptime) amd64
`-- dev-perl/XML-SAX-0.990.0 (dev-perl/XML-SAX) amd64
`-- virtual/perl-MIME-Base64-3.130.0-r2 (virtual/perl-MIME-Base64) amd64
`-- dev-perl/Authen-SASL-2.160.0 (dev-perl/Authen-SASL) amd64
`-- dev-perl/Net-SMTPS-0.30.0 (dev-perl/Net-SMTPS) ~amd64
`-- dev-perl/text-template-1.450.0 (dev-perl/text-template) amd64
`-- virtual/perl-Getopt-Long-2.380.0-r2 (virtual/perl-Getopt-Long) amd64
`-- dev-perl/Parallel-ForkManager-1.20.0 (dev-perl/Parallel-ForkManager) amd64
`-- dev-perl/Time-Stopwatch-1.0.0 (dev-perl/Time-Stopwatch) amd64
`-- app-backup/rsnapshot-1.3.1-r1 (app-backup/rsnapshot) amd64
[ sys-apps/sst-backup-utils-0.1.0 stats: packages (20), max depth (1) ]
</pre>

For more information, visit the [http://www.gentoo.org/doc/en/gentoolkit.xml Gentoolkit] page.

= Base Server Software Configuration =
== OpenSSH ==
=== OpenSSH Configuration ===
Configure the OpenSSH daemon:
<source lang="bash">
vi /etc/ssh/sshd_config
</source>

Set following options:
<source lang="bash">
PubkeyAuthentication yes
PasswordAuthentication yes
UsePAM yes
Subsystem sftp internal-sftp
</source>

Make sure, that <code>Subsystem sftp internal-sftp</code> is the last line in the configuration file.

We want to reduce the numbers of chroot environments in one folder. As the <code>ChrootDirectory</code> configuration option only allows <code>%h</code> (home directory of the user) and <code>%u</code> (username of the user), we need to create the necessary matching rules in the form of:
<source lang="bash">
Match User *000
ChrootDirectory /var/backup/000/%u
AuthorizedKeysFile /var/backup/000/%u/%h/.ssh/authorized_keys
Match
Match User *001
ChrootDirectory /var/backup/001/%u
AuthorizedKeysFile /var/backup/001/%u/%h/.ssh/authorized_keys
Match
...
Match User *999
ChrootDirectory /var/backup/999/%u
AuthorizedKeysFile /var/backup/999/%u/%h/.ssh/authorized_keys
Match
</source>

The creation of the matching rules is done by executing the following bash commands:
<source lang="bash">
FILE=/etc/ssh/sshd_config;

for x in {0..999} ; do \
printf "Match User *%03d\n" $x >> ${FILE}; \
printf " ChrootDirectory /var/backup/%03d/%%u\n" $x >> ${FILE}; \
printf " AuthorizedKeysFile /var/backup/%03d/%%u/%%h/.ssh/authorized_keys\n" $x >> ${FILE}; \
printf "Match\n" >> ${FILE}; \
done
</source>

Don't forget to restart the OpenSSH daemon:
<source lang="bash">
/etc/init.d/sshd restart
</source>

=== OpenSSH Host Keys ===
If you migrate from a existing backup server, you might want to copy the ssh host keys to the new server. If you do so clients want see a difference between the two hosts as the fingerprint remains the same. Copy the following files from the existing host to the new:
* /etc/ssh/ssh_host_dsa_key
* /etc/ssh/ssh_host_ecdsa_key
* /etc/ssh/ssh_host_key
* /etc/ssh/ssh_host_rsa_key
* /etc/ssh/ssh_host_dsa_key.pub
* /etc/ssh/ssh_host_ecdsa_key.pub
* /etc/ssh/ssh_host_key.pub
* /etc/ssh/ssh_host_rsa_key.pub

Set the correct permissions on the new host:
chmod 600 /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_ecdsa_key /etc/ssh/ssh_host_key /etc/ssh/ssh_host_rsa_key
chmod 644 /etc/ssh/*.pub

And restart the ssh daemon. ''Caution'': do not close your existing ssh session as long as you are not sure the ssh daemon has restarted properly and you can login again.
/etc/init.d/sshd restart

== OpenLDAP ==
=== /etc/hosts ===
Update the <code>/etc/hosts</code> with the LDAP server:
/etc/hosts

# VIP of the LDAP Server
31.216.40.4 ldapm.stoney-cloud.org

=== Root CA Certificate Installation ===
Install the root CA certificate into the OpenSSL default certificate storage directory:
fqdn="cloud.stoney-cloud.org" # The fully qualified domain name of the server containing the root certificate.

cd /etc/ssl/certs/
wget --no-check-certificate https://${fqdn}/ca/FOSS-Cloud_CA.cert.pem
chown root:root /etc/ssl/certs/FOSS-Cloud_CA.cert.pem
chmod 444 /etc/ssl/certs/FOSS-Cloud_CA.cert.pem

Rebuild the CA hashes
c_rehash /etc/ssl/certs/

=== /etc/openldap/ldap.conf ===
Update the <code>/etc/openldap/ldap.conf</code>LDAP configuration file/environment variables:
/etc/openldap/ldap.conf

<pre>
# Used to specify a size limit to use when performing searches. The number should be an
# non-negative integer. SIZELIMIT of zero (0) specifies unlimited search size.
SIZELIMIT 20000

# Used to specify a time limit to use when performing searches. The number should be an
# non-negative integer. TIMELIMIT of zero (0) specifies unlimited search time to be used.
TIMELIMIT 45

# Specify how aliases dereferencing is done. DEREF should be set to one of never, always, search,
# or find to specify that aliases are never dereferenced, always dereferenced, dereferenced when
# searching, or dereferenced only when locating the base object for the search. The default is to
# never dereference aliases.
DEREF never

# Specifies the URI(s) of an LDAP server(s) to which the LDAP library should connect. The URI
# scheme may be either ldapor ldaps which refer to LDAP over TCP and LDAP over SSL (TLS)
# respectively. Each server's name can be specified as a domain- style name or an IP address
# literal. Optionally, the server's name can followed by a ':' and the port number the LDAP
# server is listening on. If no port number is provided, the default port for the scheme is
# used (389 for ldap://, 636 for ldaps://). A space separated list of URIs may be provided.
URI ldaps://ldapm.stoney-cloud.org

# Used to specify the default base DN to use when performing ldap operations. The base must be
# specified as a Distinguished Name in LDAP format.
BASE dc=stoney-cloud,dc=org

# This is a local copy of the certificate of the certificate authority
# used to sign the server certificate for the LDAP server I am using
TLS_CACERT /etc/ssl/certs/FOSS-Cloud_CA.cert.pem
</pre>

Check you configuration by doing a search:
ldapsearch -v -H "ldaps://ldapm.stoney-cloud.org" \
-b "dc=stoney-cloud,dc=org" \
-D "cn=Manager,dc=stoney-cloud,dc=org" \
-s one "(objectClass=*)" \
-LLL -W

The result should look something like:
ldap_initialize( ldaps://ldapm.stoney-cloud.org:636/??base )
filter: (objectClass=*)
requesting: All userApplication attributes
dn: ou=administration,dc=stoney-cloud,dc=org
objectClass: top
objectClass: organizationalUnit
ou: administration
...

== Random Number Generator (haveged) ==
Tools like putty are dependent on random numbers to be able to create certificates.

=== haveged - Generate random numbers and feed linux random device ===
The haveged daemon doesn't need any special configuration, therefore you can start it from the command line interface:
/etc/init.d/haveged start

Check, if the start was successful:
ps auxf | grep haveged

root 18001 1.0 0.0 7420 3616 ? Ss 08:48 0:00 /usr/sbin/haveged -r 0 -w 1024 -v 1

Add the haveged daemon to the default run level:
rc-update add haveged default

== nss-pam-ldapd ==
=== nslcd.conf — configuration file for LDAP nameservice daemon ===
/etc/nslcd.conf

<pre>
# This is the configuration file for the LDAP nameservice
# switch library's nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The uri pointing to the LDAP server to use for name lookups.
# Multiple entries may be specified. The address that is used
# here should be resolvable without using LDAP (obviously).
#uri ldap://127.0.0.1/
#uri ldaps://127.0.0.1/
#uri ldapi://%2fvar%2frun%2fldapi_sock/
# Note: %2f encodes the '/' used as directory separator
uri ldaps://ldapm.tombstone.ch

# The LDAP version to use (defaults to 3
# if supported by client library)
#ldap_version 3

# The distinguished name of the search base.
base dc=stoney-cloud,dc=org

# The distinguished name to bind to the server with.
# Optional: default is to bind anonymously.
binddn cn=Manager,dc=stoney-cloud,dc=org

# The credentials to bind with.
# Optional: default is no credentials.
# Note that if you set a bindpw you should check the permissions of this file.
bindpw myverysecretpassword

# The distinguished name to perform password modifications by root by.
#rootpwmoddn cn=admin,dc=example,dc=com

# The default search scope.
#scope sub
#scope one
#scope base

# Customize certain database lookups.
#base group ou=Groups,dc=example,dc=com
base group ou=groups,ou=backup,ou=services,dc=stoney-cloud,dc=org
base passwd ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
base shadow ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
#scope group onelevel
#scope hosts sub

#filter group (&(objectClass=posixGroup)(sstIsActive=TRUE))
filter passwd (&(objectClass=posixAccount)(sstIsActive=TRUE))
filter shadow (&(objectClass=shadowAccount)(sstIsActive=TRUE))

# Bind/connect timelimit.
#bind_timelimit 30

# Search timelimit.
#timelimit 30

# Idle timelimit. nslcd will close connections if the
# server has not been contacted for the number of seconds.
#idle_timelimit 3600

# Use StartTLS without verifying the server certificate.
#ssl start_tls
tls_reqcert never

# CA certificates for server certificate verification
#tls_cacertdir /etc/ssl/certs
#tls_cacertfile /etc/ssl/ca.cert

# Seed the PRNG if /dev/urandom is not provided
#tls_randfile /var/run/egd-pool

# SSL cipher suite
# See man ciphers for syntax
#tls_ciphers TLSv1

# Client certificate and key
# Use these, if your server requires client authentication.
#tls_cert
#tls_key

# Mappings for Services for UNIX 3.5
#filter passwd (objectClass=User)
#map passwd uid msSFU30Name
#map passwd userPassword msSFU30Password
#map passwd homeDirectory msSFU30HomeDirectory
#map passwd homeDirectory msSFUHomeDirectory
#filter shadow (objectClass=User)
#map shadow uid msSFU30Name
#map shadow userPassword msSFU30Password
#filter group (objectClass=Group)
#map group member msSFU30PosixMember

# Mappings for Services for UNIX 2.0
#filter passwd (objectClass=User)
#map passwd uid msSFUName
#map passwd userPassword msSFUPassword
#map passwd homeDirectory msSFUHomeDirectory
#map passwd gecos msSFUName
#filter shadow (objectClass=User)
#map shadow uid msSFUName
#map shadow userPassword msSFUPassword
#map shadow shadowLastChange pwdLastSet
#filter group (objectClass=Group)
#map group member posixMember

# Mappings for Active Directory
#pagesize 1000
#referrals off
#idle_timelimit 800
#filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map passwd uid sAMAccountName
#map passwd homeDirectory unixHomeDirectory
#map passwd gecos displayName
#filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map shadow uid sAMAccountName
#map shadow shadowLastChange pwdLastSet
#filter group (objectClass=group)

# Alternative mappings for Active Directory
# (replace the SIDs in the objectSid mappings with the value for your domain)
#pagesize 1000
#referrals off
#idle_timelimit 800
#filter passwd (&(objectClass=user)(objectClass=person)(!(objectClass=computer)))
#map passwd uid cn
#map passwd uidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820
#map passwd gidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820
#map passwd homeDirectory "/home/$cn"
#map passwd gecos displayName
#map passwd loginShell "/bin/bash"
#filter group (|(objectClass=group)(objectClass=person))
#map group gidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820

# Mappings for AIX SecureWay
#filter passwd (objectClass=aixAccount)
#map passwd uid userName
#map passwd userPassword passwordChar
#map passwd uidNumber uid
#map passwd gidNumber gid
#filter group (objectClass=aixAccessGroup)
#map group cn groupName
#map group gidNumber gid
</pre>

=== nsswitch.conf - Name Service Switch configuration file ===
/etc/nsswitch.conf

<pre>
passwd: files ldap
shadow: files ldap
group: files ldap

# passwd: db files nis
# shadow: db files nis
# group: db files nis

hosts: files dns
networks: files dns

services: db files
protocols: db files
rpc: db files
ethers: db files
netmasks: files
netgroup: files
bootparams: files

automount: files
aliases: files
</pre>

=== system-auth ===
vi /etc/pam.d/system-auth

<pre>
auth required pam_env.so
auth sufficient pam_unix.so try_first_pass likeauth nullok
auth sufficient pam_ldap.so minimum_uid=1000 use_first_pass
auth required pam_deny.so

account required pam_unix.so
account sufficient pam_ldap.so minimum_uid=1000 use_first_pass

password required pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
password required pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password sufficient pam_ldap.so minimum_uid=1000 use_first_pass
password required pam_deny.so

session required pam_limits.so
session required pam_env.so
session required pam_unix.so
session sufficient pam_ldap.so minimum_uid=1000 use_first_pass
</pre>

=== Test the Setup ===
nslcd -d

=== Update the Default Run Levels ===
rc-update add nslcd default
rc-update add nscd default

=== Start the necessary Daemons ===
/etc/init.d/nslcd start
/etc/init.d/nscd start

== Quota ==
=== 32-bit Project Identifier Support ===
We need to enable 32-bit project identifier support (PROJID32BIT feature) for our naming scheme (uid numbers larger than 65'536), which is already the default on the stepping stone virtual machines:
mkfs.xfs '''-i projid32bit=1''' /dev/vg-local-01/var

=== Update /etc/fstab and Mount ===
Make sure, that you have user quota (uqota) and project quota (pquota) set as options on the chosen mount point in /etc/fstab. For example:
LABEL=LV-VAR /var xfs noatime,discard,inode64,uquota,pquota 0 2

reboot

Check, if everything went ok:
df -h | grep var

/dev/mapper/vg--local--01-var 1023G 220G 804G 22% /var

=== Verify ===
Some important options for xfs_quota:
* -x: Enable expert mode.
* -c: Pass arguments on the command line. Multiple arguments may be given.

Remount the file system /var and check, if /var has the desired values:
xfs_quota -x -c state /var

As you can see (items marked bold), we have achieved our goal:
User quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: '''ON'''
Enforcement: '''ON'''
Inode: #131 (1 blocks, 1 extents)
Group quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: OFF
Enforcement: OFF
Inode: #132 (1 blocks, 1 extents)
Project quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: '''ON'''
Enforcement: '''ON'''
Inode: #132 (1 blocks, 1 extents)
Blocks grace time: [7 days 00:00:30]
Inodes grace time: [7 days 00:00:30]
Realtime Blocks grace time: [7 days 00:00:30]

=== User Quotas ===
==== Adding a User Quota ====
Set a quota of 1 Gigabyte for the user 4000187 (the values are in kilobytes, so 1048576 kilobyte are 1024 megabytes which corresponds to 1 gigabyte):
xfs_quota -x -c 'limit bhard=1048576k 4000187' /var

Or in bytes:
xfs_quota -x -c 'limit bhard=1073741824 4000187' /var

Read the quota information for the user 4000187:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If the user has data in the project, that belongs to him, the result will change:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

==== Modifiying a User Quota ====
To modify a users quota, you just set a new quota (limit):
xfs_quota -x -c 'limit bhard=1048576k 4000187' /var

Read the quota information for the user 4000187:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If the user has data in the project, that belongs to him, the result will change:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

==== Removing a User Quota ====
Removing a quota for a user:
xfs_quota -x -c 'limit bhard=0 4000187' /var

The following command should give you an empty result:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

=== Project (Directory) Quotas ===
==== Adding a Project (Directory) Quota ====
The XFS file system additionally allows you to set quotas on individual directory hierarchies in the file system that are known as managed trees. Each managed tree is uniquely identified by a project ID and an optional project name. We'll use the following values in the examples:
* project_ID: The uid of the online backup account (4000187).
* project_name: The uid of the online backup account (4000187). This could be a human readable name.
* mountpoint: The mountpoint of the xfs-filesystem (/var). See the <code>/etc/fstab</code> entry from above.
* directory: The directory of the project (187/4000187), starting from the mountpoint of the xfs-filesystem (/var).

Define a unique project ID for the directory hierarchy in the <code>/etc/projects</code> file (project_ID:mountpoint/directory):
echo "4000187:/var/backup/187/4000187/home/4000187" >> /etc/projects

Create an entry in the <code>/etc/projid</code> file that maps a project name to the project ID (project_name:project_ID):
echo "4000187:4000187" >> /etc/projid

Set Project:
xfs_quota -x -c 'project -s -p /var/backup/187/4000187/home/4000187 4000187' /var

Set Quota (limit) on Project:
xfs_quota -x -c 'limit -p bhard=1048576k 4000187' /var

Check your Quota (limit)
xfs_quota -x -c 'quota -p 4000187' /var

Check the Quota:
* <code>-v</code>: increase verbosity in reporting (also dumps zero values).
* <code>-N</code>: suppress the initial header.
* <code>-p</code>: display project quota information.
* <code>-h</code>: human readable format.
xfs_quota -x -c 'quota -v -N -p 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If you copied data into the project, the output will look something like:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

To give you an overall view of the whole system:
xfs_quota -x -c report /var

<pre>
User quota on /var (/dev/mapper/vg--local--01-var)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 1024000 0 0 00 [--------]
4000187 0 0 1048576 00 [--------]

Project quota on /var (/dev/mapper/vg--local--01-var)
Blocks
Project ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
4000187 512000 0 1048576 00 [--------]
</pre>

==== Modifying a Project (Directory) Quota ====
To modify a project (directory) quota, you just set an new quota (limit) on the chosen project:
xfs_quota -x -c 'limit -p bhard=1048576k 4000187' /var

Check your quota (limit)
xfs_quota -x -c 'quota -p 4000187' /var

==== Removing a Project (Directory) Quota ====
Removing a quota from a project:
xfs_quota -x -c 'limit -p bhard=0 4000187' /var

Chreck the results:
xfs_quota -x -c report /var

<pre>
User quota on /var (/dev/mapper/vg--local--01-var)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 512000 0 0 00 [--------]
4000187 0 0 1024 00 [--------]
</pre>

As you can see, the line with the Project ID 4000187 has disappeared:
4000187 512000 0 1048576 00 [--------]

Don't forget to remove the project from <code>/etc/projects</code> and <code>/etc/projid</code>:
sed -i -e '/4000187/d' /etc/projects
sed -i -e '/4000187/d' /etc/projid

=== Some important notes concerning XFS ===
# The '''quotacheck''' command has no effect on XFS filesystems. The first time quota accounting is turned on (at mount time), XFS does an automatic quotacheck internally; afterwards, the quota system will always be completely consistent until quotas are manually turned off.
# There is '''no need for quota file(s)''' in the root of the XFS filesystem.

== prov-backup-rsnapshot ==
Install the [[stoney_backup:_prov-backup-rsnapshot | prov-backup-rsnasphot ]] daemon script using the package manager:
<pre>
emerge -va sys-apps/sst-prov-backup-rsnapshot
</pre>

=== Configuration ===
If it is the first provisioning module running on this server (very likely) you first have to configure the provisioning daemon (you can skip this step if you have already another provisioning module running on this server)

==== Provisioning global configuration ====
The global configuration for the provisioning daemon (which was installed with the first provisioning module and the <code>sys-apps/sst-provisioning</code> package) applies to all provisioning modules running on the server. This configuration therefore contains information about the provisioning daemon itself and no information at all about the specific modules.
/etc/Provisioning/Global.conf
<pre>
# Copyright (C) 2012 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 0

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The number of seconds to wait before retry contacting the backend server during startup.
SLEEP = 10

# Number of backend server connection retries during startup.
ATTEMPTS = 3

[Operation Mode]
# The number of seconds to wait before retry contacting the backend server in case of a service interruptions.
SLEEP = 30

# Number of backend server connection retries in case of a service interruptions.
ATTEMPTS = 3

[Mail]
# Error messages are sent to the mail configured below.
SENDTO = <YOUR-MAIL-ADDRESS>
HOST = mail.stepping-stone.ch
PORT = 587
USERNAME = <YOUR-NOTIFICATION-EMAIL-ADDRESS>
PASSWORD = <PASSWORD>
FROMNAME = Provisioning daemon
CA_DIR = /etc/ssl/certs
SSL = starttls
AUTH_METHOD = LOGIN

# Additionally, you can be informed about creation, modification and deletion of services.
WANTINFOMAIL = 1
</pre>

==== Provisioning daemon prov-backup-rsnapshot module ====
The module specific configuration is located in /etc/Provisioning/<Service>/<Type>.conf. In the case of the prov-backup-rsnapshot module this is <code>/etc/Provisioning/Backup/Rsnapshot.conf</code>. (Note: Comments starting with /* are not in the configuration file, they are only in the wiki to add some additional information)

<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

/* If you want, you can override the log information from the global configuration file this might be useful for debugging */
[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

/* Specify the hosts fully qualified domain name. This name will be used to perform some checks and also appear in the information and error mails */
ENVIRONMENT = <FQDN>

[Database]
BACKEND = LDAP
SERVER = ldaps://ldapm.tombstone.org
PORT = 636
ADMIN_USER = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <PASSWORD>
SERVICE_SUBTREE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
COOKIE_FILE = /etc/Provisioning/Backup/rsnapshot.cookie
DEFAULT_COOKIE = rid=001,csn=
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(sstProvisioningState=0))

/* Specifies the service itself. As it is the prov-backup-rsnapshot module, the SERVICE is "Backup" and the TYPE is "Rsnapshot".
* The MODUS is as usual selfcare and the TRANSPORTAPI is LocalCLI. This is because the daemon is running on the same host as the
* backup accounts are provisioned and the commands can be executed on this host using the cli.
* For more information about MODUS and TRANSPORTAPI see https://int.stepping-stone.ch/wiki/provisioning.pl#Service_Konfiguration
*/
[Service]
MODUS = selfcare
TRANSPORTAPI = LocalCLI
SERVICE = Backup
TYPE = Rsnapshot

SYSLOG = prov-backup-rsnapshot

/* For the TRANSPORTAPI LocalCLI there is no gateway required because there is no connection to establish. So set HOST, USER and
* DSA_FILE to whatever you want. Don't leave it blank, otherwise the provisioning daemon would log some error messages saying
* these attributes are empty
*/
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

/* Information about the backup itself (how to setup everything). Note that the %uid% int the RSNAPSHOT_CONFIG_FILE parameter will
* be replaced by the accounts UID. The script CREATE_CHROOT_CMD was installed with the prov-backup-rsnapshot module, so do not
* change this parameter. The quota parameters (SET_QUOTA_CMD, MOUNTPOINT, QUOTA_FILE, PROJECTS_FILE and PROJID_FILE) represent
* the quota setup as described on http://wiki.stoney-cloud.org/index.php/stoney_backup:_Server_set-up#Quota. If you followed this
* manual, you can copy-paste them into your configuration file, otherwise adapt them according to your quota setup.
*/
[Backup]
RSNAPSHOT_CONFIG_FILE = /etc/rsnapshot/rsnapshot.conf.%uid%
SET_QUOTA_CMD = /usr/sbin/xfs_quota
CREATE_CHROOT_CMD = /usr/libexec/createBackupDirectory.sh
MOUNTPOINT = /var
QUOTA_FILE = /etc/backupSize
PROJECTS_FILE = /etc/projects
PROJID_FILE = /etc/projid
</pre>

== backup utils ==
Install the backup utils (multiple scripts which help you to manage and monitor your backup server and backup accounts) using the package manager. For more information about the scripts please see the [[stoney_backup:_Service_Software | stoney backup Service Software]] page.
<pre>
emerge -va sys-apps/sst-backup-utils
</pre>
=== Configuration ===
Please refer to the configuration sections for the different scripts in [[stoney_backup:_Service_Software | stoney backup Service Software]].

= Links =
* [http://www.openldap.org/ OpenLDAP], an open source implementation of the Lightweight Directory Access Protocol.
* [http://arthurdejong.org/nss-pam-ldapd/ nss-pam-ldapd], a Name Service Switch (NSS) module that allows your LDAP server to provide user account, group, host name, alias, netgroup, and basically any other information that you would normally get from /etc flat files or NIS.
* [http://www.gentoo.org/doc/de/ldap-howto.xml Gentoo Leitfaden zur OpenLDAP Authentifikation].
* [http://wiki.gentoo.org/wiki/Centralized_authentication_using_OpenLDAP Centralized authentication using OpenLDAP].
* [https://code.google.com/p/openssh-lpk/source/browse/trunk/schemas/openssh-lpk_openldap.schema openssh-lpk_openldap.schema] OpenSSH LDAP Public Keys.
* [http://sourceforge.net/projects/linuxquota/ linuxquota] Linux DiskQuota.
* [http://www.rsnapshot.org/ rsnapshot], a remote filesystem snapshot utility, based on rsync.
* [http://olivier.sessink.nl/jailkit/ Jailkit], set of utilities to limit user accounts to specific files using chroot() and or specific commands. Also includes a tool to build a chroot environment.
* [http://www.busybox.net/ Busybox] BusyBox combines tiny versions of many common UNIX utilities into a single small executable. Useful to reduce the number of files (and thus the complexity) when building a chroot.

[[Category:stoney backup]]

stoney conductor: VM Backup

2014-06-26T14:57:22Z

Pat: /* Restore */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (create-, export- and commitSnapshot) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commit command, it sets the '''sstProvisioningMode''' to '''comitted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the comitting of the changes from the overlay disk-images back to the underlying ones is done.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: comitted
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''committed''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the KVMBackupWrapper.pl script with the machines list as a parameter
# Wait for the KVMBackupWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
'''Attention:''' The restore process is not yet defined / nor implemented. The following documentation is about the old restore process.
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
'''Attention''': The restore process is not yet defined / nor implemented. The following documentation is about the old restore process.

* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#Current_Implementation_.28Backup.29]]), the restore process does not work when clicking the icon in the webinterface.

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: prov-backup-kvm

2014-06-26T14:55:54Z

Pat: /* Backend */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the createsSnaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the commitSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
''' Restore is currently not implemented '''

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
''' Currently the backend configuration is not active. If it is active it will be (maybe some minor modifications) as the following: '''

In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script:
<pre>
use constant
{
SUCCESS_CODE => 0,
ERROR_CODE => 1,

TRUE => 1,
FALSE => 0,

# Specific error codes
TEMPLATE_NOT_READABLE => 101,
NO_MACHINE => 102,
NO_XML_DESCRIPTION => 103,
CANNOT_CREATE_SNAPSHOT => 104,
NO_STATE_INFORMATION => 105,
CANNOT_START_MACHINE => 106,
NO_BACKUP_LOCATION => 107,
UNCONSISTENT_BACKUP => 108,
CANNOT_GET_SNAPSHOT => 109,
CANNOT_DELETE_SNAPSHOT => 110,
CANNOT_UPDATE_XML => 111,
NO_SPACE_LEFT => 112,
};
</pre>

= Next steps =
* Implement restore

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stoney-cloud/stoney-conductor/tree/master/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: prov-backup-kvm

2014-06-26T14:54:39Z

Pat: /* Exit codes */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the createsSnaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the commitSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
''' Restore is currently not implemented '''

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script:
<pre>
use constant
{
SUCCESS_CODE => 0,
ERROR_CODE => 1,

TRUE => 1,
FALSE => 0,

# Specific error codes
TEMPLATE_NOT_READABLE => 101,
NO_MACHINE => 102,
NO_XML_DESCRIPTION => 103,
CANNOT_CREATE_SNAPSHOT => 104,
NO_STATE_INFORMATION => 105,
CANNOT_START_MACHINE => 106,
NO_BACKUP_LOCATION => 107,
UNCONSISTENT_BACKUP => 108,
CANNOT_GET_SNAPSHOT => 109,
CANNOT_DELETE_SNAPSHOT => 110,
CANNOT_UPDATE_XML => 111,
NO_SPACE_LEFT => 112,
};
</pre>

= Next steps =
* Implement restore

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stoney-cloud/stoney-conductor/tree/master/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: prov-backup-kvm

2014-06-26T14:53:47Z

Pat: /* Source Code */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the createsSnaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the commitSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
''' Restore is currently not implemented '''

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Implement restore

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stoney-cloud/stoney-conductor/tree/master/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: prov-backup-kvm

2014-06-26T14:53:00Z

Pat: /* Next steps */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the createsSnaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the commitSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
''' Restore is currently not implemented '''

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Implement restore

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: prov-backup-kvm

2014-06-26T14:52:28Z

Pat: /* Restore */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the createsSnaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the commitSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
''' Restore is currently not implemented '''

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: prov-backup-kvm

2014-06-26T14:52:00Z

Pat: /* commitSnapshot */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the createsSnaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the commitSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
* Task for the control-instance daemon
* Task for the prov-backup-kvm daemon
* Task for the vm-manager
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]

See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

File:KVM-Backup-Workflow-Retain.png

2014-06-26T14:51:35Z

Pat: Pat uploaded a new version of "File:KVM-Backup-Workflow-Retain.png": New commit method for committing changes from overlay back to underlay image

Detailed workflow for the retain process

stoney conductor: prov-backup-kvm

2014-06-26T14:50:52Z

Pat: /* createSnapshot */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the createsSnaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
* Task for the control-instance daemon
* Task for the prov-backup-kvm daemon
* Task for the vm-manager
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]

See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: prov-backup-kvm

2014-06-26T14:50:38Z

Pat: /* exportSnapshot */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the snaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
* Task for the control-instance daemon
* Task for the prov-backup-kvm daemon
* Task for the vm-manager
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]

See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

File:KVM-Backup-Workflow-Merge.png

2014-06-26T14:50:09Z

Pat: Pat uploaded a new version of "File:KVM-Backup-Workflow-Merge.png": New export method with only copy the live image away

Detailed workflow for the merge process

File:KVM-Backup-Workflow-detailed.xmi

2014-06-26T14:49:05Z

Pat: Pat uploaded a new version of "File:KVM-Backup-Workflow-detailed.xmi": The new backup process with disk only snapshot and committing changes back

Detailed workflow for the three subprocesses (snapshot, merge and retain)

File:KVM-Backup-Workflow-Snapshot.png

2014-06-26T14:48:22Z

Pat: Pat uploaded a new version of "File:KVM-Backup-Workflow-Snapshot.png": New createSnapshot method with disk only snapshots

Detailed snapshot workflow

File:KVM-Backup-simple.xmi

2014-06-26T14:01:37Z

Pat: Pat uploaded a new version of "File:KVM-Backup-simple.xmi"

Simplified prov-backup-kvm workflow

File:KVM-Backup-Workflow.png

2014-06-26T14:01:09Z

Pat: Pat uploaded a new version of "File:KVM-Backup-Workflow.png": Adapted to the new sub steps (create-, export- and commitSnapshots)

Simplified prov-backup-kvm workflow

stoney conductor: prov-backup-kvm

2014-06-26T14:00:25Z

Pat: /* Backup */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the snaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the merge process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
* Task for the control-instance daemon
* Task for the prov-backup-kvm daemon
* Task for the vm-manager
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]

See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: VM Backup

2014-06-26T13:58:20Z

Pat: /* Current Implementation (Restore) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (create-, export- and commitSnapshot) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commit command, it sets the '''sstProvisioningMode''' to '''comitted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the comitting of the changes from the overlay disk-images back to the underlying ones is done.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: comitted
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''committed''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the KVMBackupWrapper.pl script with the machines list as a parameter
# Wait for the KVMBackupWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
'''Attention''': The restore process is not yet defined / nor implemented. The following documentation is about the old restore process.

* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#Current_Implementation_.28Backup.29]]), the restore process does not work when clicking the icon in the webinterface.

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:54:55Z

Pat: /* Current Implementation (Backup) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (create-, export- and commitSnapshot) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commit command, it sets the '''sstProvisioningMode''' to '''comitted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the comitting of the changes from the overlay disk-images back to the underlying ones is done.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: comitted
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''committed''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the KVMBackupWrapper.pl script with the machines list as a parameter
# Wait for the KVMBackupWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:52:47Z

Pat: /* Backup a single machine */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (create-, export- and commitSnapshot) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commit command, it sets the '''sstProvisioningMode''' to '''comitted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the comitting of the changes from the overlay disk-images back to the underlying ones is done.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: comitted
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''committed''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:52:15Z

Pat: /* Step 12: Finalizing the Backup Process (Control instance daemon) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commit command, it sets the '''sstProvisioningMode''' to '''comitted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the comitting of the changes from the overlay disk-images back to the underlying ones is done.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: comitted
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''committed''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:51:56Z

Pat: /* Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commit command, it sets the '''sstProvisioningMode''' to '''comitted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the comitting of the changes from the overlay disk-images back to the underlying ones is done.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: comitted
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:50:47Z

Pat: /* Step 10: Starting the Retain Process (Provisioning-Backup-KVM daemon) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the commit Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the commit command, it sets the '''sstProvisioningMode''' to '''comitting''' to tell the Control instance daemon and other interested parties, that it is committing changes from the overlay disk-images back to the underlying ones.
<pre>
# The attribute sstProvisioningMode is set to comitting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: committing
</pre>

==== Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the retain command, it sets the '''sstProvisioningMode''' to '''retained''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the retaining of all the necessary files to the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retained
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:49:35Z

Pat: /* Step 09: Start the commit Process (Control instance daemon) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''commit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the Retain Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the retain command, it sets the '''sstProvisioningMode''' to '''retaining''' to tell the Control instance daemon and other interested parties, that it is retaining the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningMode is set to retaining by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: retaining
</pre>

==== Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the retain command, it sets the '''sstProvisioningMode''' to '''retained''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the retaining of all the necessary files to the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retained
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:49:24Z

Pat: /* Step 09: Start the Retain Process (Control instance daemon) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the commit Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''copmmit''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to commit the changes from the overlay file to the underlying disk-image
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# commit (this way the Provisioning-Backup-VKM daemon knows, that it must start the commit process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: commit
</pre>

==== Step 10: Starting the Retain Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the retain command, it sets the '''sstProvisioningMode''' to '''retaining''' to tell the Control instance daemon and other interested parties, that it is retaining the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningMode is set to retaining by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: retaining
</pre>

==== Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the retain command, it sets the '''sstProvisioningMode''' to '''retained''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the retaining of all the necessary files to the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retained
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:48:23Z

Pat: /* Step 08: Finalizing the Merging Process (Provisioning-Backup-KVM daemon) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the export command, it sets the '''sstProvisioningMode''' to '''exported''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the export of the virtual machine or virtual machine template disk-images is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: exported
</pre>

==== Step 09: Start the Retain Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''retain''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to retain (copy and then delete) all the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# retain (this way the Provisioning-Backup-VKM daemon knows, that it must start the retaining process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retain
</pre>

==== Step 10: Starting the Retain Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the retain command, it sets the '''sstProvisioningMode''' to '''retaining''' to tell the Control instance daemon and other interested parties, that it is retaining the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningMode is set to retaining by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: retaining
</pre>

==== Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the retain command, it sets the '''sstProvisioningMode''' to '''retained''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the retaining of all the necessary files to the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retained
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:47:31Z

Pat: /* Step 07: Starting the Merge Process (Provisioning-Backup-KVM daemon) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the export Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the export command, it sets the '''sstProvisioningMode''' to '''exporting''' to tell the Control instance daemon and other interested parties, that it is exporting the virtual machine or virtual machine template disk images.
<pre>
# The attribute sstProvisioningMode is set to exporting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: exporting
</pre>

==== Step 08: Finalizing the Merging Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the merge command, it sets the '''sstProvisioningMode''' to '''merged''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the merging of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: merged
</pre>

==== Step 09: Start the Retain Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''retain''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to retain (copy and then delete) all the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# retain (this way the Provisioning-Backup-VKM daemon knows, that it must start the retaining process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retain
</pre>

==== Step 10: Starting the Retain Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the retain command, it sets the '''sstProvisioningMode''' to '''retaining''' to tell the Control instance daemon and other interested parties, that it is retaining the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningMode is set to retaining by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: retaining
</pre>

==== Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the retain command, it sets the '''sstProvisioningMode''' to '''retained''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the retaining of all the necessary files to the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retained
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: VM Backup

2014-06-26T13:46:41Z

Pat: /* Step 06: Start the Merge Process (Control instance daemon) */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the export Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''export''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to export the disk image to the backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# export (this way the Provisioning-Backup-VKM daemon knows, that it must start the export process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: export
</pre>

==== Step 07: Starting the Merge Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the merge command, it sets the '''sstProvisioningMode''' to '''merging''' to tell the Control instance daemon and other interested parties, that it is merging the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to merging by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: merging
</pre>

==== Step 08: Finalizing the Merging Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the merge command, it sets the '''sstProvisioningMode''' to '''merged''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the merging of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: merged
</pre>

==== Step 09: Start the Retain Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''retain''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to retain (copy and then delete) all the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# retain (this way the Provisioning-Backup-VKM daemon knows, that it must start the retaining process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retain
</pre>

==== Step 10: Starting the Retain Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the retain command, it sets the '''sstProvisioningMode''' to '''retaining''' to tell the Control instance daemon and other interested parties, that it is retaining the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningMode is set to retaining by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: retaining
</pre>

==== Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the retain command, it sets the '''sstProvisioningMode''' to '''retained''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the retaining of all the necessary files to the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retained
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

File:wrapper-interaction.png

2014-06-26T13:43:55Z

Pat: Pat uploaded a new version of "File:wrapper-interaction.png": Adapted to the new sub steps (create-, export- and commitSnapshot)

Shows how the two wrapper interact with the LDAP backend

File:wrapper-interaction.png

2014-06-26T13:43:32Z

Pat: Pat uploaded a new version of "File:wrapper-interaction.png": Reverted to version as of 13:40, 26 June 2014

Shows how the two wrapper interact with the LDAP backend

File:wrapper-interaction.png

2014-06-26T13:43:09Z

Pat: Pat uploaded a new version of "File:wrapper-interaction.png": Adapted to the new sub steps (create-, export- and commitSnapshot)

Shows how the two wrapper interact with the LDAP backend

File:Daemon-communication.png

2014-06-26T13:41:16Z

Pat: Pat uploaded a new version of "File:Daemon-communication.png": Adapted to the new sub-steps (create-, export- and commitSnapshot)

Communication between control instance and prov-backup-kvm daemon

File:wrapper-interaction.png

2014-06-26T13:40:48Z

Pat: Pat uploaded a new version of "File:wrapper-interaction.png": Reverted to version as of 11:58, 23 October 2013

Shows how the two wrapper interact with the LDAP backend

File:wrapper-interaction.png

2014-06-26T13:37:56Z

Pat: Pat uploaded a new version of "File:wrapper-interaction.png": Adapted to the new substeps (create-, export- and commitSnapshot)

Shows how the two wrapper interact with the LDAP backend

stoney conductor: prov-backup-kvm

2014-06-26T13:25:28Z

Pat: /* commitSnapshot */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (snapshot, merge and retain) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the snaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the merge process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]

== Restore ==
* Task for the control-instance daemon
* Task for the prov-backup-kvm daemon
* Task for the vm-manager
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]

See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: prov-backup-kvm

2014-06-26T13:25:06Z

Pat: /* exportSnapshot */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (snapshot, merge and retain) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the snaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the merge process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#Retain | Retain: Basic idea]]

== Restore ==
* Task for the control-instance daemon
* Task for the prov-backup-kvm daemon
* Task for the vm-manager
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]

See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: prov-backup-kvm

2014-06-26T13:24:37Z

Pat: /* createSnapshot */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (snapshot, merge and retain) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the snaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the merge process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#Merge | Merge: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#Retain | Retain: Basic idea]]

== Restore ==
* Task for the control-instance daemon
* Task for the prov-backup-kvm daemon
* Task for the vm-manager
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]

See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]

stoney conductor: VM Backup

2014-06-26T13:24:07Z

Pat: /* Snapshot */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
* sstBackupRootDirectory: file:///var/backup/virtualization
** This directory might be a single partition which needs to have the same size as your partition for the live images (it's a "copy" of the live partition)
* sstBackupRetainDirectory: file:///var/virtualization/retain
** This directory must be on the same partition as your life images are
* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* createSnapshot: Create a disk only snapshot. A new overlay file is created, all write operations are performed to this file. The underlying disk-image is now read only.
* exportSnapshot: Copy the read only disk-image to the backup location.
* commitSnapshot: Commit the performed write operations from the overlay back to the underlying (original) disk image. Now the underlying image is read-write again and the overlay image can be deleted.
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( createSsnapshot( machine ) )
{

if ( exportSnapshot( machine ) )
{

if ( commitSnapshot( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while committing snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the snapshots for the machines are as close together as possible. Therefore the control instance should call first the createSnapshot process for all machines. After every machine has been snapshotted, the control instance can call the exportSnapshot and commitSnapshot process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the exportSnapshot and commitSnapshot process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( createSnapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# export and commit all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( exportSnapshot( successful_snapshots[i] ) )
{
if ( commitSnapshot( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while committing snapshot for machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while exporting snapshot for machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
See also [[Libvirt_external_snapshot_with_GlusterFS]]
==== createSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Part_2:_Create_the_snapshot_using_virsh]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#createSnapshot]]

==== exportSnapshot ====
# Simply copy the underlying image to the backup location
#* <source lang="bash">cp -p /<path>/<to>/<image>.qcow2 /<path>/<to>/<backup>/<location>/.</source>

For the workflow see [[stoney_conductor:_prov-backup-kvm#exportSnapshot]]

==== commitSnapshot ====
For the commands see [[Libvirt_external_snapshot_with_GlusterFS#Cleanup.2FCommit_.28Online.29]]

For the workflow see [[stoney_conductor:_prov-backup-kvm#commitSnapshot]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the Merge Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''merge''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to merge the backing file disk image back into the current disk image.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# merge (this way the Provisioning-Backup-VKM daemon knows, that it must start the merging process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: merge
</pre>

==== Step 07: Starting the Merge Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the merge command, it sets the '''sstProvisioningMode''' to '''merging''' to tell the Control instance daemon and other interested parties, that it is merging the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to merging by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: merging
</pre>

==== Step 08: Finalizing the Merging Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the merge command, it sets the '''sstProvisioningMode''' to '''merged''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the merging of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: merged
</pre>

==== Step 09: Start the Retain Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''retain''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to retain (copy and then delete) all the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# retain (this way the Provisioning-Backup-VKM daemon knows, that it must start the retaining process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retain
</pre>

==== Step 10: Starting the Retain Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the retain command, it sets the '''sstProvisioningMode''' to '''retaining''' to tell the Control instance daemon and other interested parties, that it is retaining the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningMode is set to retaining by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: retaining
</pre>

==== Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the retain command, it sets the '''sstProvisioningMode''' to '''retained''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the retaining of all the necessary files to the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retained
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
add: objectClass
objectClass: sstVirtualizationBackupObjectClass
-
add: sstbackupexcludefrombackup
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney conductor: prov-backup-kvm

2014-06-26T13:23:42Z

Pat: /* commitSnapshot */

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (snapshot, merge and retain) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the snaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#Snapshot | Snapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the merge process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#Merge | Merge: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#Retain | Retain: Basic idea]]

== Restore ==
* Task for the control-instance daemon
* Task for the prov-backup-kvm daemon
* Task for the vm-manager
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]

See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>

# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started)
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE => 0;

### Error codes constants
use constant UNDEFINED_ERROR => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK => 4;
use constant CANNOT_SAVE_MACHINE_STATE => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE => 12;
use constant CANNOT_REMOVE_FILE => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE => 15;
use constant CANNOT_RENAME_DISK_IMAGE => 16;
use constant CANNOT_CONNECT_TO_BACKEND => 17;
use constant WRONG_STATE_INFORMATION => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION => 20;
use constant CANNOT_RESTORE_MACHINE => 21;
use constant CANNOT_LOCK_MACHINE => 22;
use constant CANNOT_FIND_MACHINE => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST => 26;
use constant CANNOT_CREATE_DIRECTORY => 27;
use constant CANNOT_SAVE_XML => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY => 32;
use constant BACKEND_XML_UNCONSISTENCY => 33;
use constant CANNOT_CREATE_TARBALL => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL => 35;
use constant UNKNOWN_BACKEND_TYPE => 36;
use constant MISSING_NECESSARY_FILES => 37;
use constant CORRUPT_DISK_IMAGE_FOUND => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE => 41;
use constant CANNOT_START_MACHINE => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT => 43;
use constant CANNOT_READ_STATE_FILE => 44;
use constant CANNOT_READ_XML_FILE => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE => 47;
use constant NO_DISK_SPACE_INFORMATION => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]