View source for stoney conductor: prov-backup-kvm

= Overview =
The '''Provisioning-Backup-KVM Daemon''' is written in Perl and uses the mechanisms described under [[stoney core: OpenLDAP directory data organisation]].

= Workflow =
== Backup ==
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (snapshot, merge and retain) are shown later.

[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]

You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== createSnapshot ===
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the snaphshot process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]

=== exportSnapshot ===
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the merge process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]

=== commitSnapshot ===
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]

See also [[stoney_conductor:_Backup#Retain | Retain: Basic idea]]

== Restore ==
* <span style="background:#00FF00">Task for the control-instance daemon</span>
* <span style="background:#FFFF00">Task for the prov-backup-kvm daemon</span>
* <span style="background:#FF8000">Task for the vm-manager</span>
[[File:KVM-Backup-Workflow-Restore.png|thumb|none|500px|Figure 4: Detailed workflow for the restore process]]

You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-restore.xmi]]



See also [[stoney_conductor:_Backup#Basic_idea_2 | Restore: Basic idea]]

= Configuration =
== Global ==
<pre>
# Copyright (C) 2013 stepping stone GmbH
#                    Switzerland
#                    http://www.stepping-stone.ch
#                    support@stepping-stone.ch
#
# Authors:
#  Pat Kläy <pat.klaey@stepping-stone.ch>
#  
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#



[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm 
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>
 
# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started) 
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific 
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p
</pre>

== Backend ==
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
* A VM-Pool
* A single VM
The configuration which applies for the VM is evaluated in the following way:
# Check if the VM has a VM-specific configuration
#* If yes, this one applies
#* If not, continue
# Check if the VM-Pool has a specific configuration
#* If yes, this one applies
#* If not, continue
# The stoney-cloud-wide configuration applies

=== Mandatory Configuration-Parameters ===
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

=== Optional Configuration-Parameters ===
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID1
** 1: UUID2
** 2: UUID3
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
** 0: UUID3
** 1: UUID2
** 2: UUID1

= Exit codes =
The following list defines the return codes and their meaning for the KVM-Backup script see also [https://github.com/stepping-stone/prov-backup-kvm/blob/master/lib/Provisioning/Backup/KVM/Constants.pm KVMConstants.pm]:
<pre>
use constant SUCCESS_CODE                               => 0;

### Error codes constants
use constant UNDEFINED_ERROR                            => 1; # Always the first!
use constant MISSING_PARAMETER_IN_CONFIG_FILE           => 2;
use constant CONFIGURED_RAM_DISK_IS_NOT_VALUD           => 3;
use constant NOT_ENOUGH_SPACE_ON_RAM_DISK               => 4;
use constant CANNOT_SAVE_MACHINE_STATE                  => 5;
use constant CANNOT_WRITE_TO_BACKUP_LOCATION            => 6;
use constant CANNOT_COPY_FILE_TO_BACKUP_LOCATION        => 7;
use constant CANNOT_COPY_IMAGE_TO_BACKUP_LOCATION       => 8;
use constant CANNOT_COPY_XML_TO_BACKUP_LOCATION         => 9;
use constant CANNOT_COPY_BACKEND_FILE_TO_BACKUP_LOCATION=> 10;
use constant CANNOT_MERGE_DISK_IMAGES                   => 11;
use constant CANNOT_REMOVE_OLD_DISK_IMAGE               => 12;
use constant CANNOT_REMOVE_FILE                         => 13;
use constant CANNOT_CREATE_EMPTY_DISK_IMAGE             => 15;
use constant CANNOT_RENAME_DISK_IMAGE                   => 16;
use constant CANNOT_CONNECT_TO_BACKEND                  => 17;
use constant WRONG_STATE_INFORMATION                    => 18;
use constant CANNOT_SET_DISK_IMAGE_OWNERSHIP            => 19;
use constant CANNOT_SET_DISK_IMAGE_PERMISSION           => 20;
use constant CANNOT_RESTORE_MACHINE                     => 21;
use constant CANNOT_LOCK_MACHINE                        => 22;
use constant CANNOT_FIND_MACHINE                        => 23;
use constant CANNOT_COPY_STATE_FILE_TO_RETAIN           => 24;
use constant RETAIN_ROOT_DIRECTORY_DOES_NOT_EXIST       => 25;
use constant BACKUP_ROOT_DIRECTORY_DOES_NOT_EXIST       => 26;
use constant CANNOT_CREATE_DIRECTORY                    => 27;
use constant CANNOT_SAVE_XML                            => 28;
use constant CANNOT_SAVE_BACKEND_ENTRY                  => 29;
use constant CANNOT_SET_DIRECTORY_OWNERSHIP             => 30;
use constant CANNOT_SET_DIRECTORY_PERMISSION            => 31;
use constant CANNOT_FIND_CONFIGURATION_ENTRY            => 32;
use constant BACKEND_XML_UNCONSISTENCY                  => 33;
use constant CANNOT_CREATE_TARBALL                      => 34;
use constant UNSUPPORTED_FILE_TRANSFER_PROTOCOL         => 35;
use constant UNKNOWN_BACKEND_TYPE                       => 36;
use constant MISSING_NECESSARY_FILES                    => 37;
use constant CORRUPT_DISK_IMAGE_FOUND                   => 38;
use constant UNSUPPORTED_CONFIGURATION_PARAMETER        => 39;
use constant CANNOT_MOVE_DISK_IMAGE_TO_ORIGINAL_LOCATION=> 40;
use constant CANNOT_DEFINE_MACHINE                      => 41;
use constant CANNOT_START_MACHINE                       => 42;
use constant CANNOT_WORK_ON_UNDEFINED_OBJECT            => 43;
use constant CANNOT_READ_STATE_FILE                     => 44;
use constant CANNOT_READ_XML_FILE                       => 45;
use constant NOT_ALL_FILES_DELETED_FROM_RETAIN_LOCATION => 46;
use constant NOT_ENOUGH_DISK_SPACE                      => 47;
use constant NO_DISK_SPACE_INFORMATION                  => 48;
</pre>

= Next steps =
* Change the behaviour of the snapshot/merge process
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
*** Like that we are able to reduce the backup (merge) time a lot.
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge

= Source Code =
The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-kvm

= Links =

[[Category:stoney conductor]][[Category:Provisioning Modules]]