Difference between revisions of "stoney conductor: prov-backup-kvm"

From stoney cloud
Jump to: navigation, search
[unchecked revision][unchecked revision]
(→‎Next steps)
(→‎createSnapshot)
 
(37 intermediate revisions by 2 users not shown)
Line 3: Line 3:
  
 
= Workflow =
 
= Workflow =
−
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (snapshot, merge and retain) are shown later.
+
== Backup ==
 +
This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.
  
−
[[File:KVM-Backup-Workflow.png|thumb|none|400px|Figure 1: Simplified prov-backup-kvm workflow]]
+
[[File:KVM-Backup-Workflow.png|thumb|none|650px|Figure 1: Simplified prov-backup-kvm workflow]]
  
 
You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).
 
You can modify/update this workflow by editing [[File:KVM-Backup-simple.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).
  
−
== Snapshot ==
+
=== createSnapshot ===
−
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|500px|Figure 2: Detailed workflow for the snaphshot process]]
+
[[File:KVM-Backup-Workflow-Snapshot.png|thumb|none|650px|Figure 2: Detailed workflow for the createsSnaphshot process]]
  
 
You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]
 
You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]
  
−
== Merge ==
+
See also: [[stoney_conductor:_Backup#createSnapshot | createSnapshot: Basic idea]]
−
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the merge process]]
+
 
 +
=== exportSnapshot ===
 +
[[File:KVM-Backup-Workflow-Merge.png|thumb|none|500px|Figure 2: Detailed workflow for the exportSnapshot process]]
  
 
You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]
 
You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]
  
−
== Retain ==
+
See also: [[ stoney_conductor:_Backup#exportSnapshot | exportSnapshot: Basic idea]]
−
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the retain process]]
+
 
 +
=== commitSnapshot ===
 +
[[File:KVM-Backup-Workflow-Retain.png|thumb|none|500px|Figure 3: Detailed workflow for the commitSnapshot process]]
  
 
You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]
 
You can edit this workflow with the following file (you may need umbrello to modify it): [[File:KVM-Backup-Workflow-detailed.xmi]]
 +
 +
See also [[stoney_conductor:_Backup#commitSnaphsot | commitSnapshot: Basic idea]]
 +
 +
== Restore ==
 +
''' Restore is currently not implemented '''
 +
 +
= Configuration =
 +
== Global ==
 +
<pre>
 +
# Copyright (C) 2013 stepping stone GmbH
 +
#                    Switzerland
 +
#                    http://www.stepping-stone.ch
 +
#                    support@stepping-stone.ch
 +
#
 +
# Authors:
 +
#  Pat Kläy <pat.klaey@stepping-stone.ch>
 +
# 
 +
# Licensed under the EUPL, Version 1.1.
 +
#
 +
# You may not use this work except in compliance with the
 +
# Licence.
 +
# You may obtain a copy of the Licence at:
 +
#
 +
# http://www.osor.eu/eupl
 +
#
 +
# Unless required by applicable law or agreed to in
 +
# writing, software distributed under the Licence is
 +
# distributed on an "AS IS" basis,
 +
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
 +
# express or implied.
 +
# See the Licence for the specific language governing
 +
# permissions and limitations under the Licence.
 +
#
 +
 +
 +
 +
[Global]
 +
# If true the script logs every information to the log-file.
 +
LOG_DEBUG = 1
 +
 +
# If true the script logs additional information to the log-file.
 +
LOG_INFO = 1
 +
 +
#If true the script logs warnings to the log-file.
 +
LOG_WARNING = 1
 +
 +
#If true the script logs errors to the log-file.
 +
LOG_ERR = 1
 +
 +
# The environment indicates the hostname (fqdn) on which the prov-backup-kvm
 +
# daemon is running
 +
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>
 +
 +
# All information related to the database (backend) the daemon connects to
 +
[Database]
 +
BACKEND = LDAP
 +
SERVER = <STONEY-CLOUD-LDAP-SERVER>
 +
PORT = <STONEY-CLOUD-LDAP-PORT>
 +
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
 +
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
 +
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>
 +
 +
# A cookie file will be used to be able to restart the daemon without
 +
# processing every entry again (they appear as new if the daemon is started)
 +
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>
 +
 +
# The default cookie just contains an empty CSN, in that way, all entries
 +
# are processed
 +
DEFAULT_COOKIE = rid=001,csn=
 +
 +
# The search filter for the database. Only process entries found with this
 +
# filter
 +
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))
 +
 +
# Indicates the prov-backup-kvm configuration which applies for every
 +
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific
 +
# configuration
 +
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>
 +
 +
# Configuration concerining the provisioning module
 +
[Service]
 +
 +
# The modus should always be selfcare
 +
MODUS = selfcare
 +
 +
# Which TransportApi is used to execute the commands on the destination system
 +
# TransportApi can be "LocalCLI" or "CLISSH"
 +
TRANSPORTAPI = LocalCLI
 +
 +
# The name of the provisioning service
 +
SERVICE = Backup
 +
 +
# The name of the provisioning type
 +
TYPE = KVM
 +
 +
# The syslog tag (normally service-type)
 +
SYSLOG = Backup-KVM
 +
 +
# All information concerning the gateway (TransportApi)
 +
[Gateway]
 +
HOST = localhost
 +
USER = provisioning
 +
DSA_FILE = none
 +
 +
# Service specific configuration which is not present in the backend
 +
[Backup]
 +
 +
# Which command is used to export files
 +
EXPORT_COMMAND = cp -p
 +
</pre>
 +
 +
== Backend ==
 +
''' Currently the backend configuration is not active. If it is active it will be (maybe some minor modifications) as the following: '''
 +
 +
In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the [[#Global|global configuration]]. You are able to overwrite the stoney-cloud-wide configuration for
 +
* A VM-Pool
 +
* A single VM
 +
The configuration which applies for the VM is evaluated in the following way:
 +
# Check if the VM has a VM-specific configuration
 +
#* If yes, this one applies
 +
#* If not, continue
 +
# Check if the VM-Pool has a specific configuration
 +
#* If yes, this one applies
 +
#* If not, continue
 +
# The stoney-cloud-wide configuration applies
 +
 +
=== Mandatory Configuration-Parameters ===
 +
* '''sstBackupNumberOfIterations''': An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
 +
* '''sstBackupRootDirectory''': The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
 +
* '''sstBackupRetainDirectory''': The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
 +
* '''sstRestoreVMWithoutState''': Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
 +
* '''sstBackupRamDiskLocation''': Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
 +
* '''sstVirtualizationVirtualMachineForceStart''': Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
 +
* '''sstVirtualizationBandwidthMerge''': Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
 +
* '''sstVirtualizationDiskImageFormat''': The format for the new disk image that is created during the backup process. Default is qcow2.
 +
* '''sstVirtualizationDiskImageOwner''': The owner for the new disk image that is created during the backup process. Default is root.
 +
* '''sstVirtualizationDiskImageGroup''' : The group for the new disk image that is created during the backup process. Default is vm-storage.
 +
* '''sstVirtualizationDiskImagePermission''': The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
 +
* '''sstVirtualizationDiskImageDirectoryOwner''': The owner for the new directory where the disk image is located. Default is root.
 +
* '''sstVirtualizationDiskImageDirectoryGroup''': The group for the new directory where the disk image is located. Default is vm-storage.
 +
* '''sstVirtualizationDiskImageDirectoryPermission''': The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).
 +
 +
=== Optional Configuration-Parameters ===
 +
* '''sstBackupExcludeFromBackup''': Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
 +
* '''sstVirtualizationVirtualMachineSequenceStop''': Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
 +
** 0: UUID1
 +
** 1: UUID2
 +
** 2: UUID3
 +
* '''sstVirtualizationVirtualMachineSequenceStart''': Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
 +
** 0: UUID3
 +
** 1: UUID2
 +
** 2: UUID1
 +
 +
= Exit codes =
 +
The following list defines the return codes and their meaning for the KVM-Backup script:
 +
<pre>
 +
use constant
 +
{
 +
    SUCCESS_CODE => 0,
 +
    ERROR_CODE => 1,
 +
   
 +
    TRUE => 1,
 +
    FALSE => 0,
 +
   
 +
    # Specific error codes
 +
    TEMPLATE_NOT_READABLE => 101,
 +
    NO_MACHINE => 102,
 +
    NO_XML_DESCRIPTION => 103,
 +
    CANNOT_CREATE_SNAPSHOT => 104,
 +
    NO_STATE_INFORMATION => 105,
 +
    CANNOT_START_MACHINE => 106,
 +
    NO_BACKUP_LOCATION => 107,
 +
    UNCONSISTENT_BACKUP => 108,
 +
    CANNOT_GET_SNAPSHOT => 109,
 +
    CANNOT_DELETE_SNAPSHOT => 110,
 +
    CANNOT_UPDATE_XML => 111,
 +
    NO_SPACE_LEFT => 112,
 +
};
 +
</pre>
  
 
= Next steps =
 
= Next steps =
−
* Change the behaviour of the snapshot/merge process
+
* Implement restore
−
** No longer merge the original file into the new one but merge (commit) backing store file back into original one
+
 
−
*** Like that we are able to reduce the backup (merge) time a lot.
+
= Source Code =
−
*** Needs different behaviour for save -> copy/move -> create new image -> restore -> merge
+
The source code is located in our GitHub Repository:
 +
 
 +
https://github.com/stoney-cloud/stoney-conductor/tree/master/prov-backup-kvm
 +
 
 +
= Links =
  
−
[[Category:stoney conductor]]
+
[[Category:stoney conductor]][[Category:Provisioning Modules]]

Latest revision as of 15:39, 27 June 2014

Overview

The Provisioning-Backup-KVM Daemon is written in Perl and uses the mechanisms described under stoney core: OpenLDAP directory data organisation.

Workflow

Backup

This is the simplified workflow for the Provisioning-Backup-KVM Daemon. The Subroutines (create-, export- and commitSnapshot) are shown later.

Figure 1: Simplified prov-backup-kvm workflow

You can modify/update this workflow by editing File:KVM-Backup-simple.xmi (you may need Umbrello UML Modeller diagram programme for KDE to display the content properly).

createSnapshot

Figure 2: Detailed workflow for the createsSnaphshot process

You can edit this workflow with the following file (you may need umbrello to modify it): File:KVM-Backup-Workflow-detailed.xmi

See also: createSnapshot: Basic idea

exportSnapshot

Figure 2: Detailed workflow for the exportSnapshot process

You can edit this workflow with the following file (you may need umbrello to modify it): File:KVM-Backup-Workflow-detailed.xmi

See also: exportSnapshot: Basic idea

commitSnapshot

Figure 3: Detailed workflow for the commitSnapshot process

You can edit this workflow with the following file (you may need umbrello to modify it): File:KVM-Backup-Workflow-detailed.xmi

See also commitSnapshot: Basic idea

Restore

Restore is currently not implemented

Configuration

Global

# Copyright (C) 2013 stepping stone GmbH
#                    Switzerland
#                    http://www.stepping-stone.ch
#                    support@stepping-stone.ch
#
# Authors:
#  Pat Kläy <pat.klaey@stepping-stone.ch>
#  
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#



[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The environment indicates the hostname (fqdn) on which the prov-backup-kvm 
# daemon is running
ENVIRONMENT = <STONEY-CLOUD-NODE-NAME>
 
# All information related to the database (backend) the daemon connects to
[Database]
BACKEND = LDAP
SERVER = <STONEY-CLOUD-LDAP-SERVER>
PORT = <STONEY-CLOUD-LDAP-PORT>
ADMIN_USER = <STONEY-CLOUD-LDAP-BINDDN>
ADMIN_PASSWORD = <STONEY-CLOUD-LDAP-BIND-PASSWORD>
SERVICE_SUBTREE = <STONEY-CLOUD-LDAP-SERVICE-SUBTREE>

# A cookie file will be used to be able to restart the daemon without
# processing every entry again (they appear as new if the daemon is started) 
COOKIE_FILE = <STONEY-CLOUD-LDAP-COOKIE-FILE>

# The default cookie just contains an empty CSN, in that way, all entries
# are processed
DEFAULT_COOKIE = rid=001,csn=

# The search filter for the database. Only process entries found with this
# filter
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))

# Indicates the prov-backup-kvm configuration which applies for every
# VM-Pool and every VM if not overwritten by a VM-Pool- or VM-specific 
# configuration
STONEY_CLOUD_WIDE_CONFIGURATION = <STONEY-CLOUD-LDAP-PROV-BACKUP-KVM-DEFAULT-CONFIGURATION>

# Configuration concerining the provisioning module
[Service]

# The modus should always be selfcare
MODUS = selfcare

# Which TransportApi is used to execute the commands on the destination system
# TransportApi can be "LocalCLI" or "CLISSH"
TRANSPORTAPI = LocalCLI

# The name of the provisioning service
SERVICE = Backup

# The name of the provisioning type
TYPE = KVM

# The syslog tag (normally service-type)
SYSLOG = Backup-KVM

# All information concerning the gateway (TransportApi)
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

# Service specific configuration which is not present in the backend
[Backup]

# Which command is used to export files
EXPORT_COMMAND = cp -p

Backend

Currently the backend configuration is not active. If it is active it will be (maybe some minor modifications) as the following:

In the backend, you need to have at least one configuration which applies for the whole stoney cloud. This configuration is referenced in the global configuration. You are able to overwrite the stoney-cloud-wide configuration for

  • A VM-Pool
  • A single VM

The configuration which applies for the VM is evaluated in the following way:

  1. Check if the VM has a VM-specific configuration
    • If yes, this one applies
    • If not, continue
  2. Check if the VM-Pool has a specific configuration
    • If yes, this one applies
    • If not, continue
  3. The stoney-cloud-wide configuration applies

Mandatory Configuration-Parameters

  • sstBackupNumberOfIterations: An integer value how many backup iterations should be kept. Default is 1 (for disaster recovery).
  • sstBackupRootDirectory: The path to the backup root directory where all iterations of disk-images and state files are stored. Default is file:///var/backup/virtualization.
  • sstBackupRetainDirectory: The path to the local retain directory where the temporary snapshots (disk-image and state file) are stored. Default is file:///var/virtualization/retain.
  • sstRestoreVMWithoutState: Boolean value which indicates whether or not to restore a virtual machine without the state. Default is FALSE (most often we want to restore the state together with the virtual machine).
  • sstBackupRamDiskLocation: Path to the RAM-Disk. Default is /mnt/ramdisk. Because this attribute can be set for the whole FOSS-Cloud, for a specific VM-Pool, for a specific virtual machine or a specific virtual machine template, this attribute is independent from the VM-Nodes. There for no guarantee can be given, that this RAM-Disk exists on all the VM-Nodes. A check for its existence is mandatory!
  • sstVirtualizationVirtualMachineForceStart: Force start VM in the case of not being able to restore the VM State during the backup process. TRUE or FALSE, default is FALSE. Attention: If set to TRUE, this could lead to file system inconsistencies in the virtual machine.
  • sstVirtualizationBandwidthMerge: Bandwidth of the disk merging process (specifies the maximum I/O rate to allow in Megabyte/s). Default is 0 (unlimited). Integer Attribute, single value.
  • sstVirtualizationDiskImageFormat: The format for the new disk image that is created during the backup process. Default is qcow2.
  • sstVirtualizationDiskImageOwner: The owner for the new disk image that is created during the backup process. Default is root.
  • sstVirtualizationDiskImageGroup : The group for the new disk image that is created during the backup process. Default is vm-storage.
  • sstVirtualizationDiskImagePermission: The permission (in octal representation) for the new disk image that is created during the backup process. Default is 660 (equivalent to 0660).
  • sstVirtualizationDiskImageDirectoryOwner: The owner for the new directory where the disk image is located. Default is root.
  • sstVirtualizationDiskImageDirectoryGroup: The group for the new directory where the disk image is located. Default is vm-storage.
  • sstVirtualizationDiskImageDirectoryPermission: The permission (in octal representation) for the new directory where the disk image is located. Default is 770 (equivalent to 0770).

Optional Configuration-Parameters

  • sstBackupExcludeFromBackup: Do we want to exclude a virtual machine from the default backup plan? Default is FALSE.
  • sstVirtualizationVirtualMachineSequenceStop: Multiple dependencies for the stopping order can be defined. Example: a web VM depends on the corresponding database VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be stopped in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
    • 0: UUID1
    • 1: UUID2
    • 2: UUID3
  • sstVirtualizationVirtualMachineSequenceStart: Multiple dependencies for the starting order can be defined. Example: a database VM must be started before the corresponding web VM. IA5String, multi valued. This attribute must exist in all of the virtual machine entries, that are to be started in a certain order. Example (0,1,2, ... is the order, UUID1, UUID2, ... is the uuid of a virtual machine):
    • 0: UUID3
    • 1: UUID2
    • 2: UUID1

Exit codes

The following list defines the return codes and their meaning for the KVM-Backup script: 

use constant 
{
    SUCCESS_CODE => 0,
    ERROR_CODE => 1,
    
    TRUE => 1,
    FALSE => 0,
    
    # Specific error codes
    TEMPLATE_NOT_READABLE => 101,
    NO_MACHINE => 102,
    NO_XML_DESCRIPTION => 103,
    CANNOT_CREATE_SNAPSHOT => 104,
    NO_STATE_INFORMATION => 105,
    CANNOT_START_MACHINE => 106,
    NO_BACKUP_LOCATION => 107,
    UNCONSISTENT_BACKUP => 108,
    CANNOT_GET_SNAPSHOT => 109,
    CANNOT_DELETE_SNAPSHOT => 110,
    CANNOT_UPDATE_XML => 111,
    NO_SPACE_LEFT => 112,
};

Next steps

  • Implement restore

Source Code

The source code is located in our GitHub Repository:

https://github.com/stoney-cloud/stoney-conductor/tree/master/prov-backup-kvm

Links

Retrieved from "https://wiki.stoney-cloud.org/mediawiki/index.php?title=stoney_conductor:_prov-backup-kvm&oldid=3786"