stoney-cloud.org - User contributions [en]

stoney backup: Server set-up

2014-06-13T13:51:13Z

David: /* Cronjobs */

= Abstract =
This document describes server setup for the stoney cloud (Online) Backup service, built upon the [http://www.gentoo.org/ Gentoo] Linux distribution.

= Overview =
After working through this documentation, you will be able to set up and configure your own (Online) Backup service server.

= Software Installation =

== Requirements ==
A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].

== Keywords & USE-Flags ==
For a minimal OpenLDAP directory installation:
echo "net-nds/openldap minimal sasl" >> /etc/portage/package.use
echo "net-nds/openldap ~amd64" >> /etc/portage/package.keywords

NSS and PAM modules for lookups using LDAP:
echo "sys-auth/nss-pam-ldapd sasl" >> /etc/portage/package.use
echo "sys-auth/nss-pam-ldapd ~amd64" >> /etc/portage/package.keywords
echo "sys-fs/quota ldap" >> /etc/portage/package.use

echo "=app-admin/jailkit-2.16 ~amd64" >> /etc/portage/package.keywords

For the prov-backup-rsnapshot daemon:
echo "dev-perl/Net-SMTPS ~amd64" >> /etc/portage/package.keywords
echo "perl-core/Switch ~amd64" >> /etc/portage/package.keywords

To build puttygen only without X11:
echo "net-misc/putty ~amd64" >> /etc/portage/package.keywords
echo "net-misc/putty -gtk" >> /etc/portage/package.use

== Emerge ==
emerge -va nss-pam-ldapd \
quota \
net-misc/putty \
app-admin/jailkit \
sys-apps/haveged \
net-misc/putty \
sys-apps/sst-backup-utils \
sys-apps/sst-prov-backup-rsnapshot

To list the dependencies of ebuilds, you can use <code>equery</code>:
equery depgraph sst-backup-utils
<pre>
* Searching for sst-backup-utils ...

* dependency graph for sys-apps/sst-backup-utils-0.1.0
`-- sys-apps/sst-backup-utils-0.1.0 amd64
`-- dev-perl/PerlUtil-0.1.0 (>=dev-perl/PerlUtil-0.1.0) amd64
`-- virtual/perl-Sys-Syslog-0.320.0 (virtual/perl-Sys-Syslog) amd64
`-- dev-perl/perl-ldap-0.530.0 (dev-perl/perl-ldap) amd64
`-- dev-perl/XML-Simple-2.200.0 (dev-perl/XML-Simple) amd64
`-- dev-perl/Config-IniFiles-2.780.0 (dev-perl/Config-IniFiles) amd64
`-- dev-perl/XML-Validator-Schema-1.100.0 (dev-perl/XML-Validator-Schema) amd64
`-- dev-perl/Date-Calc-6.300.0 (dev-perl/Date-Calc) amd64
`-- dev-perl/DateManip-6.310.0 (dev-perl/DateManip) amd64
`-- dev-perl/Schedule-Cron-Events-1.930.0 (dev-perl/Schedule-Cron-Events) amd64
`-- dev-perl/DateTime-Format-Strptime-1.520.0 (dev-perl/DateTime-Format-Strptime) amd64
`-- dev-perl/XML-SAX-0.990.0 (dev-perl/XML-SAX) amd64
`-- virtual/perl-MIME-Base64-3.130.0-r2 (virtual/perl-MIME-Base64) amd64
`-- dev-perl/Authen-SASL-2.160.0 (dev-perl/Authen-SASL) amd64
`-- dev-perl/Net-SMTPS-0.30.0 (dev-perl/Net-SMTPS) ~amd64
`-- dev-perl/text-template-1.450.0 (dev-perl/text-template) amd64
`-- virtual/perl-Getopt-Long-2.380.0-r2 (virtual/perl-Getopt-Long) amd64
`-- dev-perl/Parallel-ForkManager-1.20.0 (dev-perl/Parallel-ForkManager) amd64
`-- dev-perl/Time-Stopwatch-1.0.0 (dev-perl/Time-Stopwatch) amd64
`-- app-backup/rsnapshot-1.3.1-r1 (app-backup/rsnapshot) amd64
[ sys-apps/sst-backup-utils-0.1.0 stats: packages (20), max depth (1) ]
</pre>

For more information, visit the [http://www.gentoo.org/doc/en/gentoolkit.xml Gentoolkit] page.

= Base Server Software Configuration =
== OpenSSH ==
=== OpenSSH Configuration ===
Configure the OpenSSH daemon:
<source lang="bash">
vi /etc/ssh/sshd_config
</source>

Set following options:
<source lang="bash">
PubkeyAuthentication yes
PasswordAuthentication yes
UsePAM yes
Subsystem sftp internal-sftp
</source>

Make sure, that <code>Subsystem sftp internal-sftp</code> is the last line in the configuration file.

We want to reduce the numbers of chroot environments in one folder. As the <code>ChrootDirectory</code> configuration option only allows <code>%h</code> (home directory of the user) and <code>%u</code> (username of the user), we need to create the necessary matching rules in the form of:
<source lang="bash">
Match User *000
ChrootDirectory /var/backup/000/%u
AuthorizedKeysFile /var/backup/000/%u/%h/.ssh/authorized_keys
Match
Match User *001
ChrootDirectory /var/backup/001/%u
AuthorizedKeysFile /var/backup/001/%u/%h/.ssh/authorized_keys
Match
...
Match User *999
ChrootDirectory /var/backup/999/%u
AuthorizedKeysFile /var/backup/999/%u/%h/.ssh/authorized_keys
Match
</source>

The creation of the matching rules is done by executing the following bash commands:
<source lang="bash">
FILE=/etc/ssh/sshd_config;

for x in {0..999} ; do \
printf "Match User *%03d\n" $x >> ${FILE}; \
printf " ChrootDirectory /var/backup/%03d/%%u\n" $x >> ${FILE}; \
printf " AuthorizedKeysFile /var/backup/%03d/%%u/%%h/.ssh/authorized_keys\n" $x >> ${FILE}; \
printf "Match\n" >> ${FILE}; \
done
</source>

Don't forget to restart the OpenSSH daemon:
<source lang="bash">
/etc/init.d/sshd restart
</source>

=== OpenSSH Host Keys ===
If you migrate from a existing backup server, you might want to copy the ssh host keys to the new server. If you do so clients want see a difference between the two hosts as the fingerprint remains the same. Copy the following files from the existing host to the new:
* /etc/ssh/ssh_host_dsa_key
* /etc/ssh/ssh_host_ecdsa_key
* /etc/ssh/ssh_host_key
* /etc/ssh/ssh_host_rsa_key
* /etc/ssh/ssh_host_dsa_key.pub
* /etc/ssh/ssh_host_ecdsa_key.pub
* /etc/ssh/ssh_host_key.pub
* /etc/ssh/ssh_host_rsa_key.pub

Set the correct permissions on the new host:
chmod 600 /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_ecdsa_key /etc/ssh/ssh_host_key /etc/ssh/ssh_host_rsa_key
chmod 644 /etc/ssh/*.pub

And restart the ssh daemon. ''Caution'': do not close your existing ssh session as long as you are not sure the ssh daemon has restarted properly and you can login again.
/etc/init.d/sshd restart

== OpenLDAP ==
=== /etc/hosts ===
Update the <code>/etc/hosts</code> with the LDAP server:
/etc/hosts

# VIP of the LDAP Server
31.216.40.4 ldapm.stoney-cloud.org

=== Root CA Certificate Installation ===
Install the root CA certificate into the OpenSSL default certificate storage directory:
fqdn="cloud.stoney-cloud.org" # The fully qualified domain name of the server containing the root certificate.

cd /etc/ssl/certs/
wget --no-check-certificate https://${fqdn}/ca/FOSS-Cloud_CA.cert.pem
chown root:root /etc/ssl/certs/FOSS-Cloud_CA.cert.pem
chmod 444 /etc/ssl/certs/FOSS-Cloud_CA.cert.pem

Rebuild the CA hashes
c_rehash /etc/ssl/certs/

=== /etc/openldap/ldap.conf ===
Update the <code>/etc/openldap/ldap.conf</code>LDAP configuration file/environment variables:
/etc/openldap/ldap.conf

<pre>
# Used to specify a size limit to use when performing searches. The number should be an
# non-negative integer. SIZELIMIT of zero (0) specifies unlimited search size.
SIZELIMIT 20000

# Used to specify a time limit to use when performing searches. The number should be an
# non-negative integer. TIMELIMIT of zero (0) specifies unlimited search time to be used.
TIMELIMIT 45

# Specify how aliases dereferencing is done. DEREF should be set to one of never, always, search,
# or find to specify that aliases are never dereferenced, always dereferenced, dereferenced when
# searching, or dereferenced only when locating the base object for the search. The default is to
# never dereference aliases.
DEREF never

# Specifies the URI(s) of an LDAP server(s) to which the LDAP library should connect. The URI
# scheme may be either ldapor ldaps which refer to LDAP over TCP and LDAP over SSL (TLS)
# respectively. Each server's name can be specified as a domain- style name or an IP address
# literal. Optionally, the server's name can followed by a ':' and the port number the LDAP
# server is listening on. If no port number is provided, the default port for the scheme is
# used (389 for ldap://, 636 for ldaps://). A space separated list of URIs may be provided.
URI ldaps://ldapm.stoney-cloud.org

# Used to specify the default base DN to use when performing ldap operations. The base must be
# specified as a Distinguished Name in LDAP format.
BASE dc=stoney-cloud,dc=org

# This is a local copy of the certificate of the certificate authority
# used to sign the server certificate for the LDAP server I am using
TLS_CACERT /etc/ssl/certs/FOSS-Cloud_CA.cert.pem
</pre>

Check you configuration by doing a search:
ldapsearch -v -H "ldaps://ldapm.stoney-cloud.org" \
-b "dc=stoney-cloud,dc=org" \
-D "cn=Manager,dc=stoney-cloud,dc=org" \
-s one "(objectClass=*)" \
-LLL -W

The result should look something like:
ldap_initialize( ldaps://ldapm.stoney-cloud.org:636/??base )
filter: (objectClass=*)
requesting: All userApplication attributes
dn: ou=administration,dc=stoney-cloud,dc=org
objectClass: top
objectClass: organizationalUnit
ou: administration
...

== Random Number Generator (haveged) ==
Tools like putty are dependent on random numbers to be able to create certificates.

=== haveged - Generate random numbers and feed linux random device ===
The haveged daemon doesn't need any special configuration, therefore you can start it from the command line interface:
/etc/init.d/haveged start

Check, if the start was successful:
ps auxf | grep haveged

root 18001 1.0 0.0 7420 3616 ? Ss 08:48 0:00 /usr/sbin/haveged -r 0 -w 1024 -v 1

Add the haveged daemon to the default run level:
rc-update add haveged default

== nss-pam-ldapd ==
=== nslcd.conf — configuration file for LDAP nameservice daemon ===
/etc/nslcd.conf

<pre>
# This is the configuration file for the LDAP nameservice
# switch library's nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The uri pointing to the LDAP server to use for name lookups.
# Multiple entries may be specified. The address that is used
# here should be resolvable without using LDAP (obviously).
#uri ldap://127.0.0.1/
#uri ldaps://127.0.0.1/
#uri ldapi://%2fvar%2frun%2fldapi_sock/
# Note: %2f encodes the '/' used as directory separator
uri ldaps://ldapm.tombstone.ch

# The LDAP version to use (defaults to 3
# if supported by client library)
#ldap_version 3

# The distinguished name of the search base.
base dc=stoney-cloud,dc=org

# The distinguished name to bind to the server with.
# Optional: default is to bind anonymously.
binddn cn=Manager,dc=stoney-cloud,dc=org

# The credentials to bind with.
# Optional: default is no credentials.
# Note that if you set a bindpw you should check the permissions of this file.
bindpw myverysecretpassword

# The distinguished name to perform password modifications by root by.
#rootpwmoddn cn=admin,dc=example,dc=com

# The default search scope.
#scope sub
#scope one
#scope base

# Customize certain database lookups.
#base group ou=Groups,dc=example,dc=com
base group ou=groups,ou=backup,ou=services,dc=stoney-cloud,dc=org
base passwd ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
base shadow ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
#scope group onelevel
#scope hosts sub

#filter group (&(objectClass=posixGroup)(sstIsActive=TRUE))
filter passwd (&(objectClass=posixAccount)(sstIsActive=TRUE))
filter shadow (&(objectClass=shadowAccount)(sstIsActive=TRUE))

# Bind/connect timelimit.
#bind_timelimit 30

# Search timelimit.
#timelimit 30

# Idle timelimit. nslcd will close connections if the
# server has not been contacted for the number of seconds.
#idle_timelimit 3600

# Use StartTLS without verifying the server certificate.
#ssl start_tls
tls_reqcert never

# CA certificates for server certificate verification
#tls_cacertdir /etc/ssl/certs
#tls_cacertfile /etc/ssl/ca.cert

# Seed the PRNG if /dev/urandom is not provided
#tls_randfile /var/run/egd-pool

# SSL cipher suite
# See man ciphers for syntax
#tls_ciphers TLSv1

# Client certificate and key
# Use these, if your server requires client authentication.
#tls_cert
#tls_key

# Mappings for Services for UNIX 3.5
#filter passwd (objectClass=User)
#map passwd uid msSFU30Name
#map passwd userPassword msSFU30Password
#map passwd homeDirectory msSFU30HomeDirectory
#map passwd homeDirectory msSFUHomeDirectory
#filter shadow (objectClass=User)
#map shadow uid msSFU30Name
#map shadow userPassword msSFU30Password
#filter group (objectClass=Group)
#map group member msSFU30PosixMember

# Mappings for Services for UNIX 2.0
#filter passwd (objectClass=User)
#map passwd uid msSFUName
#map passwd userPassword msSFUPassword
#map passwd homeDirectory msSFUHomeDirectory
#map passwd gecos msSFUName
#filter shadow (objectClass=User)
#map shadow uid msSFUName
#map shadow userPassword msSFUPassword
#map shadow shadowLastChange pwdLastSet
#filter group (objectClass=Group)
#map group member posixMember

# Mappings for Active Directory
#pagesize 1000
#referrals off
#idle_timelimit 800
#filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map passwd uid sAMAccountName
#map passwd homeDirectory unixHomeDirectory
#map passwd gecos displayName
#filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map shadow uid sAMAccountName
#map shadow shadowLastChange pwdLastSet
#filter group (objectClass=group)

# Alternative mappings for Active Directory
# (replace the SIDs in the objectSid mappings with the value for your domain)
#pagesize 1000
#referrals off
#idle_timelimit 800
#filter passwd (&(objectClass=user)(objectClass=person)(!(objectClass=computer)))
#map passwd uid cn
#map passwd uidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820
#map passwd gidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820
#map passwd homeDirectory "/home/$cn"
#map passwd gecos displayName
#map passwd loginShell "/bin/bash"
#filter group (|(objectClass=group)(objectClass=person))
#map group gidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820

# Mappings for AIX SecureWay
#filter passwd (objectClass=aixAccount)
#map passwd uid userName
#map passwd userPassword passwordChar
#map passwd uidNumber uid
#map passwd gidNumber gid
#filter group (objectClass=aixAccessGroup)
#map group cn groupName
#map group gidNumber gid
</pre>

=== nsswitch.conf - Name Service Switch configuration file ===
/etc/nsswitch.conf

<pre>
passwd: files ldap
shadow: files ldap
group: files ldap

# passwd: db files nis
# shadow: db files nis
# group: db files nis

hosts: files dns
networks: files dns

services: db files
protocols: db files
rpc: db files
ethers: db files
netmasks: files
netgroup: files
bootparams: files

automount: files
aliases: files
</pre>

=== system-auth ===
vi /etc/pam.d/system-auth

<pre>
auth required pam_env.so
auth sufficient pam_unix.so try_first_pass likeauth nullok
auth sufficient pam_ldap.so minimum_uid=1000 use_first_pass
auth required pam_deny.so

account required pam_unix.so
account sufficient pam_ldap.so minimum_uid=1000 use_first_pass

password required pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
password required pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password sufficient pam_ldap.so minimum_uid=1000 use_first_pass
password required pam_deny.so

session required pam_limits.so
session required pam_env.so
session required pam_unix.so
session sufficient pam_ldap.so minimum_uid=1000 use_first_pass
</pre>

=== Test the Setup ===
nslcd -d

=== Update the Default Run Levels ===
rc-update add nslcd default
rc-update add nscd default

=== Start the necessary Daemons ===
/etc/init.d/nslcd start
/etc/init.d/nscd start

== Quota ==
=== 32-bit Project Identifier Support ===
We need to enable 32-bit project identifier support (PROJID32BIT feature) for our naming scheme (uid numbers larger than 65'536), which is already the default on the stepping stone virtual machines:
mkfs.xfs '''-i projid32bit=1''' /dev/vg-local-01/var

=== Update /etc/fstab and Mount ===
Make sure, that you have user quota (uqota) and project quota (pquota) set as options on the chosen mount point in /etc/fstab. For example:
LABEL=LV-VAR /var xfs noatime,discard,inode64,uquota,pquota 0 2

reboot

Check, if everything went ok:
df -h | grep var

/dev/mapper/vg--local--01-var 1023G 220G 804G 22% /var

=== Verify ===
Some important options for xfs_quota:
* -x: Enable expert mode.
* -c: Pass arguments on the command line. Multiple arguments may be given.

Remount the file system /var and check, if /var has the desired values:
xfs_quota -x -c state /var

As you can see (items marked bold), we have achieved our goal:
User quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: '''ON'''
Enforcement: '''ON'''
Inode: #131 (1 blocks, 1 extents)
Group quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: OFF
Enforcement: OFF
Inode: #132 (1 blocks, 1 extents)
Project quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: '''ON'''
Enforcement: '''ON'''
Inode: #132 (1 blocks, 1 extents)
Blocks grace time: [7 days 00:00:30]
Inodes grace time: [7 days 00:00:30]
Realtime Blocks grace time: [7 days 00:00:30]

=== User Quotas ===
==== Adding a User Quota ====
Set a quota of 1 Gigabyte for the user 4000187 (the values are in kilobytes, so 1048576 kilobyte are 1024 megabytes which corresponds to 1 gigabyte):
xfs_quota -x -c 'limit bhard=1048576k 4000187' /var

Or in bytes:
xfs_quota -x -c 'limit bhard=1073741824 4000187' /var

Read the quota information for the user 4000187:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If the user has data in the project, that belongs to him, the result will change:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

==== Modifiying a User Quota ====
To modify a users quota, you just set a new quota (limit):
xfs_quota -x -c 'limit bhard=1048576k 4000187' /var

Read the quota information for the user 4000187:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If the user has data in the project, that belongs to him, the result will change:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

==== Removing a User Quota ====
Removing a quota for a user:
xfs_quota -x -c 'limit bhard=0 4000187' /var

The following command should give you an empty result:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

=== Project (Directory) Quotas ===
==== Adding a Project (Directory) Quota ====
The XFS file system additionally allows you to set quotas on individual directory hierarchies in the file system that are known as managed trees. Each managed tree is uniquely identified by a project ID and an optional project name. We'll use the following values in the examples:
* project_ID: The uid of the online backup account (4000187).
* project_name: The uid of the online backup account (4000187). This could be a human readable name.
* mountpoint: The mountpoint of the xfs-filesystem (/var). See the <code>/etc/fstab</code> entry from above.
* directory: The directory of the project (187/4000187), starting from the mountpoint of the xfs-filesystem (/var).

Define a unique project ID for the directory hierarchy in the <code>/etc/projects</code> file (project_ID:mountpoint/directory):
echo "4000187:/var/backup/187/4000187/home/4000187" >> /etc/projects

Create an entry in the <code>/etc/projid</code> file that maps a project name to the project ID (project_name:project_ID):
echo "4000187:4000187" >> /etc/projid

Set Project:
xfs_quota -x -c 'project -s -p /var/backup/187/4000187/home/4000187 4000187' /var

Set Quota (limit) on Project:
xfs_quota -x -c 'limit -p bhard=1048576k 4000187' /var

Check your Quota (limit)
xfs_quota -x -c 'quota -p 4000187' /var

Check the Quota:
* <code>-v</code>: increase verbosity in reporting (also dumps zero values).
* <code>-N</code>: suppress the initial header.
* <code>-p</code>: display project quota information.
* <code>-h</code>: human readable format.
xfs_quota -x -c 'quota -v -N -p 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If you copied data into the project, the output will look something like:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

To give you an overall view of the whole system:
xfs_quota -x -c report /var

<pre>
User quota on /var (/dev/mapper/vg--local--01-var)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 1024000 0 0 00 [--------]
4000187 0 0 1048576 00 [--------]

Project quota on /var (/dev/mapper/vg--local--01-var)
Blocks
Project ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
4000187 512000 0 1048576 00 [--------]
</pre>

==== Modifying a Project (Directory) Quota ====
To modify a project (directory) quota, you just set an new quota (limit) on the chosen project:
xfs_quota -x -c 'limit -p bhard=1048576k 4000187' /var

Check your quota (limit)
xfs_quota -x -c 'quota -p 4000187' /var

==== Removing a Project (Directory) Quota ====
Removing a quota from a project:
xfs_quota -x -c 'limit -p bhard=0 4000187' /var

Chreck the results:
xfs_quota -x -c report /var

<pre>
User quota on /var (/dev/mapper/vg--local--01-var)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 512000 0 0 00 [--------]
4000187 0 0 1024 00 [--------]
</pre>

As you can see, the line with the Project ID 4000187 has disappeared:
4000187 512000 0 1048576 00 [--------]

Don't forget to remove the project from <code>/etc/projects</code> and <code>/etc/projid</code>:
sed -i -e '/4000187/d' /etc/projects
sed -i -e '/4000187/d' /etc/projid

=== Some important notes concerning XFS ===
# The '''quotacheck''' command has no effect on XFS filesystems. The first time quota accounting is turned on (at mount time), XFS does an automatic quotacheck internally; afterwards, the quota system will always be completely consistent until quotas are manually turned off.
# There is '''no need for quota file(s)''' in the root of the XFS filesystem.

== prov-backup-rsnapshot ==
Install the prov-backup-rsnasphot daemon script using the package manager:
<pre>
emerge -va sys-apps/sst-prov-backup-rsnapshot
</pre>

=== Configuration ===
If it is the first provisioning module running on this server (very likely) you first have to configure the provisioning daemon (you can skip this step if you have already another provisioning module running on this server)

==== Provisioning global configuration ====
The global configuration for the provisioning daemon applies to all provisioning modules running on the server. This configuration therefore contains information about the provisioning daemon itself and no information at all about the specific modules.
/etc/Provisioning/Global.conf
<pre>
# Copyright (C) 2012 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 0

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The number of seconds to wait before retry contacting the backend server during startup.
SLEEP = 10

# Number of backend server connection retries during startup.
ATTEMPTS = 3

[Operation Mode]
# The number of seconds to wait before retry contacting the backend server in case of a service interruptions.
SLEEP = 30

# Number of backend server connection retries in case of a service interruptions.
ATTEMPTS = 3

[Mail]
# Error messages are sent to the mail configured below.
SENDTO = <YOUR-MAIL-ADDRESS>
HOST = mail.stepping-stone.ch
PORT = 587
USERNAME = <YOUR-NOTIFICATION-EMAIL-ADDRESS>
PASSWORD = <PASSWORD>
FROMNAME = Provisioning daemon
CA_DIR = /etc/ssl/certs
SSL = starttls
AUTH_METHOD = LOGIN

# Additionally, you can be informed about creation, modification and deletion of services.
WANTINFOMAIL = 1
</pre>

==== Provisioning daemon prov-backup-rsnapshot module ====
The module specific configuration is located in /etc/Provisioning/<Service>/<Type>.conf. In the case of the prov-backup-rsnapshot module this is <code>/etc/Provisioning/Backup/Rsnapshot.conf</code>. (Note: Comments starting with /* are not in the configuration file, they are only in the wiki to add some additional information)

<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

/* If you want, you can override the log information from the global configuration file this might be useful for debugging */
[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

/* Specify the hosts fully qualified domain name. This name will be used to perform some checks and also appear in the information and error mails */
ENVIRONMENT = <FQDN>

[Database]
BACKEND = LDAP
SERVER = ldaps://ldapm.tombstone.org
PORT = 636
ADMIN_USER = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <PASSWORD>
SERVICE_SUBTREE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
COOKIE_FILE = /etc/Provisioning/Backup/rsnapshot.cookie
DEFAULT_COOKIE = rid=001,csn=
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(sstProvisioningState=0))

/* Specifies the service itself. As it is the prov-backup-rsnapshot module, the SERVICE is "Backup" and the TYPE is "Rsnapshot".
* The MODUS is as usual selfcare and the TRANSPORTAPI is LocalCLI. This is because the daemon is running on the same host as the
* backup accounts are provisioned and the commands can be executed on this host using the cli.
* For more information about MODUS and TRANSPORTAPI see https://int.stepping-stone.ch/wiki/provisioning.pl#Service_Konfiguration
*/
[Service]
MODUS = selfcare
TRANSPORTAPI = LocalCLI
SERVICE = Backup
TYPE = Rsnapshot

SYSLOG = prov-backup-rsnapshot

/* For the TRANSPORTAPI LocalCLI there is no gateway required because there is no connection to establish. So set HOST, USER and
* DSA_FILE to whatever you want. Don't leave it blank, otherwise the provisioning daemon would log some error messages saying
* these attributes are empty
*/
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

/* Information about the backup itself (how to setup everything). Note that the %uid% int the RSNAPSHOT_CONFIG_FILE parameter will
* be replaced by the accounts UID. The script CREATE_CHROOT_CMD was installed with the prov-backup-rsnapshot module, so do not
* change this parameter. The quota parameters (SET_QUOTA_CMD, MOUNTPOINT, QUOTA_FILE, PROJECTS_FILE and PROJID_FILE) represent
* the quota setup as described on http://wiki.stoney-cloud.org/index.php/stoney_backup:_Server_set-up#Quota. If you followed this
* manual, you can copy-paste them into your configuration file, otherwise adapt them according to your quota setup.
*/
[Backup]
RSNAPSHOT_CONFIG_FILE = /etc/rsnapshot/rsnapshot.conf.%uid%
SET_QUOTA_CMD = /usr/sbin/xfs_quota
CREATE_CHROOT_CMD = /usr/libexec/createBackupDirectory.sh
MOUNTPOINT = /var
QUOTA_FILE = /etc/backupSize
PROJECTS_FILE = /etc/projects
PROJID_FILE = /etc/projid
</pre>

== backup utils ==
Install the backup utils (multiple scripts which help you to manage and monitor your backup server and backup accounts) using the package manager. For more information about the scripts please see the [[stoney_backup:_Server_set-up#stoney_backup_Service_Software | stoney backup Service Software]] section.
<pre>
emerge -va sys-apps/sst-backup-utils
</pre>
=== Configuration ===
Please refer to the configuration sections for the different scripts in [[stoney_backup:_Server_set-up#stoney_backup_Service_Software | stoney backup Service Software]].

= stoney backup Service Software =
The stoney backup Service comes along with multiple scripts which help you to manage and monitor your backup server and accounts:

We use rsnapshot - remote filesystem snapshot utility for the actual snapshots and a handful of wrapper scripts, that do things like:
* Read the users and their settings from the LDAP directory.
* Execute rsnapshot according to the users settings.
* Write the backup quotas backup (incoming), iterations (.snapshots) and free space to the users local backupSize file and update the LDAP directory.
* Inform the reseller, customer or user (depending on the settings in the LDAP directory) via mail, if the quota limit has been reached.
* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.

== writeAccountSize.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeAccountSize.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* Calculates the used disk space (backup and iterations) for a given account and writes the corresponding values to:
*# The LDAP backend (used by the selfcare webinterface to display quota information):
*#* Backup space used (sstBackupSize): The disk space the account uses for the backup itself (disk space used under the <code>incoming</code> folder of the users chroot-home directory)
*#* Snapshot space used (sstIncrementSize): The disk space the account uses for the iterations (disk space under the <code>.snapshot</code> folder of the users chroot-home directory)
*# The file <code>etc/backupSize</code> of the accounts chroot (used by the Sepiola Online Backup client):
* Checks if the user and/or reseller must be notified that there is no more disk space left for the given account
** Checks if the notification flag was passed, if not no notification will be triggered
** Calculates the used disk space (backup and iterations) in percentage
** Reads the notification threshold value from the LDAP backed
** If the disk space used (in percentage) is bigger than the value retrieved from the LDAP backend start the [[stoney_cloud:_Notification_Architecture | notification]] process with
*** Product: Given account UID
*** Service: Backup
*** Problem: Quota
* Pod documentation:
<pre>
NAME
writeAccountSize.pl

DESCRIPTION
This Script gets quota information from filesystem, size of incoming and
snapshots directories, write the data to a file and the LDAP backend and
sends an e-mail message for each account that is over quota to users
e-mail address (from ldap directory) if notification flag is passed.

The configuration file for this script is stored in the backup-utils
configuration directory (/etc/backup-utils/) and is called
writeAccountSize.conf.

The script needs access to the quota program to get quota information.
The script needs ldap access to get users e-mail address and quota
information. The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

-C configfile
The configuration file.

-U uid
The user id.

-n notification
Start notification process if quota threshold is reached

-d debug
Turns the debug mode on.

-h help
This online help.

USAGE
writeAccountSize.pl -U uid [-C configuration file ] []-d debug] [-h
help] [-n]

CREATED
2009-04-16 michael.rhyner@stepping-stone.ch created

VERSION
2009-04-16 michael.rhyner@stepping-stone.ch created
2009-04-30 michael.rhyner@stepping-stone.ch changed position based quota
output parsing with correctly parsed elements
2009-06-15 michael.rhyner@stepping-stone.ch added over quota check and
sending e-mail
2009-06-16 michael.rhyner@stepping-stone.ch renamed script and make it
more general usable (e.g. for online backup, online storage, ...)
2009-06-17 michael.rhyner@stepping-stone.ch changed mail message to read
from a text file instead from configuration parameter
2009-06-18 michael.rhyner@stepping-stone.ch corrected wrong regex to
weed out the asterisk (*) in getQuotaSize
2009-06-19 michael.rhyner@stepping-stone.ch corrected wrong evaluation
success from subroutines and avoid message output when not in debug mode
2009-06-22 michael.rhyner@stepping-stone.ch getQuotaSize: return
immediately if no quota was set
2009-06-24 michael.rhyner@stepping-stone.ch alert when used certain
percentge of allowed space instead of more than allowed space
2009-06-26 michael.rhyner@stepping-stone.ch values are presented in
Gigabytes within notification message
2009-07-23 michael.rhyner@stepping-stone.ch corrected wrong syslog
severities for errors
2009-07-24 michael.rhyner@stepping-stone.ch made e-mail address
available within message body
2013-08-19 pat.klaey@stepping-stone.ch write quota values also to the
LDAP

INCORPORATED CODE
Incorporate code with use:

warnings;
strict;
Config::IniFiles;
Getopt::Std;
Sys::Syslog;
File::Basename;
Text::Template;
POSIX;
Notification;
PerlUtil::Logging;
PerlUtil::LDAPUtil;
</pre>

=== Configuration ===
/etc/backup-utils/writeAccountSize.conf
<pre>
[Global]
INCOMING_DIRECTORY = /incoming
ACCOUNT_SIZE_FILE = /etc/backupSize
SNAPSHOTS = 1

[Syslog]
SYSLOG = rsnapshot

[Directory]
LDAP_SERVER = ldaps://ldapm.tombstone.ch
LDAP_PORT = 636
LDAP_BIND_DN = cn=Manager,dc=stoney-cloud,dc=org
LDAP_BIND_PW = <password>
LDAP_BASE_DN = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
LDAP_PERSON_BASE = ou=people,dc=stoney-cloud,dc=org
LDAP_RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=org
LDAP_EMAIL_ATTRIBUTE = mail

[Notification]
EMAIL_SENDER = stepping stone GmbH Supprt <support@stepping-stone.ch>
EMAIL_ALERT_THRESHOLD = 85

Salutation_Default_de-CH = Liebe Kundin / Lieber Kunde
Salutation_m_de-CH = Sehr geehrter Herr
Salutation_f_de-CH = Sehr geehrte Frau
Salutation_Default_en-GB = Dear customer
Salutation_m_en-GB = Dear Mr.
Salutation_f_en-GB = Dear Mrs.
</pre>

=== Tests ===
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d
<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 0 kilobytes
Incoming Size: 0 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 0
Debug sub getSnapshotsSize: $localSnapshotsSize: 0
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 0 0 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 0

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0

Alert Threshold: 85 %
Calculated value: 0
</pre>

Now write some data (200 megaytes in this example) into the users incoming directory and then execute the script again:
dd if=/dev/zero of=/var/backup/080/4000080/home/4000080/incoming/test.zeros bs=1024k count=200
chown 4000080:4000080 /var/backup/080/4000080/home/4000080/incoming/test.zeros
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d

<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 204800 kilobytes
Incoming Size: 204800 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 204800
Debug sub getSnapshotsSize: $localSnapshotsSize: 0
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 200 0 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0

Alert Threshold: 85 %
Calculated value: 19.53125
</pre>
Everything seems to be working fine!

== snapshot.pl ==
This script is installed to <code>/usr/libexec/backup-utils/snapshot.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* Read interval parameter value passed
** The interval parameter value can be daily, weekly, monthly (or yearly)
* Find all backup active accounts for which the rsnapshot command must be executed (depending on the given interval)
** Filter to find these accounts: <code>(&(sstBackupInterval<INTERVAL>=*)(sstIsActive=TRUE))</code> for example for the daily rsnapshot the filter is <code>(&(sstBackupIntervalDaily=*)(sstIsActive=TRUE))</code>
*** In other words this means: Get me all acounts that have for <code>sstBackupInterval<INTERVAL></code> a value defined AND <code>sstIsActive</code> is set to "TRUE"
* According to the interval given and the account UID calculate the rsnapshot command for all these accounts
** For example
*** Account UID: 4000000
*** Interval: daily
*** Resulting rsnapshot command: /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.4000000 daily (if you use the [[#snapshot.pl_Configuration | configuration]] below)
* Execute all these commands
** Use controlled parallel execution, you can specify how many commands can be executed in parallel (see [[#snapshot.pl_Configuration | configuration]] below)
* Pod documentation:
<pre>
NAME
snapshot.pl

DESCRIPTION
This script gets all active online backup accounts from the LDAP backend
for which the rsnapshot process for the given interval must be executed.
According to these accounts and the given interval, the commands to be
executed are generated and finally executed. The commands can be
executed in parallel, however there is a limit defined in the
configuration file which limits the amount of parallel running
processes.

USAGE
./snapshot.pl --interval interval [--debug] [--help]

OPTIONS
--interval/-i interval
Specifies the rsnapshot interval (can be hourly, daily, weekly,
monthly or yearly)

--debug/-d
Turns on debug mode

--help/-h
Shows this help

CREATED
2012-03-19 pat.klaey@stepping-stone.ch created

VERSION
2012-03-19 pat.klaey@stepping-stone.ch created

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
Sys::Syslog;
PerlUtil::Logging;
PerlUtil::LDAPUtil;
File::Basename;
Parallel::ForkManager;
Time::Stopwatch;
</pre>

=== Configuration ===
There are two things to do in this step:
* Configure the rsnapshot root directory
* Configure the snapshot.pl script itself

==== rsnaphot configuration directory ====
The users individual rsnapshot configurations are stored under <code>/etc/rsnapshot</code>. Please make sure, that the directory exists:
ls -al /etc | grep rsnapshot

drwx------ 2 root root 64 30. Aug 20:20 rsnapshot

If not, create it:
mkdir /etc/rsnapshot
chmod 700 /etc/rsnapshot

==== snapshot.pl Configuration ====
The snapshot.pl script is responsible for the execution of rsnapshot according to the users settings.
/etc/backup-utils/snapshot.conf
<pre>
[General]
MaxParallelProcesses = 5
Rsnapshot_command = /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.%uid% %interval%

[LDAP]
Host = ldaps://ldapm.tombstone.ch
Port = 636
User = cn=Manager,dc=stoney-cloud,dc=org
Password = <Password>
CA_Path = /etc/ssl/certs
Accounts_Base = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
</pre>

Legend: At runtime the following placeholders are replaced as follows
* '''%uid%''': The backup account and login uid as a numeric number. For example: 4000205.
* '''%interval%''': The backup level to be executed. Possible values are hourly, daily, weekly, monthly and yearly.

=== Tests ===
Before adding the necessary cronjob entries, we need to make sure, that we've configured the snapshot.pl script correctly:
/usr/libexec/backup-utils/snapshot.pl --interval daily -d

If everything worked as planned, you should receive feedback looking roughly like:
<pre>
INFO: Starting rsnapshot for interval daily with maximum 5 parallel processes

INFO: Executing snapshot for 4000080

INFO: Executing snapshot for 4000079

INFO: Snapshot process for 4000079 finished in 0.18 seconds with status 0

INFO: Snapshot process for 4000080 finished in 0.19 seconds with status 0

INFO: rsnapshot for all backups done. Took 0.24 seconds
</pre>

Just to make sure, that everything did work out fine, execute <code>writeAccountSize.pl</code> again:
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d
<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 409600 kilobytes
Incoming Size: 204800 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 409600
Debug sub getSnapshotsSize: $localSnapshotsSize: 204800
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 200 200 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 209715200

Alert Threshold: 85 %
Calculated value: 39.0625
</pre>

As you can see, the total used space has risen to 39.0625.

=== Cronjobs ===
After making sure, that everything worked as planned, you can update your crontab entry:
crontab -e
<pre>
...
# Rsnapshot for all users
30 22 * * * /usr/libexec/backup-utils/snapshot.pl --interval daily
15 22 * * sun /usr/libexec/backup-utils/snapshot.pl --interval weekly
00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly
...
</pre>

* TBD: Maybe this is not optimal if there is a lot of data to rotate. In this case, it might be that for example weekly and daily snapshot both run at the same time what might lead to strange results.
* Workarounds:
** Simple/short version: Instead one could create another wrapper script which is called everyday and does some simple logic:
** Is today the first day of a month?
*** Yes: Is today sunday?
**** Yes: Execute monthly then weekly then daily rsnapshots using the snapshot.pl script (but wait for each interval to finish before starting the next)
**** No: Execute monthly then daily rsnapshots using the snapshot.pl script (but wait for monthly interval to finish before starting the daily)
*** No: Is today sunday?
**** Yes: Execute weekly then daily rsnapshots using the snapshot.pl script (but wait for weekly interval to finish before starting the daily)
**** No: Execute daily rsnapshots using the snapshot.pl script
** Complex/long version: Adapt the snapshot.pl script and call it every day without interval parameter. The script does the logic described above.
This avoids the above mentioned collision

(Complex) / short version: rule via cronjobs:
<pre>
# Rsnapshot for all users
30 22 * * 1-6 [ $( date +\%d ) -gt 1 ] && /usr/libexec/backup-utils/snapshot.pl --interval daily
15 22 * * 7 [ $( date +\%d ) -gt 1 ] && /usr/libexec/backup-utils/snapshot.pl --interval weekly && /usr/libexec/backup-utils/snapshot.pl --interval daily
00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly && /usr/libexec/backup-utils/snapshot.pl --interval weekly && /usr/libexec/backup-utils/snapshot.pl --interval daily
...
</pre>

== scheduleWarning.pl ==
This script is installed to <code>/usr/libexec/backup-utils/scheduleWarning.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* The basic task of this script is simple: For the given account
** Check if the planned backups were started
*** If not, start the [[stoney_cloud:_Notification_Architecture | notification]] process with
**** Product: Given account UID
**** Service: Backup
**** Problem: Schedule
** Check if the planned backups finished successfully
*** If not, start the [[stoney_cloud:_Notification_Architecture | notification]] process with
**** Product: Given account UID
**** Service: Backup
**** Problem: Unsuccessful

As the backup clients distributed by stepping stone GmbH upload metadata XML before (scheduling information and start time) and after (end time and backup status) the actual backup, the scheduleWarning.pl script is able to verify whether or not a planned backup has been executed and whether or not the backup was successful.

* Pod documentation
<pre>
NAME
scheduleWarning.pl

DESCRIPTION
This script tests whether a planed backup was successful or not. There
are two different typs of failure.

1. The Backup did not start
The first type of error is that a backup is scheduled for YYYY-MM-DD
at HH:MM, but the backup don't start at this specified time. A
possible reason could be that the computer was shut down.

2.The backup was not successful
The second type of error is that a backup started as scheduled but
did not finish successfully. There are different reasons for this
error.

If a backup wasn't successful the script checks which type of error
occured. It reads the XML files which are stored on the server and
compares the given information. If the error is detected, the script
stats the norification process with the information of the error.

If a user has more than one computer backed-up, the script tests one
computer after the other. The mail(s) sent by the programm also
contain(s) the information which computer is affected.

OPTION
-U uid
The -U option is required to run the script, it indicates for which
uid the script is executed.

USAGE
scheduleWarning.pl [-U user]

CREATED
2010-04-14 created Pat Kläy <pat.klaey@stepping-stone.ch>

VERSION
2010-04-14 v0.01, created pkl
2010-08-24 v0.02, modified pkl
New using Net::SMTP::TLS to send mails

2013-09-13 v0.03, modified pat.klaey@stepping-stone.ch
Changes to use the script with the new backup infrastructure (read
more information from LDAP, use Notification lib to send mails)

USES
strict;
warnings;
XML::Simple;
Config::IniFiles;
XML::Validator::Schema;
Date::Calc qw(:all);
Date::Manip;
Schedule::Cron::Events;
DateTime::Format::Strptime;
Sys::Syslog;
XML::SAX::ParserFactory;
Getopt::Std;
MIME::Base64;
Authen::SASL;
Net::LDAPS;
Net::SMTP::TLS;
Cwd 'abs_path';
PerlUtil::Logging;
PerlUtil::LDAPUtil;
</pre>

=== Configuration ===
/etc/backup-utils/scheduleWarning.conf
<pre>
[Backup]
CHROOT_DIRECTORY = /var/backup/%lastthree%/%user%

[XML]
SCHEDULE_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/scheduler.xml
SCHEDULE_XSD = /etc/backup-utils/schema/scheduler_schema.xsd
BACKUP_ENDED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupEnded.xml
BACKUP_ENDED_XSD = /etc/backup-utils/schema/backupended_schema.xsd
BACKUP_STARTED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupStarted.xml
BACKUP_STARTED_XSD = /etc/backup-utils/schema/backupstarted_schema.xsd

[TEMPLATE]
Salutation_Default_de-CH = Liebe Kundin / Lieber Kunde
Salutation_m_de-CH = Sehr geehrter Herr
Salutation_f_de-CH = Sehr geehrte Frau
Salutation_Default_en-GB = Dear customer
Salutation_m_en-GB = Dear Mr.
Salutation_f_en-GB = Dear Mrs.

[LDAP]

SERVER = ldaps://ldapm.tombstone.ch
PORT = 636
DEBUG = 1

ADMIN_DN = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <Password>

BACKUP_BASE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
PEOPLE_BASE = ou=people,dc=stoney-cloud,dc=org
RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=org
SCOPE = sub

</pre>

== checkBackups.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. It is a wrapper around the above mentioned <code>writeAccountSize.pl</code> and <code>scheduleWarning.pl</code> scripts (if you configure it accordingly).
* As mentioned the script is a wrapper for calling the quota and scheduler information scripts:
** It gets all active backup accounts from the LDAP directory
** According to the CLI parameters passed it calles for all these accounts:
*** The quota checking script (this script can be defined in the configuration file, see [[stoney_backup:_Server_set-up#Configuration_6 | configuration]] section below)
*** The scheduler checking script (this script can be defined in the configuration file, see [[stoney_backup:_Server_set-up#Configuration_6 | configuration]] section below)
* Pod documentation:
<pre>
NAME
checkBackups.pl

DESCRIPTION
This script processes all active backup accounts and checks (according
to the command line options) the quota and/or the scheduled backups for
the given accounts.

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

OPTIONS
--schedule/-s
Checks schedule information by calling the defined script

--quota/-s
Checks the quota values by calling the defined script

--notify/-n
Passes the notify flag to the called scripts

USAGE
./checkBackups.pl [--quota] [--schedule] [--notify]

./checkBackups.pl --quota
Checks and writes the quota for all active backup accounts. The user
and/or reseller will NOT be informed if any quota threshold is
reached

./checkBackups.pl --quota --notify
Checks and writes the quota for all active backup accounts. The user
and/or reseller will be informed if any quota threshold is reached.

./checkBackups.pl --schedule
Checkes all active backup account if the scheduled backup was
executed and successful. The --schedule option will allways trigger
a notification mail to the user and/or reseller.

./checkBackups.pl --schedule --quota
Checks both, quota and schedule, for all active backup accounts.
Only schedule will trigger notification mails.

./checkBackups.pl --schedule --quota --notify
Checks both, quota and schedule, for all active backup accounts.
Both, quota and schedule, will trigger notification mails.

CREATED
2013-09-17 pat.klaey@stepping-stone.ch created

VERSION
2013-09-17 pat.klaey@stepping-stone.ch created
2013-11-19 pat.klaey@stepping-stone.ch Added options to be able to check
quota, schedule or both in one run

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
Sys::Syslog;
Cwd 'abs_path';
File::Basename;
PerlUtil::LDAPUtil;
PerlUtil::Logging;
</pre>

=== Configuration ===
In the configuration you have to define two simple things:
# Which script to call for quota checks and scheduler checks
# How to access the LDAP backend
/etc/backup-utils/checkBackups.conf
<pre>
[Scripts]
CheckQuotaScript = /usr/libexec/backup-utils/writeAccountSize.pl
CheckScheduleScript = /usr/libexec/backup-utils/scheduleWarning.pl

[LDAP]
Server = ldaps://ldapm.tombstone.ch
Port = 636
Username = cn=Manager,dc=stoney-cloud,dc=org
Password = <PASSWORD>
AccountBase = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
</pre>

==== Cronjobs ====
As you can pass the notification flag to the scripts, you can call the quota script say all 10 minutes. This makes sense if followed the configuration section, as the writeAccountSize.pl script writes the quota values to the LDAP backend which will then be visible in the selfcare webinterface. Like that you have a more or less up to date quota representation in the selfcare webinterface. On the other side, you don't want to send notification mails to the user every 10 minutes. It might be enough to send them once a day, so the cronjobs may look like:
<pre>
# Check backups
15 12 * * * /usr/libexec/backup-utils/checkBackups.pl --schedule --quota --notify

# Write quota values every 10 minutes
*/10 * * * * /usr/libexec/backup-utils/checkBackups.pl --quota
</pre>

== ResellerBackupBilling.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. The pod documentation says all what needs to be said about this script.
* Pod documentation
<pre>
NAME
ResellerBackupBilling.pl

DESCRIPTION
This script processes the reseller with the passed UID and creates a
very simplistic backup-billing overview which then is sent to the
address specified in the configuration file. The billing overview
contains all backup accounts which belong to the given reseller as well
as their quota. According to the billing plan specified in the
configuration file, the script also calculates the price for each backup
account.

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

OPTIONS
--reseller/-r UID
Process the reseller with this UID

--debug/-d
Turn on debug mode

--help/-h
Display this help

USAGE
./ResellerBackupBilling.pl --reseller UID [--debug] [--help]

./ResellerBackupBilling.pl --reseller 2000000
This processes the reseller with the UID 200000 and sends the
billing overview to the address specified in the configuration file

CREATED
2014-01-24 pat.klaey@stepping-stone.ch created

VERSION
2014-01-24 pat.klaey@stepping-stone.ch created

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
PerlUtil::LDAPUtil;
PerlUtil::Logging;
PerlUtil::Mail;
Number::Format;
File::Basename;
Cwd 'abs_path';
</pre>
=== Configuration ===

== writeDate.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. This is a very simple script, it simply writes the current date and time to a file on the backup server before the rsnapshots rotates the backup. This date is used by the Sepiola Online Backup Client to display the iterations and their dates.
* Pod documentation:
<pre>
NAME
writeDate.pl

DESCRIPTION
This script writes the current date and time into the file for each
backup before the rsnapshots rotates the backup. This information is
used by the Sepiola Online Backup Client.

The configuration file for this script is stored under
/etc/backup-utils/writeDate.conf

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

-C configfile
The configuration file.

-U uid
The user id.

-d debug
Turns the debug mode on.

-h help
This online help.

USAGE
writeDate.pl [-C configuration_file] [-U uid] [-d debug] [-h help]

CREATED
2007-09-16 michael.eichenberger@stepping-stone.ch created

VERSION
2013-09-01 michael.eichenberger@stepping-stone.ch updated to reflect
changes in the new backup environment

2007-09-16 michael.eichenberger@stepping-stone.ch created

INCORPORATED CODE
Incorporate code with use:

warnings;
strict;
Config::IniFiles;
Getopt::Std;
Sys::Syslog;
</pre>

=== Configuration ===
/etc/backup-utils/writeDate.conf
<pre>
[Global]
BACKUP_DIRECTORY = /incoming
BACKUP_BACKUP_TIME_FILE = /.sepiola_backup/backupTime

[Syslog]
SYSLOG = rsnapshot
</pre>

== rsnapshot wrapper bash scripts ==

= Links =
* [http://www.openldap.org/ OpenLDAP], an open source implementation of the Lightweight Directory Access Protocol.
* [http://arthurdejong.org/nss-pam-ldapd/ nss-pam-ldapd], a Name Service Switch (NSS) module that allows your LDAP server to provide user account, group, host name, alias, netgroup, and basically any other information that you would normally get from /etc flat files or NIS.
* [http://www.gentoo.org/doc/de/ldap-howto.xml Gentoo Leitfaden zur OpenLDAP Authentifikation].
* [http://wiki.gentoo.org/wiki/Centralized_authentication_using_OpenLDAP Centralized authentication using OpenLDAP].
* [https://code.google.com/p/openssh-lpk/source/browse/trunk/schemas/openssh-lpk_openldap.schema openssh-lpk_openldap.schema] OpenSSH LDAP Public Keys.
* [http://sourceforge.net/projects/linuxquota/ linuxquota] Linux DiskQuota.
* [http://www.rsnapshot.org/ rsnapshot], a remote filesystem snapshot utility, based on rsync.
* [http://olivier.sessink.nl/jailkit/ Jailkit], set of utilities to limit user accounts to specific files using chroot() and or specific commands. Also includes a tool to build a chroot environment.
* [http://www.busybox.net/ Busybox] BusyBox combines tiny versions of many common UNIX utilities into a single small executable. Useful to reduce the number of files (and thus the complexity) when building a chroot.

[[Category:stoney backup]]

stoney backup: Server set-up

2014-06-13T13:50:28Z

David: /* Cronjobs */

= Abstract =
This document describes server setup for the stoney cloud (Online) Backup service, built upon the [http://www.gentoo.org/ Gentoo] Linux distribution.

= Overview =
After working through this documentation, you will be able to set up and configure your own (Online) Backup service server.

= Software Installation =

== Requirements ==
A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].

== Keywords & USE-Flags ==
For a minimal OpenLDAP directory installation:
echo "net-nds/openldap minimal sasl" >> /etc/portage/package.use
echo "net-nds/openldap ~amd64" >> /etc/portage/package.keywords

NSS and PAM modules for lookups using LDAP:
echo "sys-auth/nss-pam-ldapd sasl" >> /etc/portage/package.use
echo "sys-auth/nss-pam-ldapd ~amd64" >> /etc/portage/package.keywords
echo "sys-fs/quota ldap" >> /etc/portage/package.use

echo "=app-admin/jailkit-2.16 ~amd64" >> /etc/portage/package.keywords

For the prov-backup-rsnapshot daemon:
echo "dev-perl/Net-SMTPS ~amd64" >> /etc/portage/package.keywords
echo "perl-core/Switch ~amd64" >> /etc/portage/package.keywords

To build puttygen only without X11:
echo "net-misc/putty ~amd64" >> /etc/portage/package.keywords
echo "net-misc/putty -gtk" >> /etc/portage/package.use

== Emerge ==
emerge -va nss-pam-ldapd \
quota \
net-misc/putty \
app-admin/jailkit \
sys-apps/haveged \
net-misc/putty \
sys-apps/sst-backup-utils \
sys-apps/sst-prov-backup-rsnapshot

To list the dependencies of ebuilds, you can use <code>equery</code>:
equery depgraph sst-backup-utils
<pre>
* Searching for sst-backup-utils ...

* dependency graph for sys-apps/sst-backup-utils-0.1.0
`-- sys-apps/sst-backup-utils-0.1.0 amd64
`-- dev-perl/PerlUtil-0.1.0 (>=dev-perl/PerlUtil-0.1.0) amd64
`-- virtual/perl-Sys-Syslog-0.320.0 (virtual/perl-Sys-Syslog) amd64
`-- dev-perl/perl-ldap-0.530.0 (dev-perl/perl-ldap) amd64
`-- dev-perl/XML-Simple-2.200.0 (dev-perl/XML-Simple) amd64
`-- dev-perl/Config-IniFiles-2.780.0 (dev-perl/Config-IniFiles) amd64
`-- dev-perl/XML-Validator-Schema-1.100.0 (dev-perl/XML-Validator-Schema) amd64
`-- dev-perl/Date-Calc-6.300.0 (dev-perl/Date-Calc) amd64
`-- dev-perl/DateManip-6.310.0 (dev-perl/DateManip) amd64
`-- dev-perl/Schedule-Cron-Events-1.930.0 (dev-perl/Schedule-Cron-Events) amd64
`-- dev-perl/DateTime-Format-Strptime-1.520.0 (dev-perl/DateTime-Format-Strptime) amd64
`-- dev-perl/XML-SAX-0.990.0 (dev-perl/XML-SAX) amd64
`-- virtual/perl-MIME-Base64-3.130.0-r2 (virtual/perl-MIME-Base64) amd64
`-- dev-perl/Authen-SASL-2.160.0 (dev-perl/Authen-SASL) amd64
`-- dev-perl/Net-SMTPS-0.30.0 (dev-perl/Net-SMTPS) ~amd64
`-- dev-perl/text-template-1.450.0 (dev-perl/text-template) amd64
`-- virtual/perl-Getopt-Long-2.380.0-r2 (virtual/perl-Getopt-Long) amd64
`-- dev-perl/Parallel-ForkManager-1.20.0 (dev-perl/Parallel-ForkManager) amd64
`-- dev-perl/Time-Stopwatch-1.0.0 (dev-perl/Time-Stopwatch) amd64
`-- app-backup/rsnapshot-1.3.1-r1 (app-backup/rsnapshot) amd64
[ sys-apps/sst-backup-utils-0.1.0 stats: packages (20), max depth (1) ]
</pre>

For more information, visit the [http://www.gentoo.org/doc/en/gentoolkit.xml Gentoolkit] page.

= Base Server Software Configuration =
== OpenSSH ==
=== OpenSSH Configuration ===
Configure the OpenSSH daemon:
<source lang="bash">
vi /etc/ssh/sshd_config
</source>

Set following options:
<source lang="bash">
PubkeyAuthentication yes
PasswordAuthentication yes
UsePAM yes
Subsystem sftp internal-sftp
</source>

Make sure, that <code>Subsystem sftp internal-sftp</code> is the last line in the configuration file.

We want to reduce the numbers of chroot environments in one folder. As the <code>ChrootDirectory</code> configuration option only allows <code>%h</code> (home directory of the user) and <code>%u</code> (username of the user), we need to create the necessary matching rules in the form of:
<source lang="bash">
Match User *000
ChrootDirectory /var/backup/000/%u
AuthorizedKeysFile /var/backup/000/%u/%h/.ssh/authorized_keys
Match
Match User *001
ChrootDirectory /var/backup/001/%u
AuthorizedKeysFile /var/backup/001/%u/%h/.ssh/authorized_keys
Match
...
Match User *999
ChrootDirectory /var/backup/999/%u
AuthorizedKeysFile /var/backup/999/%u/%h/.ssh/authorized_keys
Match
</source>

The creation of the matching rules is done by executing the following bash commands:
<source lang="bash">
FILE=/etc/ssh/sshd_config;

for x in {0..999} ; do \
printf "Match User *%03d\n" $x >> ${FILE}; \
printf " ChrootDirectory /var/backup/%03d/%%u\n" $x >> ${FILE}; \
printf " AuthorizedKeysFile /var/backup/%03d/%%u/%%h/.ssh/authorized_keys\n" $x >> ${FILE}; \
printf "Match\n" >> ${FILE}; \
done
</source>

Don't forget to restart the OpenSSH daemon:
<source lang="bash">
/etc/init.d/sshd restart
</source>

=== OpenSSH Host Keys ===
If you migrate from a existing backup server, you might want to copy the ssh host keys to the new server. If you do so clients want see a difference between the two hosts as the fingerprint remains the same. Copy the following files from the existing host to the new:
* /etc/ssh/ssh_host_dsa_key
* /etc/ssh/ssh_host_ecdsa_key
* /etc/ssh/ssh_host_key
* /etc/ssh/ssh_host_rsa_key
* /etc/ssh/ssh_host_dsa_key.pub
* /etc/ssh/ssh_host_ecdsa_key.pub
* /etc/ssh/ssh_host_key.pub
* /etc/ssh/ssh_host_rsa_key.pub

Set the correct permissions on the new host:
chmod 600 /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_ecdsa_key /etc/ssh/ssh_host_key /etc/ssh/ssh_host_rsa_key
chmod 644 /etc/ssh/*.pub

And restart the ssh daemon. ''Caution'': do not close your existing ssh session as long as you are not sure the ssh daemon has restarted properly and you can login again.
/etc/init.d/sshd restart

== OpenLDAP ==
=== /etc/hosts ===
Update the <code>/etc/hosts</code> with the LDAP server:
/etc/hosts

# VIP of the LDAP Server
31.216.40.4 ldapm.stoney-cloud.org

=== Root CA Certificate Installation ===
Install the root CA certificate into the OpenSSL default certificate storage directory:
fqdn="cloud.stoney-cloud.org" # The fully qualified domain name of the server containing the root certificate.

cd /etc/ssl/certs/
wget --no-check-certificate https://${fqdn}/ca/FOSS-Cloud_CA.cert.pem
chown root:root /etc/ssl/certs/FOSS-Cloud_CA.cert.pem
chmod 444 /etc/ssl/certs/FOSS-Cloud_CA.cert.pem

Rebuild the CA hashes
c_rehash /etc/ssl/certs/

=== /etc/openldap/ldap.conf ===
Update the <code>/etc/openldap/ldap.conf</code>LDAP configuration file/environment variables:
/etc/openldap/ldap.conf

<pre>
# Used to specify a size limit to use when performing searches. The number should be an
# non-negative integer. SIZELIMIT of zero (0) specifies unlimited search size.
SIZELIMIT 20000

# Used to specify a time limit to use when performing searches. The number should be an
# non-negative integer. TIMELIMIT of zero (0) specifies unlimited search time to be used.
TIMELIMIT 45

# Specify how aliases dereferencing is done. DEREF should be set to one of never, always, search,
# or find to specify that aliases are never dereferenced, always dereferenced, dereferenced when
# searching, or dereferenced only when locating the base object for the search. The default is to
# never dereference aliases.
DEREF never

# Specifies the URI(s) of an LDAP server(s) to which the LDAP library should connect. The URI
# scheme may be either ldapor ldaps which refer to LDAP over TCP and LDAP over SSL (TLS)
# respectively. Each server's name can be specified as a domain- style name or an IP address
# literal. Optionally, the server's name can followed by a ':' and the port number the LDAP
# server is listening on. If no port number is provided, the default port for the scheme is
# used (389 for ldap://, 636 for ldaps://). A space separated list of URIs may be provided.
URI ldaps://ldapm.stoney-cloud.org

# Used to specify the default base DN to use when performing ldap operations. The base must be
# specified as a Distinguished Name in LDAP format.
BASE dc=stoney-cloud,dc=org

# This is a local copy of the certificate of the certificate authority
# used to sign the server certificate for the LDAP server I am using
TLS_CACERT /etc/ssl/certs/FOSS-Cloud_CA.cert.pem
</pre>

Check you configuration by doing a search:
ldapsearch -v -H "ldaps://ldapm.stoney-cloud.org" \
-b "dc=stoney-cloud,dc=org" \
-D "cn=Manager,dc=stoney-cloud,dc=org" \
-s one "(objectClass=*)" \
-LLL -W

The result should look something like:
ldap_initialize( ldaps://ldapm.stoney-cloud.org:636/??base )
filter: (objectClass=*)
requesting: All userApplication attributes
dn: ou=administration,dc=stoney-cloud,dc=org
objectClass: top
objectClass: organizationalUnit
ou: administration
...

== Random Number Generator (haveged) ==
Tools like putty are dependent on random numbers to be able to create certificates.

=== haveged - Generate random numbers and feed linux random device ===
The haveged daemon doesn't need any special configuration, therefore you can start it from the command line interface:
/etc/init.d/haveged start

Check, if the start was successful:
ps auxf | grep haveged

root 18001 1.0 0.0 7420 3616 ? Ss 08:48 0:00 /usr/sbin/haveged -r 0 -w 1024 -v 1

Add the haveged daemon to the default run level:
rc-update add haveged default

== nss-pam-ldapd ==
=== nslcd.conf — configuration file for LDAP nameservice daemon ===
/etc/nslcd.conf

<pre>
# This is the configuration file for the LDAP nameservice
# switch library's nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The uri pointing to the LDAP server to use for name lookups.
# Multiple entries may be specified. The address that is used
# here should be resolvable without using LDAP (obviously).
#uri ldap://127.0.0.1/
#uri ldaps://127.0.0.1/
#uri ldapi://%2fvar%2frun%2fldapi_sock/
# Note: %2f encodes the '/' used as directory separator
uri ldaps://ldapm.tombstone.ch

# The LDAP version to use (defaults to 3
# if supported by client library)
#ldap_version 3

# The distinguished name of the search base.
base dc=stoney-cloud,dc=org

# The distinguished name to bind to the server with.
# Optional: default is to bind anonymously.
binddn cn=Manager,dc=stoney-cloud,dc=org

# The credentials to bind with.
# Optional: default is no credentials.
# Note that if you set a bindpw you should check the permissions of this file.
bindpw myverysecretpassword

# The distinguished name to perform password modifications by root by.
#rootpwmoddn cn=admin,dc=example,dc=com

# The default search scope.
#scope sub
#scope one
#scope base

# Customize certain database lookups.
#base group ou=Groups,dc=example,dc=com
base group ou=groups,ou=backup,ou=services,dc=stoney-cloud,dc=org
base passwd ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
base shadow ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
#scope group onelevel
#scope hosts sub

#filter group (&(objectClass=posixGroup)(sstIsActive=TRUE))
filter passwd (&(objectClass=posixAccount)(sstIsActive=TRUE))
filter shadow (&(objectClass=shadowAccount)(sstIsActive=TRUE))

# Bind/connect timelimit.
#bind_timelimit 30

# Search timelimit.
#timelimit 30

# Idle timelimit. nslcd will close connections if the
# server has not been contacted for the number of seconds.
#idle_timelimit 3600

# Use StartTLS without verifying the server certificate.
#ssl start_tls
tls_reqcert never

# CA certificates for server certificate verification
#tls_cacertdir /etc/ssl/certs
#tls_cacertfile /etc/ssl/ca.cert

# Seed the PRNG if /dev/urandom is not provided
#tls_randfile /var/run/egd-pool

# SSL cipher suite
# See man ciphers for syntax
#tls_ciphers TLSv1

# Client certificate and key
# Use these, if your server requires client authentication.
#tls_cert
#tls_key

# Mappings for Services for UNIX 3.5
#filter passwd (objectClass=User)
#map passwd uid msSFU30Name
#map passwd userPassword msSFU30Password
#map passwd homeDirectory msSFU30HomeDirectory
#map passwd homeDirectory msSFUHomeDirectory
#filter shadow (objectClass=User)
#map shadow uid msSFU30Name
#map shadow userPassword msSFU30Password
#filter group (objectClass=Group)
#map group member msSFU30PosixMember

# Mappings for Services for UNIX 2.0
#filter passwd (objectClass=User)
#map passwd uid msSFUName
#map passwd userPassword msSFUPassword
#map passwd homeDirectory msSFUHomeDirectory
#map passwd gecos msSFUName
#filter shadow (objectClass=User)
#map shadow uid msSFUName
#map shadow userPassword msSFUPassword
#map shadow shadowLastChange pwdLastSet
#filter group (objectClass=Group)
#map group member posixMember

# Mappings for Active Directory
#pagesize 1000
#referrals off
#idle_timelimit 800
#filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map passwd uid sAMAccountName
#map passwd homeDirectory unixHomeDirectory
#map passwd gecos displayName
#filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map shadow uid sAMAccountName
#map shadow shadowLastChange pwdLastSet
#filter group (objectClass=group)

# Alternative mappings for Active Directory
# (replace the SIDs in the objectSid mappings with the value for your domain)
#pagesize 1000
#referrals off
#idle_timelimit 800
#filter passwd (&(objectClass=user)(objectClass=person)(!(objectClass=computer)))
#map passwd uid cn
#map passwd uidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820
#map passwd gidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820
#map passwd homeDirectory "/home/$cn"
#map passwd gecos displayName
#map passwd loginShell "/bin/bash"
#filter group (|(objectClass=group)(objectClass=person))
#map group gidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820

# Mappings for AIX SecureWay
#filter passwd (objectClass=aixAccount)
#map passwd uid userName
#map passwd userPassword passwordChar
#map passwd uidNumber uid
#map passwd gidNumber gid
#filter group (objectClass=aixAccessGroup)
#map group cn groupName
#map group gidNumber gid
</pre>

=== nsswitch.conf - Name Service Switch configuration file ===
/etc/nsswitch.conf

<pre>
passwd: files ldap
shadow: files ldap
group: files ldap

# passwd: db files nis
# shadow: db files nis
# group: db files nis

hosts: files dns
networks: files dns

services: db files
protocols: db files
rpc: db files
ethers: db files
netmasks: files
netgroup: files
bootparams: files

automount: files
aliases: files
</pre>

=== system-auth ===
vi /etc/pam.d/system-auth

<pre>
auth required pam_env.so
auth sufficient pam_unix.so try_first_pass likeauth nullok
auth sufficient pam_ldap.so minimum_uid=1000 use_first_pass
auth required pam_deny.so

account required pam_unix.so
account sufficient pam_ldap.so minimum_uid=1000 use_first_pass

password required pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
password required pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password sufficient pam_ldap.so minimum_uid=1000 use_first_pass
password required pam_deny.so

session required pam_limits.so
session required pam_env.so
session required pam_unix.so
session sufficient pam_ldap.so minimum_uid=1000 use_first_pass
</pre>

=== Test the Setup ===
nslcd -d

=== Update the Default Run Levels ===
rc-update add nslcd default
rc-update add nscd default

=== Start the necessary Daemons ===
/etc/init.d/nslcd start
/etc/init.d/nscd start

== Quota ==
=== 32-bit Project Identifier Support ===
We need to enable 32-bit project identifier support (PROJID32BIT feature) for our naming scheme (uid numbers larger than 65'536), which is already the default on the stepping stone virtual machines:
mkfs.xfs '''-i projid32bit=1''' /dev/vg-local-01/var

=== Update /etc/fstab and Mount ===
Make sure, that you have user quota (uqota) and project quota (pquota) set as options on the chosen mount point in /etc/fstab. For example:
LABEL=LV-VAR /var xfs noatime,discard,inode64,uquota,pquota 0 2

reboot

Check, if everything went ok:
df -h | grep var

/dev/mapper/vg--local--01-var 1023G 220G 804G 22% /var

=== Verify ===
Some important options for xfs_quota:
* -x: Enable expert mode.
* -c: Pass arguments on the command line. Multiple arguments may be given.

Remount the file system /var and check, if /var has the desired values:
xfs_quota -x -c state /var

As you can see (items marked bold), we have achieved our goal:
User quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: '''ON'''
Enforcement: '''ON'''
Inode: #131 (1 blocks, 1 extents)
Group quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: OFF
Enforcement: OFF
Inode: #132 (1 blocks, 1 extents)
Project quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: '''ON'''
Enforcement: '''ON'''
Inode: #132 (1 blocks, 1 extents)
Blocks grace time: [7 days 00:00:30]
Inodes grace time: [7 days 00:00:30]
Realtime Blocks grace time: [7 days 00:00:30]

=== User Quotas ===
==== Adding a User Quota ====
Set a quota of 1 Gigabyte for the user 4000187 (the values are in kilobytes, so 1048576 kilobyte are 1024 megabytes which corresponds to 1 gigabyte):
xfs_quota -x -c 'limit bhard=1048576k 4000187' /var

Or in bytes:
xfs_quota -x -c 'limit bhard=1073741824 4000187' /var

Read the quota information for the user 4000187:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If the user has data in the project, that belongs to him, the result will change:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

==== Modifiying a User Quota ====
To modify a users quota, you just set a new quota (limit):
xfs_quota -x -c 'limit bhard=1048576k 4000187' /var

Read the quota information for the user 4000187:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If the user has data in the project, that belongs to him, the result will change:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

==== Removing a User Quota ====
Removing a quota for a user:
xfs_quota -x -c 'limit bhard=0 4000187' /var

The following command should give you an empty result:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

=== Project (Directory) Quotas ===
==== Adding a Project (Directory) Quota ====
The XFS file system additionally allows you to set quotas on individual directory hierarchies in the file system that are known as managed trees. Each managed tree is uniquely identified by a project ID and an optional project name. We'll use the following values in the examples:
* project_ID: The uid of the online backup account (4000187).
* project_name: The uid of the online backup account (4000187). This could be a human readable name.
* mountpoint: The mountpoint of the xfs-filesystem (/var). See the <code>/etc/fstab</code> entry from above.
* directory: The directory of the project (187/4000187), starting from the mountpoint of the xfs-filesystem (/var).

Define a unique project ID for the directory hierarchy in the <code>/etc/projects</code> file (project_ID:mountpoint/directory):
echo "4000187:/var/backup/187/4000187/home/4000187" >> /etc/projects

Create an entry in the <code>/etc/projid</code> file that maps a project name to the project ID (project_name:project_ID):
echo "4000187:4000187" >> /etc/projid

Set Project:
xfs_quota -x -c 'project -s -p /var/backup/187/4000187/home/4000187 4000187' /var

Set Quota (limit) on Project:
xfs_quota -x -c 'limit -p bhard=1048576k 4000187' /var

Check your Quota (limit)
xfs_quota -x -c 'quota -p 4000187' /var

Check the Quota:
* <code>-v</code>: increase verbosity in reporting (also dumps zero values).
* <code>-N</code>: suppress the initial header.
* <code>-p</code>: display project quota information.
* <code>-h</code>: human readable format.
xfs_quota -x -c 'quota -v -N -p 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If you copied data into the project, the output will look something like:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

To give you an overall view of the whole system:
xfs_quota -x -c report /var

<pre>
User quota on /var (/dev/mapper/vg--local--01-var)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 1024000 0 0 00 [--------]
4000187 0 0 1048576 00 [--------]

Project quota on /var (/dev/mapper/vg--local--01-var)
Blocks
Project ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
4000187 512000 0 1048576 00 [--------]
</pre>

==== Modifying a Project (Directory) Quota ====
To modify a project (directory) quota, you just set an new quota (limit) on the chosen project:
xfs_quota -x -c 'limit -p bhard=1048576k 4000187' /var

Check your quota (limit)
xfs_quota -x -c 'quota -p 4000187' /var

==== Removing a Project (Directory) Quota ====
Removing a quota from a project:
xfs_quota -x -c 'limit -p bhard=0 4000187' /var

Chreck the results:
xfs_quota -x -c report /var

<pre>
User quota on /var (/dev/mapper/vg--local--01-var)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 512000 0 0 00 [--------]
4000187 0 0 1024 00 [--------]
</pre>

As you can see, the line with the Project ID 4000187 has disappeared:
4000187 512000 0 1048576 00 [--------]

Don't forget to remove the project from <code>/etc/projects</code> and <code>/etc/projid</code>:
sed -i -e '/4000187/d' /etc/projects
sed -i -e '/4000187/d' /etc/projid

=== Some important notes concerning XFS ===
# The '''quotacheck''' command has no effect on XFS filesystems. The first time quota accounting is turned on (at mount time), XFS does an automatic quotacheck internally; afterwards, the quota system will always be completely consistent until quotas are manually turned off.
# There is '''no need for quota file(s)''' in the root of the XFS filesystem.

== prov-backup-rsnapshot ==
Install the prov-backup-rsnasphot daemon script using the package manager:
<pre>
emerge -va sys-apps/sst-prov-backup-rsnapshot
</pre>

=== Configuration ===
If it is the first provisioning module running on this server (very likely) you first have to configure the provisioning daemon (you can skip this step if you have already another provisioning module running on this server)

==== Provisioning global configuration ====
The global configuration for the provisioning daemon applies to all provisioning modules running on the server. This configuration therefore contains information about the provisioning daemon itself and no information at all about the specific modules.
/etc/Provisioning/Global.conf
<pre>
# Copyright (C) 2012 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 0

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The number of seconds to wait before retry contacting the backend server during startup.
SLEEP = 10

# Number of backend server connection retries during startup.
ATTEMPTS = 3

[Operation Mode]
# The number of seconds to wait before retry contacting the backend server in case of a service interruptions.
SLEEP = 30

# Number of backend server connection retries in case of a service interruptions.
ATTEMPTS = 3

[Mail]
# Error messages are sent to the mail configured below.
SENDTO = <YOUR-MAIL-ADDRESS>
HOST = mail.stepping-stone.ch
PORT = 587
USERNAME = <YOUR-NOTIFICATION-EMAIL-ADDRESS>
PASSWORD = <PASSWORD>
FROMNAME = Provisioning daemon
CA_DIR = /etc/ssl/certs
SSL = starttls
AUTH_METHOD = LOGIN

# Additionally, you can be informed about creation, modification and deletion of services.
WANTINFOMAIL = 1
</pre>

==== Provisioning daemon prov-backup-rsnapshot module ====
The module specific configuration is located in /etc/Provisioning/<Service>/<Type>.conf. In the case of the prov-backup-rsnapshot module this is <code>/etc/Provisioning/Backup/Rsnapshot.conf</code>. (Note: Comments starting with /* are not in the configuration file, they are only in the wiki to add some additional information)

<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

/* If you want, you can override the log information from the global configuration file this might be useful for debugging */
[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

/* Specify the hosts fully qualified domain name. This name will be used to perform some checks and also appear in the information and error mails */
ENVIRONMENT = <FQDN>

[Database]
BACKEND = LDAP
SERVER = ldaps://ldapm.tombstone.org
PORT = 636
ADMIN_USER = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <PASSWORD>
SERVICE_SUBTREE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
COOKIE_FILE = /etc/Provisioning/Backup/rsnapshot.cookie
DEFAULT_COOKIE = rid=001,csn=
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(sstProvisioningState=0))

/* Specifies the service itself. As it is the prov-backup-rsnapshot module, the SERVICE is "Backup" and the TYPE is "Rsnapshot".
* The MODUS is as usual selfcare and the TRANSPORTAPI is LocalCLI. This is because the daemon is running on the same host as the
* backup accounts are provisioned and the commands can be executed on this host using the cli.
* For more information about MODUS and TRANSPORTAPI see https://int.stepping-stone.ch/wiki/provisioning.pl#Service_Konfiguration
*/
[Service]
MODUS = selfcare
TRANSPORTAPI = LocalCLI
SERVICE = Backup
TYPE = Rsnapshot

SYSLOG = prov-backup-rsnapshot

/* For the TRANSPORTAPI LocalCLI there is no gateway required because there is no connection to establish. So set HOST, USER and
* DSA_FILE to whatever you want. Don't leave it blank, otherwise the provisioning daemon would log some error messages saying
* these attributes are empty
*/
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

/* Information about the backup itself (how to setup everything). Note that the %uid% int the RSNAPSHOT_CONFIG_FILE parameter will
* be replaced by the accounts UID. The script CREATE_CHROOT_CMD was installed with the prov-backup-rsnapshot module, so do not
* change this parameter. The quota parameters (SET_QUOTA_CMD, MOUNTPOINT, QUOTA_FILE, PROJECTS_FILE and PROJID_FILE) represent
* the quota setup as described on http://wiki.stoney-cloud.org/index.php/stoney_backup:_Server_set-up#Quota. If you followed this
* manual, you can copy-paste them into your configuration file, otherwise adapt them according to your quota setup.
*/
[Backup]
RSNAPSHOT_CONFIG_FILE = /etc/rsnapshot/rsnapshot.conf.%uid%
SET_QUOTA_CMD = /usr/sbin/xfs_quota
CREATE_CHROOT_CMD = /usr/libexec/createBackupDirectory.sh
MOUNTPOINT = /var
QUOTA_FILE = /etc/backupSize
PROJECTS_FILE = /etc/projects
PROJID_FILE = /etc/projid
</pre>

== backup utils ==
Install the backup utils (multiple scripts which help you to manage and monitor your backup server and backup accounts) using the package manager. For more information about the scripts please see the [[stoney_backup:_Server_set-up#stoney_backup_Service_Software | stoney backup Service Software]] section.
<pre>
emerge -va sys-apps/sst-backup-utils
</pre>
=== Configuration ===
Please refer to the configuration sections for the different scripts in [[stoney_backup:_Server_set-up#stoney_backup_Service_Software | stoney backup Service Software]].

= stoney backup Service Software =
The stoney backup Service comes along with multiple scripts which help you to manage and monitor your backup server and accounts:

We use rsnapshot - remote filesystem snapshot utility for the actual snapshots and a handful of wrapper scripts, that do things like:
* Read the users and their settings from the LDAP directory.
* Execute rsnapshot according to the users settings.
* Write the backup quotas backup (incoming), iterations (.snapshots) and free space to the users local backupSize file and update the LDAP directory.
* Inform the reseller, customer or user (depending on the settings in the LDAP directory) via mail, if the quota limit has been reached.
* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.

== writeAccountSize.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeAccountSize.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* Calculates the used disk space (backup and iterations) for a given account and writes the corresponding values to:
*# The LDAP backend (used by the selfcare webinterface to display quota information):
*#* Backup space used (sstBackupSize): The disk space the account uses for the backup itself (disk space used under the <code>incoming</code> folder of the users chroot-home directory)
*#* Snapshot space used (sstIncrementSize): The disk space the account uses for the iterations (disk space under the <code>.snapshot</code> folder of the users chroot-home directory)
*# The file <code>etc/backupSize</code> of the accounts chroot (used by the Sepiola Online Backup client):
* Checks if the user and/or reseller must be notified that there is no more disk space left for the given account
** Checks if the notification flag was passed, if not no notification will be triggered
** Calculates the used disk space (backup and iterations) in percentage
** Reads the notification threshold value from the LDAP backed
** If the disk space used (in percentage) is bigger than the value retrieved from the LDAP backend start the [[stoney_cloud:_Notification_Architecture | notification]] process with
*** Product: Given account UID
*** Service: Backup
*** Problem: Quota
* Pod documentation:
<pre>
NAME
writeAccountSize.pl

DESCRIPTION
This Script gets quota information from filesystem, size of incoming and
snapshots directories, write the data to a file and the LDAP backend and
sends an e-mail message for each account that is over quota to users
e-mail address (from ldap directory) if notification flag is passed.

The configuration file for this script is stored in the backup-utils
configuration directory (/etc/backup-utils/) and is called
writeAccountSize.conf.

The script needs access to the quota program to get quota information.
The script needs ldap access to get users e-mail address and quota
information. The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

-C configfile
The configuration file.

-U uid
The user id.

-n notification
Start notification process if quota threshold is reached

-d debug
Turns the debug mode on.

-h help
This online help.

USAGE
writeAccountSize.pl -U uid [-C configuration file ] []-d debug] [-h
help] [-n]

CREATED
2009-04-16 michael.rhyner@stepping-stone.ch created

VERSION
2009-04-16 michael.rhyner@stepping-stone.ch created
2009-04-30 michael.rhyner@stepping-stone.ch changed position based quota
output parsing with correctly parsed elements
2009-06-15 michael.rhyner@stepping-stone.ch added over quota check and
sending e-mail
2009-06-16 michael.rhyner@stepping-stone.ch renamed script and make it
more general usable (e.g. for online backup, online storage, ...)
2009-06-17 michael.rhyner@stepping-stone.ch changed mail message to read
from a text file instead from configuration parameter
2009-06-18 michael.rhyner@stepping-stone.ch corrected wrong regex to
weed out the asterisk (*) in getQuotaSize
2009-06-19 michael.rhyner@stepping-stone.ch corrected wrong evaluation
success from subroutines and avoid message output when not in debug mode
2009-06-22 michael.rhyner@stepping-stone.ch getQuotaSize: return
immediately if no quota was set
2009-06-24 michael.rhyner@stepping-stone.ch alert when used certain
percentge of allowed space instead of more than allowed space
2009-06-26 michael.rhyner@stepping-stone.ch values are presented in
Gigabytes within notification message
2009-07-23 michael.rhyner@stepping-stone.ch corrected wrong syslog
severities for errors
2009-07-24 michael.rhyner@stepping-stone.ch made e-mail address
available within message body
2013-08-19 pat.klaey@stepping-stone.ch write quota values also to the
LDAP

INCORPORATED CODE
Incorporate code with use:

warnings;
strict;
Config::IniFiles;
Getopt::Std;
Sys::Syslog;
File::Basename;
Text::Template;
POSIX;
Notification;
PerlUtil::Logging;
PerlUtil::LDAPUtil;
</pre>

=== Configuration ===
/etc/backup-utils/writeAccountSize.conf
<pre>
[Global]
INCOMING_DIRECTORY = /incoming
ACCOUNT_SIZE_FILE = /etc/backupSize
SNAPSHOTS = 1

[Syslog]
SYSLOG = rsnapshot

[Directory]
LDAP_SERVER = ldaps://ldapm.tombstone.ch
LDAP_PORT = 636
LDAP_BIND_DN = cn=Manager,dc=stoney-cloud,dc=org
LDAP_BIND_PW = <password>
LDAP_BASE_DN = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
LDAP_PERSON_BASE = ou=people,dc=stoney-cloud,dc=org
LDAP_RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=org
LDAP_EMAIL_ATTRIBUTE = mail

[Notification]
EMAIL_SENDER = stepping stone GmbH Supprt <support@stepping-stone.ch>
EMAIL_ALERT_THRESHOLD = 85

Salutation_Default_de-CH = Liebe Kundin / Lieber Kunde
Salutation_m_de-CH = Sehr geehrter Herr
Salutation_f_de-CH = Sehr geehrte Frau
Salutation_Default_en-GB = Dear customer
Salutation_m_en-GB = Dear Mr.
Salutation_f_en-GB = Dear Mrs.
</pre>

=== Tests ===
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d
<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 0 kilobytes
Incoming Size: 0 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 0
Debug sub getSnapshotsSize: $localSnapshotsSize: 0
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 0 0 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 0

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0

Alert Threshold: 85 %
Calculated value: 0
</pre>

Now write some data (200 megaytes in this example) into the users incoming directory and then execute the script again:
dd if=/dev/zero of=/var/backup/080/4000080/home/4000080/incoming/test.zeros bs=1024k count=200
chown 4000080:4000080 /var/backup/080/4000080/home/4000080/incoming/test.zeros
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d

<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 204800 kilobytes
Incoming Size: 204800 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 204800
Debug sub getSnapshotsSize: $localSnapshotsSize: 0
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 200 0 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0

Alert Threshold: 85 %
Calculated value: 19.53125
</pre>
Everything seems to be working fine!

== snapshot.pl ==
This script is installed to <code>/usr/libexec/backup-utils/snapshot.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* Read interval parameter value passed
** The interval parameter value can be daily, weekly, monthly (or yearly)
* Find all backup active accounts for which the rsnapshot command must be executed (depending on the given interval)
** Filter to find these accounts: <code>(&(sstBackupInterval<INTERVAL>=*)(sstIsActive=TRUE))</code> for example for the daily rsnapshot the filter is <code>(&(sstBackupIntervalDaily=*)(sstIsActive=TRUE))</code>
*** In other words this means: Get me all acounts that have for <code>sstBackupInterval<INTERVAL></code> a value defined AND <code>sstIsActive</code> is set to "TRUE"
* According to the interval given and the account UID calculate the rsnapshot command for all these accounts
** For example
*** Account UID: 4000000
*** Interval: daily
*** Resulting rsnapshot command: /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.4000000 daily (if you use the [[#snapshot.pl_Configuration | configuration]] below)
* Execute all these commands
** Use controlled parallel execution, you can specify how many commands can be executed in parallel (see [[#snapshot.pl_Configuration | configuration]] below)
* Pod documentation:
<pre>
NAME
snapshot.pl

DESCRIPTION
This script gets all active online backup accounts from the LDAP backend
for which the rsnapshot process for the given interval must be executed.
According to these accounts and the given interval, the commands to be
executed are generated and finally executed. The commands can be
executed in parallel, however there is a limit defined in the
configuration file which limits the amount of parallel running
processes.

USAGE
./snapshot.pl --interval interval [--debug] [--help]

OPTIONS
--interval/-i interval
Specifies the rsnapshot interval (can be hourly, daily, weekly,
monthly or yearly)

--debug/-d
Turns on debug mode

--help/-h
Shows this help

CREATED
2012-03-19 pat.klaey@stepping-stone.ch created

VERSION
2012-03-19 pat.klaey@stepping-stone.ch created

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
Sys::Syslog;
PerlUtil::Logging;
PerlUtil::LDAPUtil;
File::Basename;
Parallel::ForkManager;
Time::Stopwatch;
</pre>

=== Configuration ===
There are two things to do in this step:
* Configure the rsnapshot root directory
* Configure the snapshot.pl script itself

==== rsnaphot configuration directory ====
The users individual rsnapshot configurations are stored under <code>/etc/rsnapshot</code>. Please make sure, that the directory exists:
ls -al /etc | grep rsnapshot

drwx------ 2 root root 64 30. Aug 20:20 rsnapshot

If not, create it:
mkdir /etc/rsnapshot
chmod 700 /etc/rsnapshot

==== snapshot.pl Configuration ====
The snapshot.pl script is responsible for the execution of rsnapshot according to the users settings.
/etc/backup-utils/snapshot.conf
<pre>
[General]
MaxParallelProcesses = 5
Rsnapshot_command = /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.%uid% %interval%

[LDAP]
Host = ldaps://ldapm.tombstone.ch
Port = 636
User = cn=Manager,dc=stoney-cloud,dc=org
Password = <Password>
CA_Path = /etc/ssl/certs
Accounts_Base = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
</pre>

Legend: At runtime the following placeholders are replaced as follows
* '''%uid%''': The backup account and login uid as a numeric number. For example: 4000205.
* '''%interval%''': The backup level to be executed. Possible values are hourly, daily, weekly, monthly and yearly.

=== Tests ===
Before adding the necessary cronjob entries, we need to make sure, that we've configured the snapshot.pl script correctly:
/usr/libexec/backup-utils/snapshot.pl --interval daily -d

If everything worked as planned, you should receive feedback looking roughly like:
<pre>
INFO: Starting rsnapshot for interval daily with maximum 5 parallel processes

INFO: Executing snapshot for 4000080

INFO: Executing snapshot for 4000079

INFO: Snapshot process for 4000079 finished in 0.18 seconds with status 0

INFO: Snapshot process for 4000080 finished in 0.19 seconds with status 0

INFO: rsnapshot for all backups done. Took 0.24 seconds
</pre>

Just to make sure, that everything did work out fine, execute <code>writeAccountSize.pl</code> again:
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d
<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 409600 kilobytes
Incoming Size: 204800 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 409600
Debug sub getSnapshotsSize: $localSnapshotsSize: 204800
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 200 200 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 209715200

Alert Threshold: 85 %
Calculated value: 39.0625
</pre>

As you can see, the total used space has risen to 39.0625.

=== Cronjobs ===
After making sure, that everything worked as planned, you can update your crontab entry:
crontab -e
<pre>
...
# Rsnapshot for all users
30 22 * * * /usr/libexec/backup-utils/snapshot.pl --interval daily
15 22 * * sun /usr/libexec/backup-utils/snapshot.pl --interval weekly
00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly
...
</pre>

* TBD: Maybe this is not optimal if there is a lot of data to rotate. In this case, it might be that for example weekly and daily snapshot both run at the same time what might lead to strange results.
* Workarounds:
** Simple/short version: Instead one could create another wrapper script which is called everyday and does some simple logic:
** Is today the first day of a month?
*** Yes: Is today sunday?
**** Yes: Execute monthly then weekly then daily rsnapshots using the snapshot.pl script (but wait for each interval to finish before starting the next)
**** No: Execute monthly then daily rsnapshots using the snapshot.pl script (but wait for monthly interval to finish before starting the daily)
*** No: Is today sunday?
**** Yes: Execute weekly then daily rsnapshots using the snapshot.pl script (but wait for weekly interval to finish before starting the daily)
**** No: Execute daily rsnapshots using the snapshot.pl script
** Complex/long version: Adapt the snapshot.pl script and call it every day without interval parameter. The script does the logic described above.
This avoids the above mentioned collision

(Complex) / short version: rule via cronjobs:
<pre>
# Rsnapshot for all users
30 22 * * 1-6 [ $( date +\%d ) -gt 1 ] && /usr/libexec/backup-utils/snapshot.pl --interval daily
15 22 * * 7 [ $( date +\%d ) -gt 1 ] && /usr/libexec/backup-utils/snapshot.pl --interval weekly; /usr/libexec/backup-utils/snapshot.pl --interval daily
00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly; /usr/libexec/backup-utils/snapshot.pl --interval weekly; /usr/libexec/backup-utils/snapshot.pl --interval daily
...
</pre>

== scheduleWarning.pl ==
This script is installed to <code>/usr/libexec/backup-utils/scheduleWarning.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* The basic task of this script is simple: For the given account
** Check if the planned backups were started
*** If not, start the [[stoney_cloud:_Notification_Architecture | notification]] process with
**** Product: Given account UID
**** Service: Backup
**** Problem: Schedule
** Check if the planned backups finished successfully
*** If not, start the [[stoney_cloud:_Notification_Architecture | notification]] process with
**** Product: Given account UID
**** Service: Backup
**** Problem: Unsuccessful

As the backup clients distributed by stepping stone GmbH upload metadata XML before (scheduling information and start time) and after (end time and backup status) the actual backup, the scheduleWarning.pl script is able to verify whether or not a planned backup has been executed and whether or not the backup was successful.

* Pod documentation
<pre>
NAME
scheduleWarning.pl

DESCRIPTION
This script tests whether a planed backup was successful or not. There
are two different typs of failure.

1. The Backup did not start
The first type of error is that a backup is scheduled for YYYY-MM-DD
at HH:MM, but the backup don't start at this specified time. A
possible reason could be that the computer was shut down.

2.The backup was not successful
The second type of error is that a backup started as scheduled but
did not finish successfully. There are different reasons for this
error.

If a backup wasn't successful the script checks which type of error
occured. It reads the XML files which are stored on the server and
compares the given information. If the error is detected, the script
stats the norification process with the information of the error.

If a user has more than one computer backed-up, the script tests one
computer after the other. The mail(s) sent by the programm also
contain(s) the information which computer is affected.

OPTION
-U uid
The -U option is required to run the script, it indicates for which
uid the script is executed.

USAGE
scheduleWarning.pl [-U user]

CREATED
2010-04-14 created Pat Kläy <pat.klaey@stepping-stone.ch>

VERSION
2010-04-14 v0.01, created pkl
2010-08-24 v0.02, modified pkl
New using Net::SMTP::TLS to send mails

2013-09-13 v0.03, modified pat.klaey@stepping-stone.ch
Changes to use the script with the new backup infrastructure (read
more information from LDAP, use Notification lib to send mails)

USES
strict;
warnings;
XML::Simple;
Config::IniFiles;
XML::Validator::Schema;
Date::Calc qw(:all);
Date::Manip;
Schedule::Cron::Events;
DateTime::Format::Strptime;
Sys::Syslog;
XML::SAX::ParserFactory;
Getopt::Std;
MIME::Base64;
Authen::SASL;
Net::LDAPS;
Net::SMTP::TLS;
Cwd 'abs_path';
PerlUtil::Logging;
PerlUtil::LDAPUtil;
</pre>

=== Configuration ===
/etc/backup-utils/scheduleWarning.conf
<pre>
[Backup]
CHROOT_DIRECTORY = /var/backup/%lastthree%/%user%

[XML]
SCHEDULE_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/scheduler.xml
SCHEDULE_XSD = /etc/backup-utils/schema/scheduler_schema.xsd
BACKUP_ENDED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupEnded.xml
BACKUP_ENDED_XSD = /etc/backup-utils/schema/backupended_schema.xsd
BACKUP_STARTED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupStarted.xml
BACKUP_STARTED_XSD = /etc/backup-utils/schema/backupstarted_schema.xsd

[TEMPLATE]
Salutation_Default_de-CH = Liebe Kundin / Lieber Kunde
Salutation_m_de-CH = Sehr geehrter Herr
Salutation_f_de-CH = Sehr geehrte Frau
Salutation_Default_en-GB = Dear customer
Salutation_m_en-GB = Dear Mr.
Salutation_f_en-GB = Dear Mrs.

[LDAP]

SERVER = ldaps://ldapm.tombstone.ch
PORT = 636
DEBUG = 1

ADMIN_DN = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <Password>

BACKUP_BASE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
PEOPLE_BASE = ou=people,dc=stoney-cloud,dc=org
RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=org
SCOPE = sub

</pre>

== checkBackups.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. It is a wrapper around the above mentioned <code>writeAccountSize.pl</code> and <code>scheduleWarning.pl</code> scripts (if you configure it accordingly).
* As mentioned the script is a wrapper for calling the quota and scheduler information scripts:
** It gets all active backup accounts from the LDAP directory
** According to the CLI parameters passed it calles for all these accounts:
*** The quota checking script (this script can be defined in the configuration file, see [[stoney_backup:_Server_set-up#Configuration_6 | configuration]] section below)
*** The scheduler checking script (this script can be defined in the configuration file, see [[stoney_backup:_Server_set-up#Configuration_6 | configuration]] section below)
* Pod documentation:
<pre>
NAME
checkBackups.pl

DESCRIPTION
This script processes all active backup accounts and checks (according
to the command line options) the quota and/or the scheduled backups for
the given accounts.

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

OPTIONS
--schedule/-s
Checks schedule information by calling the defined script

--quota/-s
Checks the quota values by calling the defined script

--notify/-n
Passes the notify flag to the called scripts

USAGE
./checkBackups.pl [--quota] [--schedule] [--notify]

./checkBackups.pl --quota
Checks and writes the quota for all active backup accounts. The user
and/or reseller will NOT be informed if any quota threshold is
reached

./checkBackups.pl --quota --notify
Checks and writes the quota for all active backup accounts. The user
and/or reseller will be informed if any quota threshold is reached.

./checkBackups.pl --schedule
Checkes all active backup account if the scheduled backup was
executed and successful. The --schedule option will allways trigger
a notification mail to the user and/or reseller.

./checkBackups.pl --schedule --quota
Checks both, quota and schedule, for all active backup accounts.
Only schedule will trigger notification mails.

./checkBackups.pl --schedule --quota --notify
Checks both, quota and schedule, for all active backup accounts.
Both, quota and schedule, will trigger notification mails.

CREATED
2013-09-17 pat.klaey@stepping-stone.ch created

VERSION
2013-09-17 pat.klaey@stepping-stone.ch created
2013-11-19 pat.klaey@stepping-stone.ch Added options to be able to check
quota, schedule or both in one run

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
Sys::Syslog;
Cwd 'abs_path';
File::Basename;
PerlUtil::LDAPUtil;
PerlUtil::Logging;
</pre>

=== Configuration ===
In the configuration you have to define two simple things:
# Which script to call for quota checks and scheduler checks
# How to access the LDAP backend
/etc/backup-utils/checkBackups.conf
<pre>
[Scripts]
CheckQuotaScript = /usr/libexec/backup-utils/writeAccountSize.pl
CheckScheduleScript = /usr/libexec/backup-utils/scheduleWarning.pl

[LDAP]
Server = ldaps://ldapm.tombstone.ch
Port = 636
Username = cn=Manager,dc=stoney-cloud,dc=org
Password = <PASSWORD>
AccountBase = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
</pre>

==== Cronjobs ====
As you can pass the notification flag to the scripts, you can call the quota script say all 10 minutes. This makes sense if followed the configuration section, as the writeAccountSize.pl script writes the quota values to the LDAP backend which will then be visible in the selfcare webinterface. Like that you have a more or less up to date quota representation in the selfcare webinterface. On the other side, you don't want to send notification mails to the user every 10 minutes. It might be enough to send them once a day, so the cronjobs may look like:
<pre>
# Check backups
15 12 * * * /usr/libexec/backup-utils/checkBackups.pl --schedule --quota --notify

# Write quota values every 10 minutes
*/10 * * * * /usr/libexec/backup-utils/checkBackups.pl --quota
</pre>

== ResellerBackupBilling.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. The pod documentation says all what needs to be said about this script.
* Pod documentation
<pre>
NAME
ResellerBackupBilling.pl

DESCRIPTION
This script processes the reseller with the passed UID and creates a
very simplistic backup-billing overview which then is sent to the
address specified in the configuration file. The billing overview
contains all backup accounts which belong to the given reseller as well
as their quota. According to the billing plan specified in the
configuration file, the script also calculates the price for each backup
account.

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

OPTIONS
--reseller/-r UID
Process the reseller with this UID

--debug/-d
Turn on debug mode

--help/-h
Display this help

USAGE
./ResellerBackupBilling.pl --reseller UID [--debug] [--help]

./ResellerBackupBilling.pl --reseller 2000000
This processes the reseller with the UID 200000 and sends the
billing overview to the address specified in the configuration file

CREATED
2014-01-24 pat.klaey@stepping-stone.ch created

VERSION
2014-01-24 pat.klaey@stepping-stone.ch created

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
PerlUtil::LDAPUtil;
PerlUtil::Logging;
PerlUtil::Mail;
Number::Format;
File::Basename;
Cwd 'abs_path';
</pre>
=== Configuration ===

== writeDate.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. This is a very simple script, it simply writes the current date and time to a file on the backup server before the rsnapshots rotates the backup. This date is used by the Sepiola Online Backup Client to display the iterations and their dates.
* Pod documentation:
<pre>
NAME
writeDate.pl

DESCRIPTION
This script writes the current date and time into the file for each
backup before the rsnapshots rotates the backup. This information is
used by the Sepiola Online Backup Client.

The configuration file for this script is stored under
/etc/backup-utils/writeDate.conf

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

-C configfile
The configuration file.

-U uid
The user id.

-d debug
Turns the debug mode on.

-h help
This online help.

USAGE
writeDate.pl [-C configuration_file] [-U uid] [-d debug] [-h help]

CREATED
2007-09-16 michael.eichenberger@stepping-stone.ch created

VERSION
2013-09-01 michael.eichenberger@stepping-stone.ch updated to reflect
changes in the new backup environment

2007-09-16 michael.eichenberger@stepping-stone.ch created

INCORPORATED CODE
Incorporate code with use:

warnings;
strict;
Config::IniFiles;
Getopt::Std;
Sys::Syslog;
</pre>

=== Configuration ===
/etc/backup-utils/writeDate.conf
<pre>
[Global]
BACKUP_DIRECTORY = /incoming
BACKUP_BACKUP_TIME_FILE = /.sepiola_backup/backupTime

[Syslog]
SYSLOG = rsnapshot
</pre>

== rsnapshot wrapper bash scripts ==

= Links =
* [http://www.openldap.org/ OpenLDAP], an open source implementation of the Lightweight Directory Access Protocol.
* [http://arthurdejong.org/nss-pam-ldapd/ nss-pam-ldapd], a Name Service Switch (NSS) module that allows your LDAP server to provide user account, group, host name, alias, netgroup, and basically any other information that you would normally get from /etc flat files or NIS.
* [http://www.gentoo.org/doc/de/ldap-howto.xml Gentoo Leitfaden zur OpenLDAP Authentifikation].
* [http://wiki.gentoo.org/wiki/Centralized_authentication_using_OpenLDAP Centralized authentication using OpenLDAP].
* [https://code.google.com/p/openssh-lpk/source/browse/trunk/schemas/openssh-lpk_openldap.schema openssh-lpk_openldap.schema] OpenSSH LDAP Public Keys.
* [http://sourceforge.net/projects/linuxquota/ linuxquota] Linux DiskQuota.
* [http://www.rsnapshot.org/ rsnapshot], a remote filesystem snapshot utility, based on rsync.
* [http://olivier.sessink.nl/jailkit/ Jailkit], set of utilities to limit user accounts to specific files using chroot() and or specific commands. Also includes a tool to build a chroot environment.
* [http://www.busybox.net/ Busybox] BusyBox combines tiny versions of many common UNIX utilities into a single small executable. Useful to reduce the number of files (and thus the complexity) when building a chroot.

[[Category:stoney backup]]

stoney backup: Server set-up

2014-06-13T13:41:38Z

David: /* Cronjobs */

= Abstract =
This document describes server setup for the stoney cloud (Online) Backup service, built upon the [http://www.gentoo.org/ Gentoo] Linux distribution.

= Overview =
After working through this documentation, you will be able to set up and configure your own (Online) Backup service server.

= Software Installation =

== Requirements ==
A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].

== Keywords & USE-Flags ==
For a minimal OpenLDAP directory installation:
echo "net-nds/openldap minimal sasl" >> /etc/portage/package.use
echo "net-nds/openldap ~amd64" >> /etc/portage/package.keywords

NSS and PAM modules for lookups using LDAP:
echo "sys-auth/nss-pam-ldapd sasl" >> /etc/portage/package.use
echo "sys-auth/nss-pam-ldapd ~amd64" >> /etc/portage/package.keywords
echo "sys-fs/quota ldap" >> /etc/portage/package.use

echo "=app-admin/jailkit-2.16 ~amd64" >> /etc/portage/package.keywords

For the prov-backup-rsnapshot daemon:
echo "dev-perl/Net-SMTPS ~amd64" >> /etc/portage/package.keywords
echo "perl-core/Switch ~amd64" >> /etc/portage/package.keywords

To build puttygen only without X11:
echo "net-misc/putty ~amd64" >> /etc/portage/package.keywords
echo "net-misc/putty -gtk" >> /etc/portage/package.use

== Emerge ==
emerge -va nss-pam-ldapd \
quota \
net-misc/putty \
app-admin/jailkit \
sys-apps/haveged \
net-misc/putty \
sys-apps/sst-backup-utils \
sys-apps/sst-prov-backup-rsnapshot

To list the dependencies of ebuilds, you can use <code>equery</code>:
equery depgraph sst-backup-utils
<pre>
* Searching for sst-backup-utils ...

* dependency graph for sys-apps/sst-backup-utils-0.1.0
`-- sys-apps/sst-backup-utils-0.1.0 amd64
`-- dev-perl/PerlUtil-0.1.0 (>=dev-perl/PerlUtil-0.1.0) amd64
`-- virtual/perl-Sys-Syslog-0.320.0 (virtual/perl-Sys-Syslog) amd64
`-- dev-perl/perl-ldap-0.530.0 (dev-perl/perl-ldap) amd64
`-- dev-perl/XML-Simple-2.200.0 (dev-perl/XML-Simple) amd64
`-- dev-perl/Config-IniFiles-2.780.0 (dev-perl/Config-IniFiles) amd64
`-- dev-perl/XML-Validator-Schema-1.100.0 (dev-perl/XML-Validator-Schema) amd64
`-- dev-perl/Date-Calc-6.300.0 (dev-perl/Date-Calc) amd64
`-- dev-perl/DateManip-6.310.0 (dev-perl/DateManip) amd64
`-- dev-perl/Schedule-Cron-Events-1.930.0 (dev-perl/Schedule-Cron-Events) amd64
`-- dev-perl/DateTime-Format-Strptime-1.520.0 (dev-perl/DateTime-Format-Strptime) amd64
`-- dev-perl/XML-SAX-0.990.0 (dev-perl/XML-SAX) amd64
`-- virtual/perl-MIME-Base64-3.130.0-r2 (virtual/perl-MIME-Base64) amd64
`-- dev-perl/Authen-SASL-2.160.0 (dev-perl/Authen-SASL) amd64
`-- dev-perl/Net-SMTPS-0.30.0 (dev-perl/Net-SMTPS) ~amd64
`-- dev-perl/text-template-1.450.0 (dev-perl/text-template) amd64
`-- virtual/perl-Getopt-Long-2.380.0-r2 (virtual/perl-Getopt-Long) amd64
`-- dev-perl/Parallel-ForkManager-1.20.0 (dev-perl/Parallel-ForkManager) amd64
`-- dev-perl/Time-Stopwatch-1.0.0 (dev-perl/Time-Stopwatch) amd64
`-- app-backup/rsnapshot-1.3.1-r1 (app-backup/rsnapshot) amd64
[ sys-apps/sst-backup-utils-0.1.0 stats: packages (20), max depth (1) ]
</pre>

For more information, visit the [http://www.gentoo.org/doc/en/gentoolkit.xml Gentoolkit] page.

= Base Server Software Configuration =
== OpenSSH ==
=== OpenSSH Configuration ===
Configure the OpenSSH daemon:
<source lang="bash">
vi /etc/ssh/sshd_config
</source>

Set following options:
<source lang="bash">
PubkeyAuthentication yes
PasswordAuthentication yes
UsePAM yes
Subsystem sftp internal-sftp
</source>

Make sure, that <code>Subsystem sftp internal-sftp</code> is the last line in the configuration file.

We want to reduce the numbers of chroot environments in one folder. As the <code>ChrootDirectory</code> configuration option only allows <code>%h</code> (home directory of the user) and <code>%u</code> (username of the user), we need to create the necessary matching rules in the form of:
<source lang="bash">
Match User *000
ChrootDirectory /var/backup/000/%u
AuthorizedKeysFile /var/backup/000/%u/%h/.ssh/authorized_keys
Match
Match User *001
ChrootDirectory /var/backup/001/%u
AuthorizedKeysFile /var/backup/001/%u/%h/.ssh/authorized_keys
Match
...
Match User *999
ChrootDirectory /var/backup/999/%u
AuthorizedKeysFile /var/backup/999/%u/%h/.ssh/authorized_keys
Match
</source>

The creation of the matching rules is done by executing the following bash commands:
<source lang="bash">
FILE=/etc/ssh/sshd_config;

for x in {0..999} ; do \
printf "Match User *%03d\n" $x >> ${FILE}; \
printf " ChrootDirectory /var/backup/%03d/%%u\n" $x >> ${FILE}; \
printf " AuthorizedKeysFile /var/backup/%03d/%%u/%%h/.ssh/authorized_keys\n" $x >> ${FILE}; \
printf "Match\n" >> ${FILE}; \
done
</source>

Don't forget to restart the OpenSSH daemon:
<source lang="bash">
/etc/init.d/sshd restart
</source>

=== OpenSSH Host Keys ===
If you migrate from a existing backup server, you might want to copy the ssh host keys to the new server. If you do so clients want see a difference between the two hosts as the fingerprint remains the same. Copy the following files from the existing host to the new:
* /etc/ssh/ssh_host_dsa_key
* /etc/ssh/ssh_host_ecdsa_key
* /etc/ssh/ssh_host_key
* /etc/ssh/ssh_host_rsa_key
* /etc/ssh/ssh_host_dsa_key.pub
* /etc/ssh/ssh_host_ecdsa_key.pub
* /etc/ssh/ssh_host_key.pub
* /etc/ssh/ssh_host_rsa_key.pub

Set the correct permissions on the new host:
chmod 600 /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_ecdsa_key /etc/ssh/ssh_host_key /etc/ssh/ssh_host_rsa_key
chmod 644 /etc/ssh/*.pub

And restart the ssh daemon. ''Caution'': do not close your existing ssh session as long as you are not sure the ssh daemon has restarted properly and you can login again.
/etc/init.d/sshd restart

== OpenLDAP ==
=== /etc/hosts ===
Update the <code>/etc/hosts</code> with the LDAP server:
/etc/hosts

# VIP of the LDAP Server
31.216.40.4 ldapm.stoney-cloud.org

=== Root CA Certificate Installation ===
Install the root CA certificate into the OpenSSL default certificate storage directory:
fqdn="cloud.stoney-cloud.org" # The fully qualified domain name of the server containing the root certificate.

cd /etc/ssl/certs/
wget --no-check-certificate https://${fqdn}/ca/FOSS-Cloud_CA.cert.pem
chown root:root /etc/ssl/certs/FOSS-Cloud_CA.cert.pem
chmod 444 /etc/ssl/certs/FOSS-Cloud_CA.cert.pem

Rebuild the CA hashes
c_rehash /etc/ssl/certs/

=== /etc/openldap/ldap.conf ===
Update the <code>/etc/openldap/ldap.conf</code>LDAP configuration file/environment variables:
/etc/openldap/ldap.conf

<pre>
# Used to specify a size limit to use when performing searches. The number should be an
# non-negative integer. SIZELIMIT of zero (0) specifies unlimited search size.
SIZELIMIT 20000

# Used to specify a time limit to use when performing searches. The number should be an
# non-negative integer. TIMELIMIT of zero (0) specifies unlimited search time to be used.
TIMELIMIT 45

# Specify how aliases dereferencing is done. DEREF should be set to one of never, always, search,
# or find to specify that aliases are never dereferenced, always dereferenced, dereferenced when
# searching, or dereferenced only when locating the base object for the search. The default is to
# never dereference aliases.
DEREF never

# Specifies the URI(s) of an LDAP server(s) to which the LDAP library should connect. The URI
# scheme may be either ldapor ldaps which refer to LDAP over TCP and LDAP over SSL (TLS)
# respectively. Each server's name can be specified as a domain- style name or an IP address
# literal. Optionally, the server's name can followed by a ':' and the port number the LDAP
# server is listening on. If no port number is provided, the default port for the scheme is
# used (389 for ldap://, 636 for ldaps://). A space separated list of URIs may be provided.
URI ldaps://ldapm.stoney-cloud.org

# Used to specify the default base DN to use when performing ldap operations. The base must be
# specified as a Distinguished Name in LDAP format.
BASE dc=stoney-cloud,dc=org

# This is a local copy of the certificate of the certificate authority
# used to sign the server certificate for the LDAP server I am using
TLS_CACERT /etc/ssl/certs/FOSS-Cloud_CA.cert.pem
</pre>

Check you configuration by doing a search:
ldapsearch -v -H "ldaps://ldapm.stoney-cloud.org" \
-b "dc=stoney-cloud,dc=org" \
-D "cn=Manager,dc=stoney-cloud,dc=org" \
-s one "(objectClass=*)" \
-LLL -W

The result should look something like:
ldap_initialize( ldaps://ldapm.stoney-cloud.org:636/??base )
filter: (objectClass=*)
requesting: All userApplication attributes
dn: ou=administration,dc=stoney-cloud,dc=org
objectClass: top
objectClass: organizationalUnit
ou: administration
...

== Random Number Generator (haveged) ==
Tools like putty are dependent on random numbers to be able to create certificates.

=== haveged - Generate random numbers and feed linux random device ===
The haveged daemon doesn't need any special configuration, therefore you can start it from the command line interface:
/etc/init.d/haveged start

Check, if the start was successful:
ps auxf | grep haveged

root 18001 1.0 0.0 7420 3616 ? Ss 08:48 0:00 /usr/sbin/haveged -r 0 -w 1024 -v 1

Add the haveged daemon to the default run level:
rc-update add haveged default

== nss-pam-ldapd ==
=== nslcd.conf — configuration file for LDAP nameservice daemon ===
/etc/nslcd.conf

<pre>
# This is the configuration file for the LDAP nameservice
# switch library's nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The uri pointing to the LDAP server to use for name lookups.
# Multiple entries may be specified. The address that is used
# here should be resolvable without using LDAP (obviously).
#uri ldap://127.0.0.1/
#uri ldaps://127.0.0.1/
#uri ldapi://%2fvar%2frun%2fldapi_sock/
# Note: %2f encodes the '/' used as directory separator
uri ldaps://ldapm.tombstone.ch

# The LDAP version to use (defaults to 3
# if supported by client library)
#ldap_version 3

# The distinguished name of the search base.
base dc=stoney-cloud,dc=org

# The distinguished name to bind to the server with.
# Optional: default is to bind anonymously.
binddn cn=Manager,dc=stoney-cloud,dc=org

# The credentials to bind with.
# Optional: default is no credentials.
# Note that if you set a bindpw you should check the permissions of this file.
bindpw myverysecretpassword

# The distinguished name to perform password modifications by root by.
#rootpwmoddn cn=admin,dc=example,dc=com

# The default search scope.
#scope sub
#scope one
#scope base

# Customize certain database lookups.
#base group ou=Groups,dc=example,dc=com
base group ou=groups,ou=backup,ou=services,dc=stoney-cloud,dc=org
base passwd ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
base shadow ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
#scope group onelevel
#scope hosts sub

#filter group (&(objectClass=posixGroup)(sstIsActive=TRUE))
filter passwd (&(objectClass=posixAccount)(sstIsActive=TRUE))
filter shadow (&(objectClass=shadowAccount)(sstIsActive=TRUE))

# Bind/connect timelimit.
#bind_timelimit 30

# Search timelimit.
#timelimit 30

# Idle timelimit. nslcd will close connections if the
# server has not been contacted for the number of seconds.
#idle_timelimit 3600

# Use StartTLS without verifying the server certificate.
#ssl start_tls
tls_reqcert never

# CA certificates for server certificate verification
#tls_cacertdir /etc/ssl/certs
#tls_cacertfile /etc/ssl/ca.cert

# Seed the PRNG if /dev/urandom is not provided
#tls_randfile /var/run/egd-pool

# SSL cipher suite
# See man ciphers for syntax
#tls_ciphers TLSv1

# Client certificate and key
# Use these, if your server requires client authentication.
#tls_cert
#tls_key

# Mappings for Services for UNIX 3.5
#filter passwd (objectClass=User)
#map passwd uid msSFU30Name
#map passwd userPassword msSFU30Password
#map passwd homeDirectory msSFU30HomeDirectory
#map passwd homeDirectory msSFUHomeDirectory
#filter shadow (objectClass=User)
#map shadow uid msSFU30Name
#map shadow userPassword msSFU30Password
#filter group (objectClass=Group)
#map group member msSFU30PosixMember

# Mappings for Services for UNIX 2.0
#filter passwd (objectClass=User)
#map passwd uid msSFUName
#map passwd userPassword msSFUPassword
#map passwd homeDirectory msSFUHomeDirectory
#map passwd gecos msSFUName
#filter shadow (objectClass=User)
#map shadow uid msSFUName
#map shadow userPassword msSFUPassword
#map shadow shadowLastChange pwdLastSet
#filter group (objectClass=Group)
#map group member posixMember

# Mappings for Active Directory
#pagesize 1000
#referrals off
#idle_timelimit 800
#filter passwd (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map passwd uid sAMAccountName
#map passwd homeDirectory unixHomeDirectory
#map passwd gecos displayName
#filter shadow (&(objectClass=user)(!(objectClass=computer))(uidNumber=*)(unixHomeDirectory=*))
#map shadow uid sAMAccountName
#map shadow shadowLastChange pwdLastSet
#filter group (objectClass=group)

# Alternative mappings for Active Directory
# (replace the SIDs in the objectSid mappings with the value for your domain)
#pagesize 1000
#referrals off
#idle_timelimit 800
#filter passwd (&(objectClass=user)(objectClass=person)(!(objectClass=computer)))
#map passwd uid cn
#map passwd uidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820
#map passwd gidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820
#map passwd homeDirectory "/home/$cn"
#map passwd gecos displayName
#map passwd loginShell "/bin/bash"
#filter group (|(objectClass=group)(objectClass=person))
#map group gidNumber objectSid:S-1-5-21-3623811015-3361044348-30300820

# Mappings for AIX SecureWay
#filter passwd (objectClass=aixAccount)
#map passwd uid userName
#map passwd userPassword passwordChar
#map passwd uidNumber uid
#map passwd gidNumber gid
#filter group (objectClass=aixAccessGroup)
#map group cn groupName
#map group gidNumber gid
</pre>

=== nsswitch.conf - Name Service Switch configuration file ===
/etc/nsswitch.conf

<pre>
passwd: files ldap
shadow: files ldap
group: files ldap

# passwd: db files nis
# shadow: db files nis
# group: db files nis

hosts: files dns
networks: files dns

services: db files
protocols: db files
rpc: db files
ethers: db files
netmasks: files
netgroup: files
bootparams: files

automount: files
aliases: files
</pre>

=== system-auth ===
vi /etc/pam.d/system-auth

<pre>
auth required pam_env.so
auth sufficient pam_unix.so try_first_pass likeauth nullok
auth sufficient pam_ldap.so minimum_uid=1000 use_first_pass
auth required pam_deny.so

account required pam_unix.so
account sufficient pam_ldap.so minimum_uid=1000 use_first_pass

password required pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
password required pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password sufficient pam_ldap.so minimum_uid=1000 use_first_pass
password required pam_deny.so

session required pam_limits.so
session required pam_env.so
session required pam_unix.so
session sufficient pam_ldap.so minimum_uid=1000 use_first_pass
</pre>

=== Test the Setup ===
nslcd -d

=== Update the Default Run Levels ===
rc-update add nslcd default
rc-update add nscd default

=== Start the necessary Daemons ===
/etc/init.d/nslcd start
/etc/init.d/nscd start

== Quota ==
=== 32-bit Project Identifier Support ===
We need to enable 32-bit project identifier support (PROJID32BIT feature) for our naming scheme (uid numbers larger than 65'536), which is already the default on the stepping stone virtual machines:
mkfs.xfs '''-i projid32bit=1''' /dev/vg-local-01/var

=== Update /etc/fstab and Mount ===
Make sure, that you have user quota (uqota) and project quota (pquota) set as options on the chosen mount point in /etc/fstab. For example:
LABEL=LV-VAR /var xfs noatime,discard,inode64,uquota,pquota 0 2

reboot

Check, if everything went ok:
df -h | grep var

/dev/mapper/vg--local--01-var 1023G 220G 804G 22% /var

=== Verify ===
Some important options for xfs_quota:
* -x: Enable expert mode.
* -c: Pass arguments on the command line. Multiple arguments may be given.

Remount the file system /var and check, if /var has the desired values:
xfs_quota -x -c state /var

As you can see (items marked bold), we have achieved our goal:
User quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: '''ON'''
Enforcement: '''ON'''
Inode: #131 (1 blocks, 1 extents)
Group quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: OFF
Enforcement: OFF
Inode: #132 (1 blocks, 1 extents)
Project quota state on /var (/dev/mapper/vg--local--01-var)
Accounting: '''ON'''
Enforcement: '''ON'''
Inode: #132 (1 blocks, 1 extents)
Blocks grace time: [7 days 00:00:30]
Inodes grace time: [7 days 00:00:30]
Realtime Blocks grace time: [7 days 00:00:30]

=== User Quotas ===
==== Adding a User Quota ====
Set a quota of 1 Gigabyte for the user 4000187 (the values are in kilobytes, so 1048576 kilobyte are 1024 megabytes which corresponds to 1 gigabyte):
xfs_quota -x -c 'limit bhard=1048576k 4000187' /var

Or in bytes:
xfs_quota -x -c 'limit bhard=1073741824 4000187' /var

Read the quota information for the user 4000187:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If the user has data in the project, that belongs to him, the result will change:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

==== Modifiying a User Quota ====
To modify a users quota, you just set a new quota (limit):
xfs_quota -x -c 'limit bhard=1048576k 4000187' /var

Read the quota information for the user 4000187:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If the user has data in the project, that belongs to him, the result will change:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

==== Removing a User Quota ====
Removing a quota for a user:
xfs_quota -x -c 'limit bhard=0 4000187' /var

The following command should give you an empty result:
xfs_quota -x -c 'quota -v -N -u 4000187' /var

=== Project (Directory) Quotas ===
==== Adding a Project (Directory) Quota ====
The XFS file system additionally allows you to set quotas on individual directory hierarchies in the file system that are known as managed trees. Each managed tree is uniquely identified by a project ID and an optional project name. We'll use the following values in the examples:
* project_ID: The uid of the online backup account (4000187).
* project_name: The uid of the online backup account (4000187). This could be a human readable name.
* mountpoint: The mountpoint of the xfs-filesystem (/var). See the <code>/etc/fstab</code> entry from above.
* directory: The directory of the project (187/4000187), starting from the mountpoint of the xfs-filesystem (/var).

Define a unique project ID for the directory hierarchy in the <code>/etc/projects</code> file (project_ID:mountpoint/directory):
echo "4000187:/var/backup/187/4000187/home/4000187" >> /etc/projects

Create an entry in the <code>/etc/projid</code> file that maps a project name to the project ID (project_name:project_ID):
echo "4000187:4000187" >> /etc/projid

Set Project:
xfs_quota -x -c 'project -s -p /var/backup/187/4000187/home/4000187 4000187' /var

Set Quota (limit) on Project:
xfs_quota -x -c 'limit -p bhard=1048576k 4000187' /var

Check your Quota (limit)
xfs_quota -x -c 'quota -p 4000187' /var

Check the Quota:
* <code>-v</code>: increase verbosity in reporting (also dumps zero values).
* <code>-N</code>: suppress the initial header.
* <code>-p</code>: display project quota information.
* <code>-h</code>: human readable format.
xfs_quota -x -c 'quota -v -N -p 4000187' /var

/dev/mapper/vg--local--01-var 0 0 1048576 00 [--------] /var

If you copied data into the project, the output will look something like:
/dev/mapper/vg--local--01-var 512000 0 1048576 00 [--------] /var

To give you an overall view of the whole system:
xfs_quota -x -c report /var

<pre>
User quota on /var (/dev/mapper/vg--local--01-var)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 1024000 0 0 00 [--------]
4000187 0 0 1048576 00 [--------]

Project quota on /var (/dev/mapper/vg--local--01-var)
Blocks
Project ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
4000187 512000 0 1048576 00 [--------]
</pre>

==== Modifying a Project (Directory) Quota ====
To modify a project (directory) quota, you just set an new quota (limit) on the chosen project:
xfs_quota -x -c 'limit -p bhard=1048576k 4000187' /var

Check your quota (limit)
xfs_quota -x -c 'quota -p 4000187' /var

==== Removing a Project (Directory) Quota ====
Removing a quota from a project:
xfs_quota -x -c 'limit -p bhard=0 4000187' /var

Chreck the results:
xfs_quota -x -c report /var

<pre>
User quota on /var (/dev/mapper/vg--local--01-var)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 512000 0 0 00 [--------]
4000187 0 0 1024 00 [--------]
</pre>

As you can see, the line with the Project ID 4000187 has disappeared:
4000187 512000 0 1048576 00 [--------]

Don't forget to remove the project from <code>/etc/projects</code> and <code>/etc/projid</code>:
sed -i -e '/4000187/d' /etc/projects
sed -i -e '/4000187/d' /etc/projid

=== Some important notes concerning XFS ===
# The '''quotacheck''' command has no effect on XFS filesystems. The first time quota accounting is turned on (at mount time), XFS does an automatic quotacheck internally; afterwards, the quota system will always be completely consistent until quotas are manually turned off.
# There is '''no need for quota file(s)''' in the root of the XFS filesystem.

== prov-backup-rsnapshot ==
Install the prov-backup-rsnasphot daemon script using the package manager:
<pre>
emerge -va sys-apps/sst-prov-backup-rsnapshot
</pre>

=== Configuration ===
If it is the first provisioning module running on this server (very likely) you first have to configure the provisioning daemon (you can skip this step if you have already another provisioning module running on this server)

==== Provisioning global configuration ====
The global configuration for the provisioning daemon applies to all provisioning modules running on the server. This configuration therefore contains information about the provisioning daemon itself and no information at all about the specific modules.
/etc/Provisioning/Global.conf
<pre>
# Copyright (C) 2012 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 0

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

# The number of seconds to wait before retry contacting the backend server during startup.
SLEEP = 10

# Number of backend server connection retries during startup.
ATTEMPTS = 3

[Operation Mode]
# The number of seconds to wait before retry contacting the backend server in case of a service interruptions.
SLEEP = 30

# Number of backend server connection retries in case of a service interruptions.
ATTEMPTS = 3

[Mail]
# Error messages are sent to the mail configured below.
SENDTO = <YOUR-MAIL-ADDRESS>
HOST = mail.stepping-stone.ch
PORT = 587
USERNAME = <YOUR-NOTIFICATION-EMAIL-ADDRESS>
PASSWORD = <PASSWORD>
FROMNAME = Provisioning daemon
CA_DIR = /etc/ssl/certs
SSL = starttls
AUTH_METHOD = LOGIN

# Additionally, you can be informed about creation, modification and deletion of services.
WANTINFOMAIL = 1
</pre>

==== Provisioning daemon prov-backup-rsnapshot module ====
The module specific configuration is located in /etc/Provisioning/<Service>/<Type>.conf. In the case of the prov-backup-rsnapshot module this is <code>/etc/Provisioning/Backup/Rsnapshot.conf</code>. (Note: Comments starting with /* are not in the configuration file, they are only in the wiki to add some additional information)

<pre>
# Copyright (C) 2013 stepping stone GmbH
# Switzerland
# http://www.stepping-stone.ch
# support@stepping-stone.ch
#
# Authors:
# Pat Kläy <pat.klaey@stepping-stone.ch>
#
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

/* If you want, you can override the log information from the global configuration file this might be useful for debugging */
[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

/* Specify the hosts fully qualified domain name. This name will be used to perform some checks and also appear in the information and error mails */
ENVIRONMENT = <FQDN>

[Database]
BACKEND = LDAP
SERVER = ldaps://ldapm.tombstone.org
PORT = 636
ADMIN_USER = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <PASSWORD>
SERVICE_SUBTREE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
COOKIE_FILE = /etc/Provisioning/Backup/rsnapshot.cookie
DEFAULT_COOKIE = rid=001,csn=
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(sstProvisioningState=0))

/* Specifies the service itself. As it is the prov-backup-rsnapshot module, the SERVICE is "Backup" and the TYPE is "Rsnapshot".
* The MODUS is as usual selfcare and the TRANSPORTAPI is LocalCLI. This is because the daemon is running on the same host as the
* backup accounts are provisioned and the commands can be executed on this host using the cli.
* For more information about MODUS and TRANSPORTAPI see https://int.stepping-stone.ch/wiki/provisioning.pl#Service_Konfiguration
*/
[Service]
MODUS = selfcare
TRANSPORTAPI = LocalCLI
SERVICE = Backup
TYPE = Rsnapshot

SYSLOG = prov-backup-rsnapshot

/* For the TRANSPORTAPI LocalCLI there is no gateway required because there is no connection to establish. So set HOST, USER and
* DSA_FILE to whatever you want. Don't leave it blank, otherwise the provisioning daemon would log some error messages saying
* these attributes are empty
*/
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

/* Information about the backup itself (how to setup everything). Note that the %uid% int the RSNAPSHOT_CONFIG_FILE parameter will
* be replaced by the accounts UID. The script CREATE_CHROOT_CMD was installed with the prov-backup-rsnapshot module, so do not
* change this parameter. The quota parameters (SET_QUOTA_CMD, MOUNTPOINT, QUOTA_FILE, PROJECTS_FILE and PROJID_FILE) represent
* the quota setup as described on http://wiki.stoney-cloud.org/index.php/stoney_backup:_Server_set-up#Quota. If you followed this
* manual, you can copy-paste them into your configuration file, otherwise adapt them according to your quota setup.
*/
[Backup]
RSNAPSHOT_CONFIG_FILE = /etc/rsnapshot/rsnapshot.conf.%uid%
SET_QUOTA_CMD = /usr/sbin/xfs_quota
CREATE_CHROOT_CMD = /usr/libexec/createBackupDirectory.sh
MOUNTPOINT = /var
QUOTA_FILE = /etc/backupSize
PROJECTS_FILE = /etc/projects
PROJID_FILE = /etc/projid
</pre>

== backup utils ==
Install the backup utils (multiple scripts which help you to manage and monitor your backup server and backup accounts) using the package manager. For more information about the scripts please see the [[stoney_backup:_Server_set-up#stoney_backup_Service_Software | stoney backup Service Software]] section.
<pre>
emerge -va sys-apps/sst-backup-utils
</pre>
=== Configuration ===
Please refer to the configuration sections for the different scripts in [[stoney_backup:_Server_set-up#stoney_backup_Service_Software | stoney backup Service Software]].

= stoney backup Service Software =
The stoney backup Service comes along with multiple scripts which help you to manage and monitor your backup server and accounts:

We use rsnapshot - remote filesystem snapshot utility for the actual snapshots and a handful of wrapper scripts, that do things like:
* Read the users and their settings from the LDAP directory.
* Execute rsnapshot according to the users settings.
* Write the backup quotas backup (incoming), iterations (.snapshots) and free space to the users local backupSize file and update the LDAP directory.
* Inform the reseller, customer or user (depending on the settings in the LDAP directory) via mail, if the quota limit has been reached.
* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.

== writeAccountSize.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeAccountSize.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* Calculates the used disk space (backup and iterations) for a given account and writes the corresponding values to:
*# The LDAP backend (used by the selfcare webinterface to display quota information):
*#* Backup space used (sstBackupSize): The disk space the account uses for the backup itself (disk space used under the <code>incoming</code> folder of the users chroot-home directory)
*#* Snapshot space used (sstIncrementSize): The disk space the account uses for the iterations (disk space under the <code>.snapshot</code> folder of the users chroot-home directory)
*# The file <code>etc/backupSize</code> of the accounts chroot (used by the Sepiola Online Backup client):
* Checks if the user and/or reseller must be notified that there is no more disk space left for the given account
** Checks if the notification flag was passed, if not no notification will be triggered
** Calculates the used disk space (backup and iterations) in percentage
** Reads the notification threshold value from the LDAP backed
** If the disk space used (in percentage) is bigger than the value retrieved from the LDAP backend start the [[stoney_cloud:_Notification_Architecture | notification]] process with
*** Product: Given account UID
*** Service: Backup
*** Problem: Quota
* Pod documentation:
<pre>
NAME
writeAccountSize.pl

DESCRIPTION
This Script gets quota information from filesystem, size of incoming and
snapshots directories, write the data to a file and the LDAP backend and
sends an e-mail message for each account that is over quota to users
e-mail address (from ldap directory) if notification flag is passed.

The configuration file for this script is stored in the backup-utils
configuration directory (/etc/backup-utils/) and is called
writeAccountSize.conf.

The script needs access to the quota program to get quota information.
The script needs ldap access to get users e-mail address and quota
information. The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

-C configfile
The configuration file.

-U uid
The user id.

-n notification
Start notification process if quota threshold is reached

-d debug
Turns the debug mode on.

-h help
This online help.

USAGE
writeAccountSize.pl -U uid [-C configuration file ] []-d debug] [-h
help] [-n]

CREATED
2009-04-16 michael.rhyner@stepping-stone.ch created

VERSION
2009-04-16 michael.rhyner@stepping-stone.ch created
2009-04-30 michael.rhyner@stepping-stone.ch changed position based quota
output parsing with correctly parsed elements
2009-06-15 michael.rhyner@stepping-stone.ch added over quota check and
sending e-mail
2009-06-16 michael.rhyner@stepping-stone.ch renamed script and make it
more general usable (e.g. for online backup, online storage, ...)
2009-06-17 michael.rhyner@stepping-stone.ch changed mail message to read
from a text file instead from configuration parameter
2009-06-18 michael.rhyner@stepping-stone.ch corrected wrong regex to
weed out the asterisk (*) in getQuotaSize
2009-06-19 michael.rhyner@stepping-stone.ch corrected wrong evaluation
success from subroutines and avoid message output when not in debug mode
2009-06-22 michael.rhyner@stepping-stone.ch getQuotaSize: return
immediately if no quota was set
2009-06-24 michael.rhyner@stepping-stone.ch alert when used certain
percentge of allowed space instead of more than allowed space
2009-06-26 michael.rhyner@stepping-stone.ch values are presented in
Gigabytes within notification message
2009-07-23 michael.rhyner@stepping-stone.ch corrected wrong syslog
severities for errors
2009-07-24 michael.rhyner@stepping-stone.ch made e-mail address
available within message body
2013-08-19 pat.klaey@stepping-stone.ch write quota values also to the
LDAP

INCORPORATED CODE
Incorporate code with use:

warnings;
strict;
Config::IniFiles;
Getopt::Std;
Sys::Syslog;
File::Basename;
Text::Template;
POSIX;
Notification;
PerlUtil::Logging;
PerlUtil::LDAPUtil;
</pre>

=== Configuration ===
/etc/backup-utils/writeAccountSize.conf
<pre>
[Global]
INCOMING_DIRECTORY = /incoming
ACCOUNT_SIZE_FILE = /etc/backupSize
SNAPSHOTS = 1

[Syslog]
SYSLOG = rsnapshot

[Directory]
LDAP_SERVER = ldaps://ldapm.tombstone.ch
LDAP_PORT = 636
LDAP_BIND_DN = cn=Manager,dc=stoney-cloud,dc=org
LDAP_BIND_PW = <password>
LDAP_BASE_DN = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
LDAP_PERSON_BASE = ou=people,dc=stoney-cloud,dc=org
LDAP_RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=org
LDAP_EMAIL_ATTRIBUTE = mail

[Notification]
EMAIL_SENDER = stepping stone GmbH Supprt <support@stepping-stone.ch>
EMAIL_ALERT_THRESHOLD = 85

Salutation_Default_de-CH = Liebe Kundin / Lieber Kunde
Salutation_m_de-CH = Sehr geehrter Herr
Salutation_f_de-CH = Sehr geehrte Frau
Salutation_Default_en-GB = Dear customer
Salutation_m_en-GB = Dear Mr.
Salutation_f_en-GB = Dear Mrs.
</pre>

=== Tests ===
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d
<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 0 kilobytes
Incoming Size: 0 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 0
Debug sub getSnapshotsSize: $localSnapshotsSize: 0
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 0 0 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 0

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0

Alert Threshold: 85 %
Calculated value: 0
</pre>

Now write some data (200 megaytes in this example) into the users incoming directory and then execute the script again:
dd if=/dev/zero of=/var/backup/080/4000080/home/4000080/incoming/test.zeros bs=1024k count=200
chown 4000080:4000080 /var/backup/080/4000080/home/4000080/incoming/test.zeros
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d

<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 204800 kilobytes
Incoming Size: 204800 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 204800
Debug sub getSnapshotsSize: $localSnapshotsSize: 0
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 200 0 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0

Alert Threshold: 85 %
Calculated value: 19.53125
</pre>
Everything seems to be working fine!

== snapshot.pl ==
This script is installed to <code>/usr/libexec/backup-utils/snapshot.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* Read interval parameter value passed
** The interval parameter value can be daily, weekly, monthly (or yearly)
* Find all backup active accounts for which the rsnapshot command must be executed (depending on the given interval)
** Filter to find these accounts: <code>(&(sstBackupInterval<INTERVAL>=*)(sstIsActive=TRUE))</code> for example for the daily rsnapshot the filter is <code>(&(sstBackupIntervalDaily=*)(sstIsActive=TRUE))</code>
*** In other words this means: Get me all acounts that have for <code>sstBackupInterval<INTERVAL></code> a value defined AND <code>sstIsActive</code> is set to "TRUE"
* According to the interval given and the account UID calculate the rsnapshot command for all these accounts
** For example
*** Account UID: 4000000
*** Interval: daily
*** Resulting rsnapshot command: /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.4000000 daily (if you use the [[#snapshot.pl_Configuration | configuration]] below)
* Execute all these commands
** Use controlled parallel execution, you can specify how many commands can be executed in parallel (see [[#snapshot.pl_Configuration | configuration]] below)
* Pod documentation:
<pre>
NAME
snapshot.pl

DESCRIPTION
This script gets all active online backup accounts from the LDAP backend
for which the rsnapshot process for the given interval must be executed.
According to these accounts and the given interval, the commands to be
executed are generated and finally executed. The commands can be
executed in parallel, however there is a limit defined in the
configuration file which limits the amount of parallel running
processes.

USAGE
./snapshot.pl --interval interval [--debug] [--help]

OPTIONS
--interval/-i interval
Specifies the rsnapshot interval (can be hourly, daily, weekly,
monthly or yearly)

--debug/-d
Turns on debug mode

--help/-h
Shows this help

CREATED
2012-03-19 pat.klaey@stepping-stone.ch created

VERSION
2012-03-19 pat.klaey@stepping-stone.ch created

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
Sys::Syslog;
PerlUtil::Logging;
PerlUtil::LDAPUtil;
File::Basename;
Parallel::ForkManager;
Time::Stopwatch;
</pre>

=== Configuration ===
There are two things to do in this step:
* Configure the rsnapshot root directory
* Configure the snapshot.pl script itself

==== rsnaphot configuration directory ====
The users individual rsnapshot configurations are stored under <code>/etc/rsnapshot</code>. Please make sure, that the directory exists:
ls -al /etc | grep rsnapshot

drwx------ 2 root root 64 30. Aug 20:20 rsnapshot

If not, create it:
mkdir /etc/rsnapshot
chmod 700 /etc/rsnapshot

==== snapshot.pl Configuration ====
The snapshot.pl script is responsible for the execution of rsnapshot according to the users settings.
/etc/backup-utils/snapshot.conf
<pre>
[General]
MaxParallelProcesses = 5
Rsnapshot_command = /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.%uid% %interval%

[LDAP]
Host = ldaps://ldapm.tombstone.ch
Port = 636
User = cn=Manager,dc=stoney-cloud,dc=org
Password = <Password>
CA_Path = /etc/ssl/certs
Accounts_Base = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
</pre>

Legend: At runtime the following placeholders are replaced as follows
* '''%uid%''': The backup account and login uid as a numeric number. For example: 4000205.
* '''%interval%''': The backup level to be executed. Possible values are hourly, daily, weekly, monthly and yearly.

=== Tests ===
Before adding the necessary cronjob entries, we need to make sure, that we've configured the snapshot.pl script correctly:
/usr/libexec/backup-utils/snapshot.pl --interval daily -d

If everything worked as planned, you should receive feedback looking roughly like:
<pre>
INFO: Starting rsnapshot for interval daily with maximum 5 parallel processes

INFO: Executing snapshot for 4000080

INFO: Executing snapshot for 4000079

INFO: Snapshot process for 4000079 finished in 0.18 seconds with status 0

INFO: Snapshot process for 4000080 finished in 0.19 seconds with status 0

INFO: rsnapshot for all backups done. Took 0.24 seconds
</pre>

Just to make sure, that everything did work out fine, execute <code>writeAccountSize.pl</code> again:
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d
<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 409600 kilobytes
Incoming Size: 204800 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 409600
Debug sub getSnapshotsSize: $localSnapshotsSize: 204800
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 200 200 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 209715200

Alert Threshold: 85 %
Calculated value: 39.0625
</pre>

As you can see, the total used space has risen to 39.0625.

=== Cronjobs ===
After making sure, that everything worked as planned, you can update your crontab entry:
crontab -e
<pre>
...
# Rsnapshot for all users
30 22 * * * /usr/libexec/backup-utils/snapshot.pl --interval daily
15 22 * * sun /usr/libexec/backup-utils/snapshot.pl --interval weekly
00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly
...
</pre>

* TBD: Maybe this is not optimal if there is a lot of data to rotate. In this case, it might be that for example weekly and daily snapshot both run at the same time what might lead to strange results.
* Workarounds:
** Simple/short version: Instead one could create another wrapper script which is called everyday and does some simple logic:
** Is today the first day of a month?
*** Yes: Is today sunday?
**** Yes: Execute monthly then weekly then daily rsnapshots using the snapshot.pl script (but wait for each interval to finish before starting the next)
**** No: Execute monthly then daily rsnapshots using the snapshot.pl script (but wait for monthly interval to finish before starting the daily)
*** No: Is today sunday?
**** Yes: Execute weekly then daily rsnapshots using the snapshot.pl script (but wait for weekly interval to finish before starting the daily)
**** No: Execute daily rsnapshots using the snapshot.pl script
** Complex/long version: Adapt the snapshot.pl script and call it every day without interval parameter. The script does the logic described above.
** Complex/short version for weekly/daily: Always run the daily after the weekly and set the cronjoby to mon-sat and sun
This avoids the above mentioned collision

== scheduleWarning.pl ==
This script is installed to <code>/usr/libexec/backup-utils/scheduleWarning.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* The basic task of this script is simple: For the given account
** Check if the planned backups were started
*** If not, start the [[stoney_cloud:_Notification_Architecture | notification]] process with
**** Product: Given account UID
**** Service: Backup
**** Problem: Schedule
** Check if the planned backups finished successfully
*** If not, start the [[stoney_cloud:_Notification_Architecture | notification]] process with
**** Product: Given account UID
**** Service: Backup
**** Problem: Unsuccessful

As the backup clients distributed by stepping stone GmbH upload metadata XML before (scheduling information and start time) and after (end time and backup status) the actual backup, the scheduleWarning.pl script is able to verify whether or not a planned backup has been executed and whether or not the backup was successful.

* Pod documentation
<pre>
NAME
scheduleWarning.pl

DESCRIPTION
This script tests whether a planed backup was successful or not. There
are two different typs of failure.

1. The Backup did not start
The first type of error is that a backup is scheduled for YYYY-MM-DD
at HH:MM, but the backup don't start at this specified time. A
possible reason could be that the computer was shut down.

2.The backup was not successful
The second type of error is that a backup started as scheduled but
did not finish successfully. There are different reasons for this
error.

If a backup wasn't successful the script checks which type of error
occured. It reads the XML files which are stored on the server and
compares the given information. If the error is detected, the script
stats the norification process with the information of the error.

If a user has more than one computer backed-up, the script tests one
computer after the other. The mail(s) sent by the programm also
contain(s) the information which computer is affected.

OPTION
-U uid
The -U option is required to run the script, it indicates for which
uid the script is executed.

USAGE
scheduleWarning.pl [-U user]

CREATED
2010-04-14 created Pat Kläy <pat.klaey@stepping-stone.ch>

VERSION
2010-04-14 v0.01, created pkl
2010-08-24 v0.02, modified pkl
New using Net::SMTP::TLS to send mails

2013-09-13 v0.03, modified pat.klaey@stepping-stone.ch
Changes to use the script with the new backup infrastructure (read
more information from LDAP, use Notification lib to send mails)

USES
strict;
warnings;
XML::Simple;
Config::IniFiles;
XML::Validator::Schema;
Date::Calc qw(:all);
Date::Manip;
Schedule::Cron::Events;
DateTime::Format::Strptime;
Sys::Syslog;
XML::SAX::ParserFactory;
Getopt::Std;
MIME::Base64;
Authen::SASL;
Net::LDAPS;
Net::SMTP::TLS;
Cwd 'abs_path';
PerlUtil::Logging;
PerlUtil::LDAPUtil;
</pre>

=== Configuration ===
/etc/backup-utils/scheduleWarning.conf
<pre>
[Backup]
CHROOT_DIRECTORY = /var/backup/%lastthree%/%user%

[XML]
SCHEDULE_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/scheduler.xml
SCHEDULE_XSD = /etc/backup-utils/schema/scheduler_schema.xsd
BACKUP_ENDED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupEnded.xml
BACKUP_ENDED_XSD = /etc/backup-utils/schema/backupended_schema.xsd
BACKUP_STARTED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupStarted.xml
BACKUP_STARTED_XSD = /etc/backup-utils/schema/backupstarted_schema.xsd

[TEMPLATE]
Salutation_Default_de-CH = Liebe Kundin / Lieber Kunde
Salutation_m_de-CH = Sehr geehrter Herr
Salutation_f_de-CH = Sehr geehrte Frau
Salutation_Default_en-GB = Dear customer
Salutation_m_en-GB = Dear Mr.
Salutation_f_en-GB = Dear Mrs.

[LDAP]

SERVER = ldaps://ldapm.tombstone.ch
PORT = 636
DEBUG = 1

ADMIN_DN = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <Password>

BACKUP_BASE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
PEOPLE_BASE = ou=people,dc=stoney-cloud,dc=org
RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=org
SCOPE = sub

</pre>

== checkBackups.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. It is a wrapper around the above mentioned <code>writeAccountSize.pl</code> and <code>scheduleWarning.pl</code> scripts (if you configure it accordingly).
* As mentioned the script is a wrapper for calling the quota and scheduler information scripts:
** It gets all active backup accounts from the LDAP directory
** According to the CLI parameters passed it calles for all these accounts:
*** The quota checking script (this script can be defined in the configuration file, see [[stoney_backup:_Server_set-up#Configuration_6 | configuration]] section below)
*** The scheduler checking script (this script can be defined in the configuration file, see [[stoney_backup:_Server_set-up#Configuration_6 | configuration]] section below)
* Pod documentation:
<pre>
NAME
checkBackups.pl

DESCRIPTION
This script processes all active backup accounts and checks (according
to the command line options) the quota and/or the scheduled backups for
the given accounts.

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

OPTIONS
--schedule/-s
Checks schedule information by calling the defined script

--quota/-s
Checks the quota values by calling the defined script

--notify/-n
Passes the notify flag to the called scripts

USAGE
./checkBackups.pl [--quota] [--schedule] [--notify]

./checkBackups.pl --quota
Checks and writes the quota for all active backup accounts. The user
and/or reseller will NOT be informed if any quota threshold is
reached

./checkBackups.pl --quota --notify
Checks and writes the quota for all active backup accounts. The user
and/or reseller will be informed if any quota threshold is reached.

./checkBackups.pl --schedule
Checkes all active backup account if the scheduled backup was
executed and successful. The --schedule option will allways trigger
a notification mail to the user and/or reseller.

./checkBackups.pl --schedule --quota
Checks both, quota and schedule, for all active backup accounts.
Only schedule will trigger notification mails.

./checkBackups.pl --schedule --quota --notify
Checks both, quota and schedule, for all active backup accounts.
Both, quota and schedule, will trigger notification mails.

CREATED
2013-09-17 pat.klaey@stepping-stone.ch created

VERSION
2013-09-17 pat.klaey@stepping-stone.ch created
2013-11-19 pat.klaey@stepping-stone.ch Added options to be able to check
quota, schedule or both in one run

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
Sys::Syslog;
Cwd 'abs_path';
File::Basename;
PerlUtil::LDAPUtil;
PerlUtil::Logging;
</pre>

=== Configuration ===
In the configuration you have to define two simple things:
# Which script to call for quota checks and scheduler checks
# How to access the LDAP backend
/etc/backup-utils/checkBackups.conf
<pre>
[Scripts]
CheckQuotaScript = /usr/libexec/backup-utils/writeAccountSize.pl
CheckScheduleScript = /usr/libexec/backup-utils/scheduleWarning.pl

[LDAP]
Server = ldaps://ldapm.tombstone.ch
Port = 636
Username = cn=Manager,dc=stoney-cloud,dc=org
Password = <PASSWORD>
AccountBase = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
</pre>

==== Cronjobs ====
As you can pass the notification flag to the scripts, you can call the quota script say all 10 minutes. This makes sense if followed the configuration section, as the writeAccountSize.pl script writes the quota values to the LDAP backend which will then be visible in the selfcare webinterface. Like that you have a more or less up to date quota representation in the selfcare webinterface. On the other side, you don't want to send notification mails to the user every 10 minutes. It might be enough to send them once a day, so the cronjobs may look like:
<pre>
# Check backups
15 12 * * * /usr/libexec/backup-utils/checkBackups.pl --schedule --quota --notify

# Write quota values every 10 minutes
*/10 * * * * /usr/libexec/backup-utils/checkBackups.pl --quota
</pre>

== ResellerBackupBilling.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. The pod documentation says all what needs to be said about this script.
* Pod documentation
<pre>
NAME
ResellerBackupBilling.pl

DESCRIPTION
This script processes the reseller with the passed UID and creates a
very simplistic backup-billing overview which then is sent to the
address specified in the configuration file. The billing overview
contains all backup accounts which belong to the given reseller as well
as their quota. According to the billing plan specified in the
configuration file, the script also calculates the price for each backup
account.

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

OPTIONS
--reseller/-r UID
Process the reseller with this UID

--debug/-d
Turn on debug mode

--help/-h
Display this help

USAGE
./ResellerBackupBilling.pl --reseller UID [--debug] [--help]

./ResellerBackupBilling.pl --reseller 2000000
This processes the reseller with the UID 200000 and sends the
billing overview to the address specified in the configuration file

CREATED
2014-01-24 pat.klaey@stepping-stone.ch created

VERSION
2014-01-24 pat.klaey@stepping-stone.ch created

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
PerlUtil::LDAPUtil;
PerlUtil::Logging;
PerlUtil::Mail;
Number::Format;
File::Basename;
Cwd 'abs_path';
</pre>
=== Configuration ===

== writeDate.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. This is a very simple script, it simply writes the current date and time to a file on the backup server before the rsnapshots rotates the backup. This date is used by the Sepiola Online Backup Client to display the iterations and their dates.
* Pod documentation:
<pre>
NAME
writeDate.pl

DESCRIPTION
This script writes the current date and time into the file for each
backup before the rsnapshots rotates the backup. This information is
used by the Sepiola Online Backup Client.

The configuration file for this script is stored under
/etc/backup-utils/writeDate.conf

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

-C configfile
The configuration file.

-U uid
The user id.

-d debug
Turns the debug mode on.

-h help
This online help.

USAGE
writeDate.pl [-C configuration_file] [-U uid] [-d debug] [-h help]

CREATED
2007-09-16 michael.eichenberger@stepping-stone.ch created

VERSION
2013-09-01 michael.eichenberger@stepping-stone.ch updated to reflect
changes in the new backup environment

2007-09-16 michael.eichenberger@stepping-stone.ch created

INCORPORATED CODE
Incorporate code with use:

warnings;
strict;
Config::IniFiles;
Getopt::Std;
Sys::Syslog;
</pre>

=== Configuration ===
/etc/backup-utils/writeDate.conf
<pre>
[Global]
BACKUP_DIRECTORY = /incoming
BACKUP_BACKUP_TIME_FILE = /.sepiola_backup/backupTime

[Syslog]
SYSLOG = rsnapshot
</pre>

== rsnapshot wrapper bash scripts ==

= Links =
* [http://www.openldap.org/ OpenLDAP], an open source implementation of the Lightweight Directory Access Protocol.
* [http://arthurdejong.org/nss-pam-ldapd/ nss-pam-ldapd], a Name Service Switch (NSS) module that allows your LDAP server to provide user account, group, host name, alias, netgroup, and basically any other information that you would normally get from /etc flat files or NIS.
* [http://www.gentoo.org/doc/de/ldap-howto.xml Gentoo Leitfaden zur OpenLDAP Authentifikation].
* [http://wiki.gentoo.org/wiki/Centralized_authentication_using_OpenLDAP Centralized authentication using OpenLDAP].
* [https://code.google.com/p/openssh-lpk/source/browse/trunk/schemas/openssh-lpk_openldap.schema openssh-lpk_openldap.schema] OpenSSH LDAP Public Keys.
* [http://sourceforge.net/projects/linuxquota/ linuxquota] Linux DiskQuota.
* [http://www.rsnapshot.org/ rsnapshot], a remote filesystem snapshot utility, based on rsync.
* [http://olivier.sessink.nl/jailkit/ Jailkit], set of utilities to limit user accounts to specific files using chroot() and or specific commands. Also includes a tool to build a chroot environment.
* [http://www.busybox.net/ Busybox] BusyBox combines tiny versions of many common UNIX utilities into a single small executable. Useful to reduce the number of files (and thus the complexity) when building a chroot.

[[Category:stoney backup]]

Manually adding a new bridge

2014-02-21T09:03:08Z

David: /* Create a new bridging device */

= Overview =
The following page describes the steps to manually add a new Ethernet bridge (<code>vmbrX</code>) to an existing stoney cloud installation. In this example we presume you want to add the new Ethernet bridge <code>vmbr4</code>.

== Before adding a new bridge ==
<pre>
+----------------+----------------+----------------+----------------+
IP interfaces | 192.168.140.1X | 10.1.110.1X | 10.1.120.1X | 10.1.130.1X |
+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+
Bridging interfaces | vmbr0 | (vmbr1) | (vmbr2) | (vmbr3) |
+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+
VLAN interfaces | vlan140 | vlan110 | vlan120 | vlan130 |
+----------------+----------------+----------------+----------------+
+-------------------------------------------------------------------+
Bonding interfaces | bond0 (bonding.mode=802.3ad) |
+-------------------------------------------------------------------+
+----------------+----------------+----------------+----------------+
Physical interfaces | eth0 | eth1 | eth2 | ethN |
+----------------+----------------+----------------+----------------+
</pre>

== After adding an additional bridge (vmbr4) ==
<pre>
+----------------+----------------+----------------+----------------+----------------+
IP interfaces | 192.168.140.1X | 10.1.110.1X | 10.1.120.1X | 10.1.130.1X | 192.168.141.1X |
+----------------+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+----------------+
Bridging interfaces | vmbr0 | (vmbr1) | (vmbr2) | (vmbr3) | vmbr4 |
+----------------+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+----------------+
VLAN interfaces | vlan140 | vlan110 | vlan120 | vlan130 | vlan141 |
+----------------+----------------+----------------+----------------+----------------+
+------------------------------------------------------------------------------------+
Bonding interfaces | bond0 (bonding.mode=802.3ad) |
+------------------------------------------------------------------------------------+
+----------------+----------------+----------------+---------------------------------+
Physical interfaces | eth0 | eth1 | eth2 | ethN |
+----------------+----------------+----------------+---------------------------------+
</pre>

= Prerequisites =
To be able to add and configure a new bridge, you need to have the following informations present:
* The name of the bridge, such as <code>vmbr4</code> (<code>vmbr1</code>, <code>vmbr3</code> and <code>vmbr3</code> are reserved for existing networks)
* VLAN ID, such as <code>141</code>
* IP prefix, such as <code>192.168.141.0/24</code>
* Default gateway address (if any), such as <code>192.168.141.1</code>
* IP address for the DHCP server on the VM node, such as <code>192.168.141.13</code>
* Broadcast address, such as <code>192.168.141.255</code>

Make sure, that you have configured the new VLAN on all involved switches and routers beforehand.

= Configuration =
== Add the new VLAN on top of the bonding interface ==
Login to the first VM node and manually add the new VLAN to the existing bonding interface (<code>bond0</code>). In the following example VLAN <code>141</code> is added:
<source lang="bash">
ip link add link bond0 name vlan141 type vlan id 141
ip link set vlan141 up
ip link show vlan141
</source>
<pre>
71: vlan141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether 38:60:77:9c:98:40 brd ff:ff:ff:ff:ff:ff
</pre>

In order to bring the newly configured VLAN up after a system reboot, the network configuration needs to be extended. Edit <code>/etc/conf.d/net</code> and add the following entries:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# VLAN (802.1q support)

### Add the new vlan to the bond device
vlans_bond0="130 120 110 140 141"

# pub VLAN
vlan140_name="vlan140"
config_vlan140="null"

### Add your new vlan at the end of the VLAN section
# pub2 VLAN
vlan141_name="vlan141"
config_vlan141="null"

# [...]
</source>

== Create a new bridging device ==
As the new VLAN interface is now up, an additional bridge can be added.

Again the network configuration needs to be edited. Append the configuration of the new bridge, below the existing <code>vmbr0</code> bridge configuration. In the following example the bridge <code>vmbr4</code> is added on top of VLAN <code>141</code> with a prefix of <code>192.168.141.13/24</code>:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# Bridging (802.1d) interfaces

bridge_vmbr0="vlan140"
bridge_add_vlan140="vmbr0"
config_vmbr0="192.168.140.13/24 brd 192.168.140.255" # 192.168.140.13 is the ip address of the vm-node.
routes_vmbr0="default via 192.168.140.1"

### Add your new bridge at the end of the bridge section
bridge_vmbr4="vlan141"
bridge_add_vlan141="vmbr4"
config_vmbr4="192.168.141.13/24 brd 192.168.141.255" # 192.168.141.13 is the ip address of the vm-node (only required on primary- and secondary master node for the dhcp server, on all others use: null)
rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.
</source>

To bring the bridge up, an init script is required. Simply symlink the <code>net.lo</code> script to the new bridge name's script, here <code>net.vmbr4</code> is assumed:
<source lang="bash">
ln -s /etc/init.d/net.lo /etc/init.d/net.vmbr4
rc-update add net.vmbr4 default
/etc/init.d/net.vmbr4 start
</source>
<pre>
* Caching service dependencies ... [ ok ]
* Bringing up interface vmbr4
* Creating bridge vmbr4 ...
* Adding ports to vmbr4
* vlan141 ... [ ok ]
* 192.168.141.13/24 ... [ ok ]
</pre>

Check if everything looks fine:
<source lang="bash">
ip addr show vmbr4 && brctl show
</source>
<pre>
82: vmbr4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 38:60:77:9c:98:40 brd ff:ff:ff:ff:ff:ff
inet 192.168.141.13/24 brd 192.168.141.255 scope global vmbr4
valid_lft forever preferred_lft forever
bridge name bridge id STP enabled interfaces
vmbr0 8000.3860779c9840 no kvm-0217_0
vlan140
vmbr4 8000.3860779c9840 no kvm-0227_0
vlan141
</pre>

Or if you would like to copy&paste:
<source lang="bash">
vlanId=141
bridgeName=vmbr4
nodeIp=192.168.141.13
prefixLength=24
broadcast=192.168.141.255

cat << EOF >> /etc/conf.d/net

bridge_${bridgeName}="vlan${vlanId}"
bridge_add_vlan${vlanId}="${bridgeName}"
config_${bridgeName}="${nodeIp}/${prefixLength} brd ${broadcast}"
rc_net_${bridgeName}_provide="!net"
EOF

ln -s /etc/init.d/net.lo /etc/init.d/net.${bridgeName}
rc-update add net.${bridgeName} default
/etc/init.d/net.${bridgeName} start
ip addr show ${bridgeName} && brctl show
</source>

== Delete a bridging device ==
Tear the bridge down but check if it would also stop all network services:
<source lang="bash">
# Set your bridge name
bridgeName=<NAME OF YOUR BRIDGE> # e.g.: vmbr4

# Check if it excludes the net services
grep rc_net_${bridgeName}_provide /etc/conf.d/net
</source>
If grep produces output you might consider running the initscript with the ''-D'' options (otherwise network services, like sshd, might get stopped).

Otherwise just stop the bridge and remove the initscript:
<source lang="bash">
/etc/init.d/net.${bridgeName} stop

rc-update del net.${bridgeName} default

rm /etc/init.d/net.${bridgeName}
</source>

Remove the bridge from the network configuration:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

### Remove the options for the bridge you want to delete
#bridge_vmbr4="vlan141"
#bridge_add_vlan141="vmbr4"
#config_vmbr4="192.168.141.13/24 brd 192.168.141.255" # 192.168.141.13 is the ip address of the vm-node.
#rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.

# [...]
</source>

== Remove a VLAN on a bonding interface ==
Delete the VLAN from the interface bond0:
<source lang="bash">
# Set your vlan id
vlanId=<VLAN ID> # e.g. 141

# Shut the vlan down
ip link set vlan${vlanId} down

# Remove it from the interface bond0
ip link delete vlan${vlanId} type vlan
</source>

Edit <code>/etc/conf.d/net</code> and remove your VLAN:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

### Remove your vlan options
# pub2 VLAN
# vlan141_name="vlan141"
# config_vlan141="null"

# [...]
</source>

= Links =
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap10 Gentoo Handbook - VLAN (802.1q support)]
* [https://wiki.archlinux.org/index.php/VLAN ArchWiki - VLAN]
* [http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge Linux Foundation - Bridge]
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap7 Gentoo Handbook - Bridging (802.1d support)]
* [http://wiki.gentoo.org/wiki/Network_bridge Gentoo Wiki - Network Bridge]

[[Category:Workarounds]]

Manually adding a new bridge

2014-02-21T08:23:40Z

David: /* Add the new VLAN on top of the bonding interface */

= Overview =
The following page describes the steps to manually add a new Ethernet bridge (<code>vmbrX</code>) to an existing stoney cloud installation. In this example we presume you want to add the new Ethernet bridge <code>vmbr4</code>.

== Before adding a new bridge ==
<pre>
+----------------+----------------+----------------+----------------+
IP interfaces | 192.168.140.1X | 10.1.110.1X | 10.1.120.1X | 10.1.130.1X |
+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+
Bridging interfaces | vmbr0 | (vmbr1) | (vmbr2) | (vmbr3) |
+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+
VLAN interfaces | vlan140 | vlan110 | vlan120 | vlan130 |
+----------------+----------------+----------------+----------------+
+-------------------------------------------------------------------+
Bonding interfaces | bond0 (bonding.mode=802.3ad) |
+-------------------------------------------------------------------+
+----------------+----------------+----------------+----------------+
Physical interfaces | eth0 | eth1 | eth2 | ethN |
+----------------+----------------+----------------+----------------+
</pre>

== After adding an additional bridge (vmbr4) ==
<pre>
+----------------+----------------+----------------+----------------+----------------+
IP interfaces | 192.168.140.1X | 10.1.110.1X | 10.1.120.1X | 10.1.130.1X | 192.168.141.1X |
+----------------+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+----------------+
Bridging interfaces | vmbr0 | (vmbr1) | (vmbr2) | (vmbr3) | vmbr4 |
+----------------+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+----------------+
VLAN interfaces | vlan140 | vlan110 | vlan120 | vlan130 | vlan141 |
+----------------+----------------+----------------+----------------+----------------+
+------------------------------------------------------------------------------------+
Bonding interfaces | bond0 (bonding.mode=802.3ad) |
+------------------------------------------------------------------------------------+
+----------------+----------------+----------------+---------------------------------+
Physical interfaces | eth0 | eth1 | eth2 | ethN |
+----------------+----------------+----------------+---------------------------------+
</pre>

= Prerequisites =
To be able to add and configure a new bridge, you need to have the following informations present:
* The name of the bridge, such as <code>vmbr4</code> (<code>vmbr1</code>, <code>vmbr3</code> and <code>vmbr3</code> are reserved for existing networks)
* VLAN ID, such as <code>141</code>
* IP prefix, such as <code>192.168.141.0/24</code>
* Default gateway address (if any), such as <code>192.168.141.1</code>
* IP address for the DHCP server on the VM node, such as <code>192.168.141.13</code>
* Broadcast address, such as <code>192.168.141.255</code>

Make sure, that you have configured the new VLAN on all involved switches and routers beforehand.

= Configuration =
== Add the new VLAN on top of the bonding interface ==
Login to the first VM node and manually add the new VLAN to the existing bonding interface (<code>bond0</code>). In the following example VLAN <code>141</code> is added:
<source lang="bash">
ip link add link bond0 name vlan141 type vlan id 141
ip link set vlan141 up
ip link show vlan141
</source>
<pre>
71: vlan141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether 38:60:77:9c:98:40 brd ff:ff:ff:ff:ff:ff
</pre>

In order to bring the newly configured VLAN up after a system reboot, the network configuration needs to be extended. Edit <code>/etc/conf.d/net</code> and add the following entries:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# VLAN (802.1q support)

### Add the new vlan to the bond device
vlans_bond0="130 120 110 140 141"

# pub VLAN
vlan140_name="vlan140"
config_vlan140="null"

### Add your new vlan at the end of the VLAN section
# pub2 VLAN
vlan141_name="vlan141"
config_vlan141="null"

# [...]
</source>

== Create a new bridging device ==
As the new VLAN interface is now up, an additional bridge can be added.

Again the network configuration needs to be edited. Append the configuration of the new bridge, below the existing <code>vmbr0</code> bridge configuration. In the following example the bridge <code>vmbr4</code> is added on top of VLAN <code>141</code> with a prefix of <code>192.168.141.13/24</code>:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# Bridging (802.1d) interfaces

bridge_vmbr0="vlan140"
bridge_add_vlan140="vmbr0"
config_vmbr0="192.168.140.13/24 brd 192.168.140.255" # 192.168.140.13 is the ip address of the vm-node.
routes_vmbr0="default via 192.168.140.1"

### Add your new bridge at the end of the bridge section
bridge_vmbr4="vlan141"
bridge_add_vlan141="vmbr4"
config_vmbr4="192.168.141.13/24 brd 192.168.141.255" # 192.168.141.13 is the ip address of the vm-node (only required on primary- and secondary master node, on all others use: null)
rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.
</source>

To bring the bridge up, an init script is required. Simply symlink the <code>net.lo</code> script to the new bridge name's script, here <code>net.vmbr4</code> is assumed:
<source lang="bash">
ln -s /etc/init.d/net.lo /etc/init.d/net.vmbr4
rc-update add net.vmbr4 default
/etc/init.d/net.vmbr4 start
</source>
<pre>
* Caching service dependencies ... [ ok ]
* Bringing up interface vmbr4
* Creating bridge vmbr4 ...
* Adding ports to vmbr4
* vlan141 ... [ ok ]
* 192.168.141.13/24 ... [ ok ]
</pre>

Check if everything looks fine:
<source lang="bash">
ip addr show vmbr4 && brctl show
</source>
<pre>
82: vmbr4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 38:60:77:9c:98:40 brd ff:ff:ff:ff:ff:ff
inet 192.168.141.13/24 brd 192.168.141.255 scope global vmbr4
valid_lft forever preferred_lft forever
bridge name bridge id STP enabled interfaces
vmbr0 8000.3860779c9840 no kvm-0217_0
vlan140
vmbr4 8000.3860779c9840 no kvm-0227_0
vlan141
</pre>

Or if you would like to copy&paste:
<source lang="bash">
vlanId=141
bridgeName=vmbr4
nodeIp=192.168.141.13
prefixLength=24
broadcast=192.168.141.255

cat << EOF >> /etc/conf.d/net

bridge_${bridgeName}="vlan${vlanId}"
bridge_add_vlan${vlanId}="${bridgeName}"
config_${bridgeName}="${nodeIp}/${prefixLength} brd ${broadcast}"
rc_net_${bridgeName}_provide="!net"
EOF

ln -s /etc/init.d/net.lo /etc/init.d/net.${bridgeName}
rc-update add net.${bridgeName} default
/etc/init.d/net.${bridgeName} start
ip addr show ${bridgeName} && brctl show
</source>

== Delete a bridging device ==
Tear the bridge down but check if it would also stop all network services:
<source lang="bash">
# Set your bridge name
bridgeName=<NAME OF YOUR BRIDGE> # e.g.: vmbr4

# Check if it excludes the net services
grep rc_net_${bridgeName}_provide /etc/conf.d/net
</source>
If grep produces output you might consider running the initscript with the ''-D'' options (otherwise network services, like sshd, might get stopped).

Otherwise just stop the bridge and remove the initscript:
<source lang="bash">
/etc/init.d/net.${bridgeName} stop

rc-update del net.${bridgeName} default

rm /etc/init.d/net.${bridgeName}
</source>

Remove the bridge from the network configuration:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

### Remove the options for the bridge you want to delete
#bridge_vmbr4="vlan141"
#bridge_add_vlan141="vmbr4"
#config_vmbr4="192.168.141.13/24 brd 192.168.141.255" # 192.168.141.13 is the ip address of the vm-node.
#rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.

# [...]
</source>

== Remove a VLAN on a bonding interface ==
Delete the VLAN from the interface bond0:
<source lang="bash">
# Set your vlan id
vlanId=<VLAN ID> # e.g. 141

# Shut the vlan down
ip link set vlan${vlanId} down

# Remove it from the interface bond0
ip link delete vlan${vlanId} type vlan
</source>

Edit <code>/etc/conf.d/net</code> and remove your VLAN:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

### Remove your vlan options
# pub2 VLAN
# vlan141_name="vlan141"
# config_vlan141="null"

# [...]
</source>

= Links =
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap10 Gentoo Handbook - VLAN (802.1q support)]
* [https://wiki.archlinux.org/index.php/VLAN ArchWiki - VLAN]
* [http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge Linux Foundation - Bridge]
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap7 Gentoo Handbook - Bridging (802.1d support)]
* [http://wiki.gentoo.org/wiki/Network_bridge Gentoo Wiki - Network Bridge]

[[Category:Workarounds]]

Manually adding a new bridge

2014-02-21T08:23:14Z

David: /* Add the new VLAN on top of the bonding interface */

= Overview =
The following page describes the steps to manually add a new Ethernet bridge (<code>vmbrX</code>) to an existing stoney cloud installation. In this example we presume you want to add the new Ethernet bridge <code>vmbr4</code>.

== Before adding a new bridge ==
<pre>
+----------------+----------------+----------------+----------------+
IP interfaces | 192.168.140.1X | 10.1.110.1X | 10.1.120.1X | 10.1.130.1X |
+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+
Bridging interfaces | vmbr0 | (vmbr1) | (vmbr2) | (vmbr3) |
+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+
VLAN interfaces | vlan140 | vlan110 | vlan120 | vlan130 |
+----------------+----------------+----------------+----------------+
+-------------------------------------------------------------------+
Bonding interfaces | bond0 (bonding.mode=802.3ad) |
+-------------------------------------------------------------------+
+----------------+----------------+----------------+----------------+
Physical interfaces | eth0 | eth1 | eth2 | ethN |
+----------------+----------------+----------------+----------------+
</pre>

== After adding an additional bridge (vmbr4) ==
<pre>
+----------------+----------------+----------------+----------------+----------------+
IP interfaces | 192.168.140.1X | 10.1.110.1X | 10.1.120.1X | 10.1.130.1X | 192.168.141.1X |
+----------------+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+----------------+
Bridging interfaces | vmbr0 | (vmbr1) | (vmbr2) | (vmbr3) | vmbr4 |
+----------------+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+----------------+
VLAN interfaces | vlan140 | vlan110 | vlan120 | vlan130 | vlan141 |
+----------------+----------------+----------------+----------------+----------------+
+------------------------------------------------------------------------------------+
Bonding interfaces | bond0 (bonding.mode=802.3ad) |
+------------------------------------------------------------------------------------+
+----------------+----------------+----------------+---------------------------------+
Physical interfaces | eth0 | eth1 | eth2 | ethN |
+----------------+----------------+----------------+---------------------------------+
</pre>

= Prerequisites =
To be able to add and configure a new bridge, you need to have the following informations present:
* The name of the bridge, such as <code>vmbr4</code> (<code>vmbr1</code>, <code>vmbr3</code> and <code>vmbr3</code> are reserved for existing networks)
* VLAN ID, such as <code>141</code>
* IP prefix, such as <code>192.168.141.0/24</code>
* Default gateway address (if any), such as <code>192.168.141.1</code>
* IP address for the DHCP server on the VM node, such as <code>192.168.141.13</code>
* Broadcast address, such as <code>192.168.141.255</code>

Make sure, that you have configured the new VLAN on all involved switches and routers beforehand.

= Configuration =
== Add the new VLAN on top of the bonding interface ==
Login to the first VM node and manually add the new VLAN to the existing bonding interface (<code>bond0</code>). In the following example VLAN <code>141</code> is added:
<source lang="bash">
ip link add link bond0 name vlan141 type vlan id 141
ip link set vlan141 up
ip link show vlan141
</source>
<pre>
71: vlan141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether 38:60:77:9c:98:40 brd ff:ff:ff:ff:ff:ff
</pre>

In order to bring the newly configured VLAN up after a system reboot, the network configuration needs to be extended. Edit <code>/etc/conf.d/net</code> and add the following entry:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# VLAN (802.1q support)

# add the new vlan to the bond device
vlans_bond0="130 120 110 140 141"

# pub VLAN
vlan140_name="vlan140"
config_vlan140="null"

### Add your new vlan at the end of the VLAN section
# pub2 VLAN
vlan141_name="vlan141"
config_vlan141="null"

# [...]
</source>

== Create a new bridging device ==
As the new VLAN interface is now up, an additional bridge can be added.

Again the network configuration needs to be edited. Append the configuration of the new bridge, below the existing <code>vmbr0</code> bridge configuration. In the following example the bridge <code>vmbr4</code> is added on top of VLAN <code>141</code> with a prefix of <code>192.168.141.13/24</code>:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# Bridging (802.1d) interfaces

bridge_vmbr0="vlan140"
bridge_add_vlan140="vmbr0"
config_vmbr0="192.168.140.13/24 brd 192.168.140.255" # 192.168.140.13 is the ip address of the vm-node.
routes_vmbr0="default via 192.168.140.1"

### Add your new bridge at the end of the bridge section
bridge_vmbr4="vlan141"
bridge_add_vlan141="vmbr4"
config_vmbr4="192.168.141.13/24 brd 192.168.141.255" # 192.168.141.13 is the ip address of the vm-node (only required on primary- and secondary master node, on all others use: null)
rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.
</source>

To bring the bridge up, an init script is required. Simply symlink the <code>net.lo</code> script to the new bridge name's script, here <code>net.vmbr4</code> is assumed:
<source lang="bash">
ln -s /etc/init.d/net.lo /etc/init.d/net.vmbr4
rc-update add net.vmbr4 default
/etc/init.d/net.vmbr4 start
</source>
<pre>
* Caching service dependencies ... [ ok ]
* Bringing up interface vmbr4
* Creating bridge vmbr4 ...
* Adding ports to vmbr4
* vlan141 ... [ ok ]
* 192.168.141.13/24 ... [ ok ]
</pre>

Check if everything looks fine:
<source lang="bash">
ip addr show vmbr4 && brctl show
</source>
<pre>
82: vmbr4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 38:60:77:9c:98:40 brd ff:ff:ff:ff:ff:ff
inet 192.168.141.13/24 brd 192.168.141.255 scope global vmbr4
valid_lft forever preferred_lft forever
bridge name bridge id STP enabled interfaces
vmbr0 8000.3860779c9840 no kvm-0217_0
vlan140
vmbr4 8000.3860779c9840 no kvm-0227_0
vlan141
</pre>

Or if you would like to copy&paste:
<source lang="bash">
vlanId=141
bridgeName=vmbr4
nodeIp=192.168.141.13
prefixLength=24
broadcast=192.168.141.255

cat << EOF >> /etc/conf.d/net

bridge_${bridgeName}="vlan${vlanId}"
bridge_add_vlan${vlanId}="${bridgeName}"
config_${bridgeName}="${nodeIp}/${prefixLength} brd ${broadcast}"
rc_net_${bridgeName}_provide="!net"
EOF

ln -s /etc/init.d/net.lo /etc/init.d/net.${bridgeName}
rc-update add net.${bridgeName} default
/etc/init.d/net.${bridgeName} start
ip addr show ${bridgeName} && brctl show
</source>

== Delete a bridging device ==
Tear the bridge down but check if it would also stop all network services:
<source lang="bash">
# Set your bridge name
bridgeName=<NAME OF YOUR BRIDGE> # e.g.: vmbr4

# Check if it excludes the net services
grep rc_net_${bridgeName}_provide /etc/conf.d/net
</source>
If grep produces output you might consider running the initscript with the ''-D'' options (otherwise network services, like sshd, might get stopped).

Otherwise just stop the bridge and remove the initscript:
<source lang="bash">
/etc/init.d/net.${bridgeName} stop

rc-update del net.${bridgeName} default

rm /etc/init.d/net.${bridgeName}
</source>

Remove the bridge from the network configuration:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

### Remove the options for the bridge you want to delete
#bridge_vmbr4="vlan141"
#bridge_add_vlan141="vmbr4"
#config_vmbr4="192.168.141.13/24 brd 192.168.141.255" # 192.168.141.13 is the ip address of the vm-node.
#rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.

# [...]
</source>

== Remove a VLAN on a bonding interface ==
Delete the VLAN from the interface bond0:
<source lang="bash">
# Set your vlan id
vlanId=<VLAN ID> # e.g. 141

# Shut the vlan down
ip link set vlan${vlanId} down

# Remove it from the interface bond0
ip link delete vlan${vlanId} type vlan
</source>

Edit <code>/etc/conf.d/net</code> and remove your VLAN:
$EDITOR /etc/conf.d/net

<source lang="bash">
# [...]

### Remove your vlan options
# pub2 VLAN
# vlan141_name="vlan141"
# config_vlan141="null"

# [...]
</source>

= Links =
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap10 Gentoo Handbook - VLAN (802.1q support)]
* [https://wiki.archlinux.org/index.php/VLAN ArchWiki - VLAN]
* [http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge Linux Foundation - Bridge]
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap7 Gentoo Handbook - Bridging (802.1d support)]
* [http://wiki.gentoo.org/wiki/Network_bridge Gentoo Wiki - Network Bridge]

[[Category:Workarounds]]

Manually adding a new bridge

2014-02-12T14:21:58Z

David: /* Remove a VLAN on a bonding interface */

= Overview =
The following page describes the steps to manually add a new Ethernet bridge (<code>vmbrX</code>) to an existing stoney cloud installation.

= Prerequisites =
To be able to add and configure a new bridge, you need to have the following informations present:
* VLAN ID, such as <code>141</code>
* IP prefix, such as <code>192.0.2.0/24</code>
* Default gateway address (if any), such as <code>192.0.2.1</code>
* IP address for the DHCP server on the VM node, such as <code>192.0.2.13</code>
* Broadcast address, such as <code>192.0.2.255</code>

Make sure, that you have configured the new VLAN on all involved switches and routers beforehand.

= Configuration =
== Add the new VLAN on top of the bonding interface ==
Login to the first VM node and manually add the new VLAN to the existing bonding interface (<code>bond0</code>). In the following example VLAN <code>141</code> is added:
<source lang="bash">
ip link add link bond0 name vlan141 type vlan id 141
ip link set vlan141 up
ip link show vlan141
</source>
<pre>
71: vlan141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
</pre>

In order to bring the newly configured VLAN up after a system reboot, the network configuration needs to be extended. Edit <code>/etc/conf.d/net</code> and add the following entry:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# VLAN (802.1q support)

vlans_bond0="130 120 110 140 141"

# pub VLAN
vlan140_name="vlan140"
config_vlan140="null"

### Add your new vlan at the end of the VLAN section
# pub2 VLAN
vlan141_name="vlan141"
config_vlan141="null"

# [...]
</source>

== Create a new bridging device ==
As the new VLAN interface is now up, an additional bridge can be added.

Again the network configuration needs to be edited. Append the configuration of the new bridge, below the existing <code>vmbr0</code> bridge configuration. In the following example the bridge <code>vmbr4</code> is added on top of VLAN <code>141</code> with a prefix of <code>192.0.2.13/24</code>:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# Bridging (802.1d) interfaces

bridge_vmbr0="vlanXXX"
bridge_add_vlanXXX="vmbr0"
config_vmbr0="XXX.XXX.XXX.XXX/XX brd XXX.XXX.XXX.XXX"
routes_vmbr0="default via XXX.XXX.XXX.XXX"

### Add your new bridge at the end of the bridge section
bridge_vmbr4="vlan141"
bridge_add_vlan141="vmbr4"
config_vmbr4="192.0.2.13/24 brd 192.0.2.255"
rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.
</source>

To bring the bridge up, an init script is required. Simply symlink the <code>net.lo</code> script to the new bridge name's script, here <code>net.vmbr4</code> is assumed:
<source lang="bash">
ln -s /etc/init.d/net.lo /etc/init.d/net.vmbr4
rc-update add net.vmbr4 default
/etc/init.d/net.vmbr4 start
</source>
<pre>
* Caching service dependencies ... [ ok ]
* Bringing up interface vmbr4
* Creating bridge vmbr4 ...
* Adding ports to vmbr4
* vlan141 ... [ ok ]
* 192.0.2.13/24 ... [ ok ]
</pre>

Check if everything looks fine:
<source lang="bash">
ip addr show vmbr4 && brctl show
</source>
<pre>
72: vmbr4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
inet 192.0.2.13/24 brd 192.0.2.255 scope global vmbr4
valid_lft forever preferred_lft forever

bridge name bridge id STP enabled interfaces
vmbr4 8000.3860779c9840 no vlan141
</pre>

Or if you would like to copy&paste:
<source lang="bash">
vlanId=141
bridgeName=vmbr4
nodeIp=192.0.2.13
prefixLength=24
broadcast=192.0.2.255

cat << EOF >> /etc/conf.d/net

bridge_${bridgeName}="vlan${vlanId}"
bridge_add_vlan${vlanId}="${bridgeName}"
config_${bridgeName}="${nodeIp}/${prefixLength} brd ${broadcast}"
rc_net_${bridgeName}_provide="!net"
EOF

ln -s /etc/init.d/net.lo /etc/init.d/net.${bridgeName}
rc-update add net.${bridgeName} default
/etc/init.d/net.${bridgeName} start
ip addr show ${bridgeName} && brctl show
</source>

== Delete a bridging device ==
Tear the bridge down but check if it would also stop all network services:
<source lang="bash">
# Set your bridge name
bridgeName=<NAME OF YOUR BRIDGE> # e.g.: vmbr4

# Check if it excludes the net services
grep rc_net_${bridgeName}_provide /etc/conf.d/net
</source>
If grep produces output you might consider running the initscript with the ''-D'' options (otherwise network services, like sshd, might get stopped).

Otherwise just stop the bridge and remove the initscript:
<source lang="bash">
/etc/init.d/net.${bridgeName} stop

rc-update del net.${bridgeName} default

rm /etc/init.d/net.${bridgeName}
</source>

Remove the bridge from the network configuration:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>
<source lang="bash">
# [...]

### Remove the options for the bridge you want to delete
#bridge_vmbr4="vlan141"
#bridge_add_vlan141="vmbr4"
#config_vmbr4="192.0.2.13/24 brd 192.0.2.255"
#rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.

# [...]
</source>

== Remove a VLAN on a bonding interface ==
Delete the VLAN from the interface bond0:
<source lang="bash">
# Set your vlan id
vlanId=<VLAN ID> # e.g. 141

# Shut the vlan down
ip link set vlan${vlanId} down

# Remove it from the interface bond0
ip link delete vlan${vlanId} type vlan
</source>

Edit <code>/etc/conf.d/net</code> and remove your VLAN:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

### Remove your vlan options
# pub2 VLAN
# vlan141_name="vlan141"
# config_vlan141="null"

# [...]
</source>

= Links =
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap10 Gentoo Handbook - VLAN (802.1q support)]
* [https://wiki.archlinux.org/index.php/VLAN ArchWiki - VLAN]
* [http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge Linux Foundation - Bridge]
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap7 Gentoo Handbook - Bridging (802.1d support)]
* [http://wiki.gentoo.org/wiki/Network_bridge Gentoo Wiki - Network Bridge]

Manually adding a new bridge

2014-02-12T14:20:00Z

David: /* Delete a bridging device */

= Overview =
The following page describes the steps to manually add a new Ethernet bridge (<code>vmbrX</code>) to an existing stoney cloud installation.

= Prerequisites =
To be able to add and configure a new bridge, you need to have the following informations present:
* VLAN ID, such as <code>141</code>
* IP prefix, such as <code>192.0.2.0/24</code>
* Default gateway address (if any), such as <code>192.0.2.1</code>
* IP address for the DHCP server on the VM node, such as <code>192.0.2.13</code>
* Broadcast address, such as <code>192.0.2.255</code>

Make sure, that you have configured the new VLAN on all involved switches and routers beforehand.

= Configuration =
== Add the new VLAN on top of the bonding interface ==
Login to the first VM node and manually add the new VLAN to the existing bonding interface (<code>bond0</code>). In the following example VLAN <code>141</code> is added:
<source lang="bash">
ip link add link bond0 name vlan141 type vlan id 141
ip link set vlan141 up
ip link show vlan141
</source>
<pre>
71: vlan141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
</pre>

In order to bring the newly configured VLAN up after a system reboot, the network configuration needs to be extended. Edit <code>/etc/conf.d/net</code> and add the following entry:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# VLAN (802.1q support)

vlans_bond0="130 120 110 140 141"

# pub VLAN
vlan140_name="vlan140"
config_vlan140="null"

### Add your new vlan at the end of the VLAN section
# pub2 VLAN
vlan141_name="vlan141"
config_vlan141="null"

# [...]
</source>

== Create a new bridging device ==
As the new VLAN interface is now up, an additional bridge can be added.

Again the network configuration needs to be edited. Append the configuration of the new bridge, below the existing <code>vmbr0</code> bridge configuration. In the following example the bridge <code>vmbr4</code> is added on top of VLAN <code>141</code> with a prefix of <code>192.0.2.13/24</code>:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# Bridging (802.1d) interfaces

bridge_vmbr0="vlanXXX"
bridge_add_vlanXXX="vmbr0"
config_vmbr0="XXX.XXX.XXX.XXX/XX brd XXX.XXX.XXX.XXX"
routes_vmbr0="default via XXX.XXX.XXX.XXX"

### Add your new bridge at the end of the bridge section
bridge_vmbr4="vlan141"
bridge_add_vlan141="vmbr4"
config_vmbr4="192.0.2.13/24 brd 192.0.2.255"
rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.
</source>

To bring the bridge up, an init script is required. Simply symlink the <code>net.lo</code> script to the new bridge name's script, here <code>net.vmbr4</code> is assumed:
<source lang="bash">
ln -s /etc/init.d/net.lo /etc/init.d/net.vmbr4
rc-update add net.vmbr4 default
/etc/init.d/net.vmbr4 start
</source>
<pre>
* Caching service dependencies ... [ ok ]
* Bringing up interface vmbr4
* Creating bridge vmbr4 ...
* Adding ports to vmbr4
* vlan141 ... [ ok ]
* 192.0.2.13/24 ... [ ok ]
</pre>

Check if everything looks fine:
<source lang="bash">
ip addr show vmbr4 && brctl show
</source>
<pre>
72: vmbr4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
inet 192.0.2.13/24 brd 192.0.2.255 scope global vmbr4
valid_lft forever preferred_lft forever

bridge name bridge id STP enabled interfaces
vmbr4 8000.3860779c9840 no vlan141
</pre>

Or if you would like to copy&paste:
<source lang="bash">
vlanId=141
bridgeName=vmbr4
nodeIp=192.0.2.13
prefixLength=24
broadcast=192.0.2.255

cat << EOF >> /etc/conf.d/net

bridge_${bridgeName}="vlan${vlanId}"
bridge_add_vlan${vlanId}="${bridgeName}"
config_${bridgeName}="${nodeIp}/${prefixLength} brd ${broadcast}"
rc_net_${bridgeName}_provide="!net"
EOF

ln -s /etc/init.d/net.lo /etc/init.d/net.${bridgeName}
rc-update add net.${bridgeName} default
/etc/init.d/net.${bridgeName} start
ip addr show ${bridgeName} && brctl show
</source>

== Delete a bridging device ==
Tear the bridge down but check if it would also stop all network services:
<source lang="bash">
# Set your bridge name
bridgeName=<NAME OF YOUR BRIDGE> # e.g.: vmbr4

# Check if it excludes the net services
grep rc_net_${bridgeName}_provide /etc/conf.d/net
</source>
If grep produces output you might consider running the initscript with the ''-D'' options (otherwise network services, like sshd, might get stopped).

Otherwise just stop the bridge and remove the initscript:
<source lang="bash">
/etc/init.d/net.${bridgeName} stop

rc-update del net.${bridgeName} default

rm /etc/init.d/net.${bridgeName}
</source>

Remove the bridge from the network configuration:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>
<source lang="bash">
# [...]

### Remove the options for the bridge you want to delete
#bridge_vmbr4="vlan141"
#bridge_add_vlan141="vmbr4"
#config_vmbr4="192.0.2.13/24 brd 192.0.2.255"
#rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.

# [...]
</source>

== Remove a VLAN on a bonding interface ==
Delete the VLAN from the interface bond0:
<source lang="bash">
# Set your vlan id
vlanId=<VLAN ID> # e.g. 141

# Shut the vlan down
ip link set vlan${vlanId} down

# Remove it from the interface bond0
ip link delete link bond0 name vlan${vlanId} type vlan id ${vlanId}
</source>

Edit <code>/etc/conf.d/net</code> and remove your VLAN:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

### Remove your vlan options
# pub2 VLAN
# vlan141_name="vlan141"
# config_vlan141="null"

# [...]
</source>

= Links =
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap10 Gentoo Handbook - VLAN (802.1q support)]
* [https://wiki.archlinux.org/index.php/VLAN ArchWiki - VLAN]
* [http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge Linux Foundation - Bridge]
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap7 Gentoo Handbook - Bridging (802.1d support)]
* [http://wiki.gentoo.org/wiki/Network_bridge Gentoo Wiki - Network Bridge]

Manually adding a new bridge

2014-02-12T14:18:21Z

David: /* Configuration */

= Overview =
The following page describes the steps to manually add a new Ethernet bridge (<code>vmbrX</code>) to an existing stoney cloud installation.

= Prerequisites =
To be able to add and configure a new bridge, you need to have the following informations present:
* VLAN ID, such as <code>141</code>
* IP prefix, such as <code>192.0.2.0/24</code>
* Default gateway address (if any), such as <code>192.0.2.1</code>
* IP address for the DHCP server on the VM node, such as <code>192.0.2.13</code>
* Broadcast address, such as <code>192.0.2.255</code>

Make sure, that you have configured the new VLAN on all involved switches and routers beforehand.

= Configuration =
== Add the new VLAN on top of the bonding interface ==
Login to the first VM node and manually add the new VLAN to the existing bonding interface (<code>bond0</code>). In the following example VLAN <code>141</code> is added:
<source lang="bash">
ip link add link bond0 name vlan141 type vlan id 141
ip link set vlan141 up
ip link show vlan141
</source>
<pre>
71: vlan141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
</pre>

In order to bring the newly configured VLAN up after a system reboot, the network configuration needs to be extended. Edit <code>/etc/conf.d/net</code> and add the following entry:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# VLAN (802.1q support)

vlans_bond0="130 120 110 140 141"

# pub VLAN
vlan140_name="vlan140"
config_vlan140="null"

### Add your new vlan at the end of the VLAN section
# pub2 VLAN
vlan141_name="vlan141"
config_vlan141="null"

# [...]
</source>

== Create a new bridging device ==
As the new VLAN interface is now up, an additional bridge can be added.

Again the network configuration needs to be edited. Append the configuration of the new bridge, below the existing <code>vmbr0</code> bridge configuration. In the following example the bridge <code>vmbr4</code> is added on top of VLAN <code>141</code> with a prefix of <code>192.0.2.13/24</code>:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# Bridging (802.1d) interfaces

bridge_vmbr0="vlanXXX"
bridge_add_vlanXXX="vmbr0"
config_vmbr0="XXX.XXX.XXX.XXX/XX brd XXX.XXX.XXX.XXX"
routes_vmbr0="default via XXX.XXX.XXX.XXX"

### Add your new bridge at the end of the bridge section
bridge_vmbr4="vlan141"
bridge_add_vlan141="vmbr4"
config_vmbr4="192.0.2.13/24 brd 192.0.2.255"
rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.
</source>

To bring the bridge up, an init script is required. Simply symlink the <code>net.lo</code> script to the new bridge name's script, here <code>net.vmbr4</code> is assumed:
<source lang="bash">
ln -s /etc/init.d/net.lo /etc/init.d/net.vmbr4
rc-update add net.vmbr4 default
/etc/init.d/net.vmbr4 start
</source>
<pre>
* Caching service dependencies ... [ ok ]
* Bringing up interface vmbr4
* Creating bridge vmbr4 ...
* Adding ports to vmbr4
* vlan141 ... [ ok ]
* 192.0.2.13/24 ... [ ok ]
</pre>

Check if everything looks fine:
<source lang="bash">
ip addr show vmbr4 && brctl show
</source>
<pre>
72: vmbr4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
inet 192.0.2.13/24 brd 192.0.2.255 scope global vmbr4
valid_lft forever preferred_lft forever

bridge name bridge id STP enabled interfaces
vmbr4 8000.3860779c9840 no vlan141
</pre>

Or if you would like to copy&paste:
<source lang="bash">
vlanId=141
bridgeName=vmbr4
nodeIp=192.0.2.13
prefixLength=24
broadcast=192.0.2.255

cat << EOF >> /etc/conf.d/net

bridge_${bridgeName}="vlan${vlanId}"
bridge_add_vlan${vlanId}="${bridgeName}"
config_${bridgeName}="${nodeIp}/${prefixLength} brd ${broadcast}"
rc_net_${bridgeName}_provide="!net"
EOF

ln -s /etc/init.d/net.lo /etc/init.d/net.${bridgeName}
rc-update add net.${bridgeName} default
/etc/init.d/net.${bridgeName} start
ip addr show ${bridgeName} && brctl show
</source>

== Delete a bridging device ==
Tear the bridge down but check if it would also stop all network services:
<source lang="bash">
# Set your bridge name
bridgeName=<NAME OF YOUR BRIDGE> # e.g.: vmbr4

# Check if it excludes the net services
grep rc_net_${bridgeName}_provide /etc/conf.d/net
</source>
If grep produces output you might consider running the initscript with the ''-D'' options (otherwise network services, like sshd, might get stopped).

Otherwise just stop the bridge and remove the initscript:
<source lang="bash">
/etc/init.d/net.vmbr4 start

rc-update add net.vmbr4 default

ln -s /etc/init.d/net.lo /etc/init.d/net.vmbr4
</source>

Remove the bridge from the network configuration:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>
<source lang="bash">
# [...]

### Remove the options for the bridge you want to delete
#bridge_vmbr4="vlan141"
#bridge_add_vlan141="vmbr4"
#config_vmbr4="192.0.2.13/24 brd 192.0.2.255"
#rc_net_vmbr4_provide="!net" # Otherwise stopping or restarting the bridge, will bring all net-services down.

# [...]
</source>

== Remove a VLAN on a bonding interface ==
Delete the VLAN from the interface bond0:
<source lang="bash">
# Set your vlan id
vlanId=<VLAN ID> # e.g. 141

# Shut the vlan down
ip link set vlan${vlanId} down

# Remove it from the interface bond0
ip link delete link bond0 name vlan${vlanId} type vlan id ${vlanId}
</source>

Edit <code>/etc/conf.d/net</code> and remove your VLAN:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

### Remove your vlan options
# pub2 VLAN
# vlan141_name="vlan141"
# config_vlan141="null"

# [...]
</source>

= Links =
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap10 Gentoo Handbook - VLAN (802.1q support)]
* [https://wiki.archlinux.org/index.php/VLAN ArchWiki - VLAN]
* [http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge Linux Foundation - Bridge]
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap7 Gentoo Handbook - Bridging (802.1d support)]
* [http://wiki.gentoo.org/wiki/Network_bridge Gentoo Wiki - Network Bridge]

Manually adding a new bridge

2014-02-12T10:47:17Z

David: /* Configuration */

= Overview =
The following page describes the steps to manually add a new Ethernet bridge (<code>vmbrX</code>) to an existing stoney cloud installation.

= Prerequisites =
To be able to add and configure a new bridge, you need to have the following informations present:
* VLAN ID, such as <code>141</code>
* IP prefix, such as <code>192.0.2.0/24</code>
* Default gateway address (if any), such as <code>192.0.2.1</code>
* IP address for the DHCP server on the VM node, such as <code>192.0.2.13</code>
* Broadcast address, such as <code>192.0.2.255</code>

Make sure, that you have configured the new VLAN on all involved switches and routers beforehand.

= Configuration =
== Add the new VLAN on top of the bonding interface ==
Login to the first VM node and manually add the new VLAN to the existing bonding interface (<code>bond0</code>). In the following example VLAN <code>141</code> is added:
<source lang="bash">
ip link add link bond0 name vlan141 type vlan id 141
ip link set vlan141 up
ip link show vlan141
</source>
<pre>
71: vlan141@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
</pre>

In order to bring the newly configured VLAN up after a system reboot, the network configuration needs to be extended. Edit <code>/etc/conf.d/net</code> and add the following entry:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# VLAN (802.1q support)

vlans_bond0="130 120 110 140 141"

# pub VLAN
vlan140_name="vlan140"
config_vlan140="null"

### Add your new vlan at the end of the VLAN section
# pub2 VLAN
vlan141_name="vlan141"
config_vlan141="null"

# [...]
</source>

== Create a new bridging device ==
As the new VLAN interface is now up, an additional bridge can be added.

Again the network configuration needs to be edited. Append the configuration of the new bridge, below the existing <code>vmbr0</code> bridge configuration. In the following example the bridge <code>vmbr4</code> is added on top of VLAN <code>141</code> with a prefix of <code>192.0.2.13/24</code>:
<source lang="bash">
$EDITOR /etc/conf.d/net
</source>

<source lang="bash">
# [...]

#-----------------------------------------------------------------------------
# Bridging (802.1d) interfaces

bridge_vmbr0="vlanXXX"
bridge_add_vlanXXX="vmbr0"
config_vmbr0="XXX.XXX.XXX.XXX/XX brd XXX.XXX.XXX.XXX"
routes_vmbr0="default via XXX.XXX.XXX.XXX"

### Add your new bridge at the end of the bridge section
bridge_vmbr4="vlan141"
bridge_add_vlan141="vmbr4"
config_vmbr4="192.0.2.13/24 brd 192.0.2.255"
</source>

To bring the bridge up, an init script is required. Simply symlink the <code>net.lo</code> script to the new bridge name's script, here <code>net.vmbr4</code> is assumed:
<source lang="bash">
ln -s /etc/init.d/net.lo /etc/init.d/net.vmbr4
rc-update add net.vmbr4 default
/etc/init.d/net.vmbr4 start
</source>
<pre>
* Caching service dependencies ... [ ok ]
* Bringing up interface vmbr4
* Creating bridge vmbr4 ...
* Adding ports to vmbr4
* vlan141 ... [ ok ]
* 192.0.2.13/24 ... [ ok ]
</pre>

Check if everything looks fine:
<source lang="bash">
ip addr show vmbr4 && brctl show
</source>
<pre>
72: vmbr4: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
inet 192.0.2.13/24 brd 192.0.2.255 scope global vmbr4
valid_lft forever preferred_lft forever

bridge name bridge id STP enabled interfaces
vmbr4 8000.3860779c9840 no vlan141
</pre>

Or if you would like to copy&paste:
<source lang="bash">
vlanId=141
bridgeName=vmbr4
nodeIp=192.0.2.13
prefixLength=24
broadcast=192.0.2.255

cat << EOF >> /etc/conf.d/net

bridge_${bridgeName}="vlan${vlanId}"
bridge_add_vlan${vlanId}="${bridgeName}"
config_${bridgeName}="${nodeIp}/${prefixLength} brd ${broadcast}"
EOF

ln -s /etc/init.d/net.lo /etc/init.d/net.${bridgeName}
rc-update add net.${bridgeName} default
/etc/init.d/net.${bridgeName} start
ip addr show ${bridgeName} && brctl show
</source>

= Links =
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap10 Gentoo Handbook - VLAN (802.1q support)]
* [https://wiki.archlinux.org/index.php/VLAN ArchWiki - VLAN]
* [http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge Linux Foundation - Bridge]
* [http://www.gentoo.org/doc/en/handbook/handbook-amd64.xml?part=4&chap=3#doc_chap7 Gentoo Handbook - Bridging (802.1d support)]
* [http://wiki.gentoo.org/wiki/Network_bridge Gentoo Wiki - Network Bridge]

stoney conductor: VM Backup

2014-01-30T13:21:48Z

David: /* How to exclude a machine from the backup */

= Overview =
This page describes how the VMs and VM-Templates are backed-up and restored inside the [http://www.stoney-cloud.org stoney cloud].

= Requirements =
TBD: PKL. Bitte bitte bei folgenden Attributen den Platzbedarf aufzeigen (wie kommt man auf die minimale Grösse)
* sstBackupRootDirectory: file:///var/backup/virtualization
* sstBackupRetainDirectory: file:///var/virtualization/retain
* sstBackupRamDiskLocation: file:///var/cache/kvmbackup

* A working stoney cloud, installed according to [[stoney cloud: Single-Node Installation]] or [[stoney cloud: Multi-Node Installation]].
* The backup configuration must be set: [[stoney_conductor:_OpenLDAP_directory_data_organisation#Backup | stoney conductor: OpenLDAP directory data organisation]].

= Backup =
== Basic idea ==
The main idea to backup a VM or a VM-Template is, to divide the task into three subtasks:
* Snapshot: Save the machines state (CPU, Memory and Disk)
* Merge: Merge the Disk-Image-Snapshot with the Live-Image
* Retain: Export the snapshot files
A more detailed and technical description for these three sub-processes can be found [[#Sub-Processes | here]].

Furthermore there is an control instance, which can independently call these three sub-processes for a given machine. Like that, the stoney cloud is able to handle different cases:
=== Backup a single machine ===
The procedure for backing up a single machine is very simple. Just call the three sub-processes (snapshot, merge and retain) one after the other. So the control instance would do some very basic stuff:
<source lang="c">
object machine = args[0];

if( snapshot( machine ) )
{

if ( merge( machine ) )
{

if ( retain( machine ) )
{
printf("Successfully backed up machine %s\n", machine);

} else
{
printf("Error while retaining machine %s: %s\n", machine, error);
}

} else
{
printf("Error while merging machine %s: %s\n", machine, error);
}

} else
{
printf("Error while snapshotting machine %s: %s\n", machine, error);
}
</source>

=== Backup multiple machines at the same time ===
When backing up multiple machines at the same time, we need to make sure that the downtime for the machines are as close together as possible. Therefore the control instance should call first the snapshot process for all machines. After every machine has been snapshotted, the control instance can call the merge and retain process for every machine. The most important part here is, that the control instance somehow remembers, if the snapshot for a given machine was successful or not. Because if the snapshot failed, it must not call the merge and retain process. So the control instance needs a little bit more logic:

<source lang="c">
object machines[] = args[0];
object successful_snapshots[];

# Snapshot all machines
for( int i = 0; i < sizeof(machines) / sizeof(object) ; i++ )
{
# If the snapshot was successful, put the machine into the
# successful_snapshots array
if ( snapshot( machines[i] ) )
{
successful_snapshots[machines[i]];
} else
{
printf("Error while snapshotting machine %s: %s\n", machines[i],error);
}
}

# Merge and reatin all successful_snapshot machines
for ( int i = 0; i < sizeof(successful_snapshots) / sizeof(object) ; i++ ) )
{
# Check if the element at this position is not null, then the snapshot
# for this machine was successful
if ( successful_snapshots[i] )
{
if ( merge( successful_snapshots[i] ) )
{
if ( retain( successful_snapshots[i] ) )
{
printf("Successfully backed-up machine %s\n", successful_snapshots[i]);
} else
{
printf("Error while retaining machine %s: %s\n", successful_snapshots[i],error);
}

} else
{
printf("Error while merging machine %s: %s\n", successful_snapshots[i],error);
}
}
}
</source>

=== Sub-Processes ===
==== Snapshot ====
# Create a snapshot with state:
#* If the VM <code>vm-001</code> is running:
#** Save the state of VM <code>vm-001</code> to the file <code>vm-001.state</code> (This file can either be created on a RAM-Disk or directly in the retain location. This example however saves the file to a RAM-Disk): <syntaxhighlight lang="bash">virsh save vm-001 /path/to/ram-disk/vm-001.state</syntaxhighlight>
#** After this command, the VMs CPU and memory state is represented by the file <code>/path/to/ram-disk/vm-001.state</code> and the VM <code>vm-001</code> is shut down.
#* If the VM <code>vm-001</code> is shut down:
#** Create a fake state file for the VM: <syntaxhighlight lang="bash">echo "Machine is not runnung, no state file" > /path/to/ram-disk/vm-001.state</syntaxhighlight>
# Move the disk image <code>/path/to/images/vm-001.qcow2</code> to the retain location: <syntaxhighlight lang="bash">mv /path/to/images/vm-001.qcow2 /path/to/retain/vm-001.qcow2</syntaxhighlight>
#* '''Please note:''' The retain directory (<code>/path/to/retain/</code>) '''has to be''' on the same partition as the images directory (<code>/path/to/images/</code>). This will make the <code>mv</code> operation very fast (only renaming the inode). So the downtime (remember the VM <code>vm-001</code> is shut down) is as short as possible.
#* '''Please note2:''' If the VM <code>vm-001</code> has more than just one disk-image, repeat this step for every disk-image
# Create the new (empty) disk image with the old as backing store file: <syntaxhighlight lang="bash">qemu-img create -f qcow2 -b /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</syntaxhighlight>
#* '''Please note:''' If the VM <code>vm-001</code> has more than just one disk-image, repeat this step for every disk-image
# Set correct ownership and permission to the newly created image:
#* <syntaxhighlight lang="bash">chmod 660 /path/to/images/vm-001.qcow2</syntaxhighlight>
#* <syntaxhighlight lang="bash">chown root:vm-storage /path/to/images/vm-001.qcow2</syntaxhighlight>
#* '''Please note:''' If the VM <code>vm-001</code> has more than just one disk-image, repeat these steps for every disk-image
# Save the VMs XML description
#* Save the current XML description of VM <code>vm-001</code> to a file at the retain location: <syntaxhighlight lang="bash">virsh dumpxml vm-001 > /path/to/retain/vm-001.xml</syntaxhighlight>
# Save the backend entry
#* There is no generic command to save the backend entry (since the command depends on the backend). Important here is, that the backend entry of the VM <code>vm-001</code> is saved to the retain location: <code>/path/to/retain/vm-001.backend</code>
# Restore the VMs <code>vm-001</code> from its saved state (this will also start the VM): <syntaxhighlight lang="bash">virsh restore /path/to/ram-disk/vm-001.state</syntaxhighlight>
#* '''Please note:''' After this operation the VM <code>vm-001</code> is running again (continues where we stopped it), and we have a consistent backup for the VM <code>vm-001</code>:
#** The file <code>/path/to/ram-disk/vm-001.state</code> contains the CPU and memory state of VM <code>vm-001</code> at time T1
#** The file <code>/path/to/retain/vm-001.qcow2</code> contains the disk state of VM <code>vm-001</code> at time T1
#*** '''Important:''' Remember: The live-disk-image <code>/path/to/images/vm-001.qcow2</code> still contains a reference to this file!! So you cannot delete or move it!!!
#** The file <code>/path/to/retain/vm-001.xml</code> contains the XML description of VM <code>vm-001</code> at time T1
#** The file <code>/path/to/retain/vm-001.backend</code> contains the backend entry of VM <code>vm-001</code> at time T1
# Move the state file from the RAM-Disk to the retain location (if you used the RAM-Disk to save the VMs state)
#* <syntaxhighlight lang="bash">mv /path/to/ram-disk/vm-001.state /path/to/retain/vm-001.state</syntaxhighlight>

See also: [[stoney_conductor:_prov_backup_kvm#Snapshot | Snapshot workflow ]]

==== Merge ====
# Check if the VM <code>vm-001</code> is running
#* If not, start the VM in paused state: <syntaxhighlight lang="bash">virsh start --paused vm-001</syntaxhighlight>
# Merge the live-disk-image (<code>/path/to/images/vm-001.qcow2</code>) with its backing store file (<code>/path/to/retain/vm-001.qcow2</code>): <syntaxhighlight lang="bash">virsh qemu-monitor-command vm-001 --hmp "block_stream drive-virtio-disk0"</syntaxhighlight>
#* '''Please note:''' If a VM has more than just one disk-image, repeat this step for every image. Just increase the number at the end of the command. So command to merge the second disk image would be: <syntaxhighlight lang="bash">virsh qemu-monitor-command vm-001 --hmp "block_stream drive-virtio-disk1"</syntaxhighlight>
# If the machine is running in paused state (means we started it in 1. because it was not running), stop it again:
#* <syntaxhighlight lang="bash">virsh shutdown vm-001</syntaxhighlight>

'''Please note:''' After these steps, the live-disk-image <code>/path/to/image/vm-001.qcow2</code> no longer contains a reference to the image at the retain location (<code>/path/to/retain/vm-001.qcow2</code>). This is important for the [[#Retain | retain process]].

See also: [[stoney_conductor:_prov_backup_kvm#Merge | Merge workflow]]

==== Retain ====
# Move the all the files in from the retain directory (<code>/path/to/retain/</code>) to the backup directory (<code>/path/to/backup/</code>)
## Move the VMs state file to the backup directory
##* <source lang="bash">mv /path/to/retain/vm-001.state /path/to/backup/vm-001.state</source>
## Move the VMs disk image to the backup directory
##* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/backup/vm-001.qcow2</source>
##** '''Please note:''' If the VM <code>vm-001</code> has more than just one disk image, repeat this step for each disk image
## Move the VMs XML description file to the backup directory
##* <source lang="bash">mv /path/to/retain/vm-001.xml /path/to/backup/vm-001.xml</source>
## Move the VMs backend entry file to the backup directory
##* <source lang="bash">mv /path/to/retain/vm-001.backend /path/to/backup/vm-001.backend</source>

See also [[stoney_conductor:_prov_backup_kvm#Retain | Retain workflow]]

== Communication through backend ==
Since the stoney cloud is (as the name says already) a cloud solution, it makes sense to have a backend (in our case openLDAP) involved in the whole process. Like that it is possible to run the backup jobs decentralized on every vm-node. The control instance can then modify the backend, and theses changes are seen by the diffenrent backup daemons on the vm-nodes. So the communication could look like shown in the following picture (Figure 1):

[[File:Daemon-communication.png|800px|thumbnail|none|Figure 1: Communication between the control instance and the prov-backup-kvm daemon through the LDAP backend]]

=== Control-Instance Daemon Interaction for creating a Backup with LDIF Examples ===
The step numbers correspond with the graphical overview from above.

==== Step 00: Backup Configuration for a virtual machine ====
<pre>
# The following backup configuration says, that the backup should be done daily, at 03:00 hours (localtime).
# * * * * * command to be executed
# - - - - -
# | | | | |
# | | | | +----- day of week (0 - 6) (Sunday=0)
# | | | +------- month (1 - 12)
# | | +--------- day of month (1 - 31)
# | +----------- hour (0 - 23)
# +------------- min (0 - 59)
# localtime in the crontab entry
dn: ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
objectclass: sstCronObjectClass
ou: backup
description: This sub tree contains the backup plan for the virtual machine kvm-005.
sstCronMinute: 0
sstCronHour: 3
sstCronDay: *
sstCronMonth: *
sstCronDayOfWeek: *
sstCronActive: TRUE
sstBackupRootDirectory: file:///var/backup/virtualization
sstBackupRetainDirectory: file:///var/virtualization/retain
sstBackupRamDiskLocation: file:///mnt/ramdisk-test
sstVirtualizationDiskImageFormat: qcow2
sstVirtualizationDiskImageOwner: root
sstVirtualizationDiskImageGroup: vm-storage
sstVirtualizationDiskImagePermission: 0660
sstBackupNumberOfIterations: 1
sstVirtualizationVirtualMachineForceStart: FALSE
sstVirtualizationBandwidthMerge: 0
</pre>

==== Step 01: Initialize Backup Sub Tree (Control instance daemon) ====
The sub tree ''' ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch''' reflects the time, when the backup is planned (in the form of [YYYY][MM][DD]T[hh][mm][ss]Z ([http://en.wikipedia.org/wiki/ISO_8601 ISO 8601]) and it should be written at the time, when the backup is planned and should be executed. The section '''20121002T010000Z''' means the following:
* Year: 2012
* Month: 10
* Day of Month: 02
* Hour of Day: 01
* Minutes: 00
* Seconds: 00
Please be aware the the time is to be written in UTC (see also the comment in the LDIF example below).
<pre>
# This entry is the place holder for the backup, which is to be executed at 03:00 hours (localtime with daylight-saving). This
# leads to the 20121002T010000Z timestamp (which is written in UTC).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: sstProvisioning
objectclass: organizationalUnit
ou: 20121002T010000Z
sstProvisioningExecutionDate: 0
sstProvisioningMode: initialize
sstProvisioningReturnValue: 0
sstProvisioningState: 20121002T014513Z
</pre>

==== Step 02: Finalize the Initialization (Control instance daemon) ====
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is modified.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010001Z
-
replace: sstProvisioningMode
sstProvisioningMode: initialized
</pre>

==== Step 03: Start the Snapshot Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''snapshot''', the actual backup process is kicked off by the Control instance daemon.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# snapshot (this way the Provisioning-Backup-VKM daemon knows, that it must start the snapshotting process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshot
</pre>

==== Step 04: Starting the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotting''' to tell the Control instance daemon and other interested parties, that it is snapshotting the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to snapshotting by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: snapshotting
</pre>

==== Step 05: Finalizing the Snapshot Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the snapshot command, it sets the '''sstProvisioningMode''' to '''snapshotted''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the snapshot of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010011Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: snapshotted
</pre>

==== Step 06: Start the Merge Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''merge''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to merge the backing file disk image back into the current disk image.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# merge (this way the Provisioning-Backup-VKM daemon knows, that it must start the merging process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: merge
</pre>

==== Step 07: Starting the Merge Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the merge command, it sets the '''sstProvisioningMode''' to '''merging''' to tell the Control instance daemon and other interested parties, that it is merging the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to merging by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: merging
</pre>

==== Step 08: Finalizing the Merging Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the merge command, it sets the '''sstProvisioningMode''' to '''merged''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the merging of the virtual machine or virtual machine template is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T010500Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: merged
</pre>

==== Step 09: Start the Retain Process (Control instance daemon) ====
With the setting of the '''sstProvisioningMode''' to '''retain''', the Control instance daemon tells the Provisioning-Backup-KVM daemon to retain (copy and then delete) all the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningState is set to zero by the fc-brokerd, when sstProvisioningMode is modified to
# retain (this way the Provisioning-Backup-VKM daemon knows, that it must start the retaining process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retain
</pre>

==== Step 10: Starting the Retain Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the retain command, it sets the '''sstProvisioningMode''' to '''retaining''' to tell the Control instance daemon and other interested parties, that it is retaining the necessary files to the configured backup location.
<pre>
# The attribute sstProvisioningMode is set to retaining by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: retaining
</pre>

==== Step 11: Finalizing the Retaing Process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the retain command, it sets the '''sstProvisioningMode''' to '''retained''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the retaining of all the necessary files to the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the fc-brokerd knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: retained
</pre>

==== Step 12: Finalizing the Backup Process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''retained''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the backup process is finished, there for a new backup process could be started.
<pre>
# The attribute sstProvisioningState is updated with current time by the fc-brokerd, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the backup process is finished, there for a new backup process could be started.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Backup) ==
Since we do not have a working control instance, we need to have a workaround for backing up the machines:

* We do already have a BackupKVMWrapper.pl script (File-Backend) which executes the three [[#Sub-Processes | sub-processes ]] in the correct order for a given list of machines (see [[#Backup multiple machines at the same_time]]).
* We do already have the implementation for the whole backup with the LDAP-Backend (see [[ stoney conductor: prov backup kvm ]]).
* We can now combine these two existing scripts and create a wrapper (lets call it LDAPKVMWrapper) which, in some way, adds some logic to the BackupKVMWrapper.pl. In fact the LDAPKVMWrapper wrapper will generate the list of machines which need a backup.

The behaviour on our servers is as follows (c.f. Figure 2):
# The (decentralized) LDAPKVMWrapper wrapper (which is executed everyday via cronjob) generates a list off all machines running on the current host.
#* Currently on the hosts the cronjobs looks like: <code>00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
#* For each of these machines:
#** Check if the machine is excluded from the backup, if yes, remove the machine from the list
#** Check if the last backup was successful, if not, remove the machine from the list
# Update the backup subtree for each machine in the list
#* Remove the old backup leaf (the "yesterday-leaf"), and add a new one (the "today-leaf")
#* After this step, the machines are ready to be backed up
# Call the BackupKVMWrapper.pl script with the machines list as a parameter
# Wait for the BackupKVMWrapper.pl script to finish
# Go again through all machines and update the backup subtree a last time
#* Check if the backup was successful, if yes, set sstProvisioningMode = finished (see also TBD)

[[File:wrapper-interaction.png|500px|thumbnail|none|Figure 2: How the two wrapper interact with the LDAP backend]]

* If for some reason something does not work at all, the whole backup process can be deactivated by simply disabling the LDAPKVMWrapper cronjob
** <code>crontab -e</code>
** Comment the LDAPKVMWrapper cronjob line: <code>#00 01 * * * /usr/bin/LDAPKVMWrapper.pl | logger -t Backup-KVM</code>
=== How to exclude a machine from the backup ===
Login to one of the [[VM-Node | vm-nodes]] and execute the following command

If you want to exclude a machine from the backup run you simply need to add the following entry to your LDAP directory:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
ou: backup
sstbackupexcludefrombackup: TRUE
EOF
</source>

If the backup subtree in the LDAP directory already exists, you need to add the sstbackupexcludefrombackup attribute:
<source lang="bash">
machineuuid="<UUID OF THE MACHINE-NAME>" # e.g.: b9d13dbc-9ab7-4948-9daa-a5709de83dc2
cat << EOF | ldapadd -D cn=Manager,o=stepping-stone,c=ch -H ldaps://ldapm.stepping-stone.ch/ -W -x
dn: ou=backup,sstVirtualMachine=${machineuuid},ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: add
objectclass: top
objectclass: organizationalUnit
objectclass: sstVirtualizationBackupObjectClass
sstbackupexcludefrombackup: TRUE
EOF
</source>

==== Re-include the machine to the backup ====
If you want to re include a machine, simply delete the machines whole backup subtree. It will be recreated during the next backup run.

== Next steps ==

= Restore =
== Basic idea ==
The restore process, similar to the backup process, can be divided into three sub-processes:
* Unretain the small files: Copy the small files (backend entry, XML description) from the backup directory to the retain directory
* Unretain the big files: Copy the big files (state file, disk image(s)) form the backup directory to the retain directory
* Restore the machine: Replace the live disk image(s) by the one(s) from the backup and restore the machine from the state file

Additionally the restore process can also be divided into two phases:
* User-Interaction phase: After the "unretain small files" the user needs to decide two things:
** On conflicts between the backend entry file and the XML description, the user need to decide how to resolve this conflict(s)
** The user can also abort the restore process up to this point. After that the restore can not be aborted or undone!
* Non-User-Interaction phase: The daemons communicate through the backend between each other and the restore process continues without further user input (c.f. [[#Communication_through_backend_2 | Communication through backend]])

=== Sub Processes ===
==== Unretain small files ====
This workflow assumes that the backup directory is on the same physical server as the retain directory (protocol is file://)
# Copy the backend-entry file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.backend /path/to/retain/vm-001.backend</source>
# Copy the XML description from the from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.xml /path/to/retain/vm-001.xml</source>
# Compare the backend-entry file (the one in the retain directory) with the live-backend entry
#* Resolve all conflicts between these two backend entries
#** Modify the backend entry at the retain location accordingly
# Apply the same changes for the XML description at the retain location (backend entry and XML description need to be consistent).

==== Unretain large files ====
# Copy the state file from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.state /path/to/retain/vm-001.state</source>
# Copy the disk image(s) from the backup directory to the retain directory:
#* <source lang="bash">cp -p /path/to/backup/vm-001.qcow2 /path/to/retain/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image

==== Restore the VM ====
# Shutdown the VM if it is running:
#* <source lang="bash">virsh shutdown vm-001</source>
# Undefine the VM if it is still defined:
#* <source lang="bash">virsh undefine vm-001</source>
# Overwrite the original disk image:
#* <source lang="bash">mv /path/to/retain/vm-001.qcow2 /path/to/images/vm-001.qcow2</source>
#** '''Important:''' If a VM has more than just one disk image, repeat this step for every disk image
# Restore the VMs backend entry:
#* Write the backend entry from the retain location (<code>/path/to/retain/vm-001.backend</code>) to the backend
# Overwrite the VMs XML description with the one from the retain location
#* <source lang="bash">cp -p /path/to/retain/vm-001.xml /path/to/xmls/vm-001.xml</source>
# Restore the VM from the state file with the corrected XML
#* <source lang="bash">virsh restore /path/to/retain/vm-001.state --xml /path/to/xmls/vm-001.xml</source>

== Communication through backend ==
The actual KVM-Restore process is controlled completely by the Control instance daemon via the OpenLDAP directory. See [[#OpenLDAP Directory Integration|OpenLDAP Directory Integration]] the involved attributes and possible values.

[[File:Daemon-interaction-restore.png|thumb|500px|none|Figure 3: Communication between all involved parties during the restore process]]

You can modify/update these interactions by editing [[File:Restore-Interaction.xmi]] (you may need [http://uml.sourceforge.net/ Umbrello UML Modeller] diagram programme for KDE to display the content properly).

=== Control instance Daemon Interaction for restoring a Backup with LDIF Examples ===
==== Step 01: Start the unretainSmallFiles process (Control instance daemon) ====
The first step of the restore process is to copy the small files (in this case the XML file and the LDIF) from the configured backup location to the configured retain location.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainSmallFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainSmallFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainSmallFiles
</pre>

==== Step 02: Starting the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainingSmallFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the small files for the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingSmallFiles
</pre>

==== Step 03: Finalizing the unretainSmallFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the small files, it sets the '''sstProvisioningMode''' to '''unretainedSmallFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the small files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedSmallFiles
</pre>

==== Step 05: Start the unretainLargeFiles process (Control instance daemon) ====
Next step in the restore process is to copy the large files (state file and disk images) from the configured backup directory to the configured retain directory.

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# unretainLargeFiles (this way the Provisioning-Backup-VKM daemon knows, that it must start the unretainLargeFiles process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainLargeFiles
</pre>

==== Step 06: Starting the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the command to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainingLargeFiles''' to tell the Control instance daemon and other interested parties, that it is unretaining the large files for the virtual machine or virtual machine template.

In the meantime the vm-manager merges the LDIF we have unretained in [[#Step_02:_Starting_the_unretainSmallFiles_process_.28Provisioning-Backup-KVM_daemon.29 | step 02]] with the one in the live directory to sort out possible differences in the configuration of the virtual machine.
<pre>
# The attribute sstProvisioningMode is set to unretainingSmallFiles by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: unretainingLargeFiles
</pre>

==== Step 07: Finalizing the unretainLargeFiles process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the commands to unretain the large files, it sets the '''sstProvisioningMode''' to '''unretainedLargeFiles''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the unretaining of all the large files from the configured backup location is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: unretainedLargeFiles
</pre>

==== Step 09: Start the restore process (Control instance daemon) ====
Since we now have all necessary files in the configured retain location, the restore process can be started. There we simply copy the disk images back to their original location and restore the VM from the state file (which is also at the configured retain location)

<pre>
# The attribute sstProvisioningState is set to zero by the Control instance daemon, when sstProvisioningMode is modified to
# restore (this way the Provisioning-Backup-VKM daemon knows, that it must start the restore process).
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restore
</pre>

==== Step 10: Starting the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon receives the restore command, it sets the '''sstProvisioningMode''' to '''restoring''' to tell the Control instance daemon and other interested parties, that it is restoring the virtual machine or virtual machine template.
<pre>
# The attribute sstProvisioningMode is set to restoring by the Provisioning-Backup-VKM daemon.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningMode
sstProvisioningMode: restoring
</pre>

==== Step 11: Finalizing the restore process (Provisioning-Backup-KVM daemon) ====
As soon as the Provisioning-Backup-KVM daemon has executed the restore command, it sets the '''sstProvisioningMode''' to '''restored''', the '''sstProvisioningState''' to the current timestamp (UTC) and '''sstProvisioningReturnValue''' to zero to tell the Control instance daemon and other interested parties, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is set with the current timestamp by the Provisioning-Backup-VKM daemon, when
# the attributes sstProvisioningReturnValue and sstProvisioningMode are set.
# With this combination, the Control instance daemon knows, that it can proceed.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012000Z
-
replace: sstProvisioningReturnValue
sstProvisioningReturnValue: 0
-
replace: sstProvisioningMode
sstProvisioningMode: restored
</pre>

==== Step 12: Finalizing the restore process (Control instance daemon) ====
As soon as the Control instance daemon notices, that the attribute '''sstProvisioningMode''' ist set to '''restored''', it sets the '''sstProvisioningMode''' to '''finished''' and the '''sstProvisioningState''' to the current timestamp (UTC). All interested parties now know, that the restore process is finished.
<pre>
# The attribute sstProvisioningState is updated with current time by the Control instance daemon, when sstProvisioningMode is
# set to finished.
# All interested parties now know, that the restore process is finished.
dn: ou=20121002T010000Z,ou=backup,sstVirtualMachine=kvm-005,ou=virtual machines,ou=virtualization,ou=services,o=stepping-stone,c=ch
changetype: modify
replace: sstProvisioningState
sstProvisioningState: 20121002T012001Z
-
replace: sstProvisioningMode
sstProvisioningMode: finished
</pre>

== Current Implementation (Restore) ==
* Since the prov-backup-kvm daemon is not running on the vm-nodes (c.f. [[stoney_conductor:_Backup#State_of_the_art]]), the restore process does not work when clicking the icon in the webinterface.
* Resolving the conflicts in the backend and XML description file is not yet done
** Actually all steps not executed by prov-backup-kvm are not yet properly implemented (c.f. [[stoney_conductor:_prov_backup_kvm#Restore]])
* The implementation is done, but the last step from the [[#Restore_2 | restore process ]] is different:
** The <code>virsh restore</code> command is not executed with the <code>--xml</code> option, the XML from the state file is taken when restoring the machine. Therefore the conflicts are not properly resolved.
*** --[[User:Pat|Pat]] ([[User talk:Pat|talk]]) 09:41, 29 October 2013 (CET): Currently the [http://search.cpan.org/~danberr/Sys-Virt-1.1.3/lib/Sys/Virt.pm Sys::Virt] library does not support the --xml parameter when restoring a domain

=== How to manually restore a machine from backup ===
'''Important''': Before you continue with this guide, make sure that you have no other possibility to restore the machine. It might be easier and safer to get lost files from the online backup if the machine has one set up.

If you really have to restore the machine from the backup:
# Stop the machine from via the [https://cloud.stepping-stone.ch/vm-manager/ web interface]
# Login (as root) on the [[VM-Node]] the machine was running on

As a first step, you would like to set some useful bash variables to be able to copy paste the following guide:

'''Double check all variables you are setting here. If one is not correct, you will restore a running machine or overwrite a live-disk image!'''
<source lang='bash'>
machinename="<MACHINE-NAME>" # For example: machinename="b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6"
vmpool="<VM-POOL>" # For example vmpool="0f83f084-8080-413e-b558-b678e504836e"
vmtype="<VM-TYPE>" # For example vmtype="vm-persistent"
</source>
Change to the backup directory for the given machine and check the iterations:
<source lang='bash'>
cd /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}
ls -al
</source>
Change into the most recent iteration
<source lang='bash'>
cd 2014...
ls -al
</source>
In there you should have:
* The state file <MACHINE-NAME>.state.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.state.20140109T134445Z)
* The XML description <MACHINE-NAME>.xml.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.xml.20140109T134445Z)
* The ldif file <MACHINE-NAME>.ldif.<BACKUP-DATE> (for example b6dc3d27-5981-4b18-8f3f-31ed3d21a3c6.ldif.20140109T134445Z)
* And at least one disk image <DISK-IMAGE>.qcow2.<BACKUP-DATE> (for example 8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2.20140109T134445Z)
Now you should save the backup date and the disk image(s) in a variable
<source lang='bash'>
backupdate="<BACKUP-DATE>" # For example: backupdate="20140109T134445Z"
diskimage1="<DISK-IMAGE-1>.qcow2" # For example: diskimage1="8798561b-d5de-471b-a6fc-ec2b4831ed12.qcow2"
diskimage2="<DISK-IMAGE-2>.qcow2" # For example: diskimage2="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee.qcow2"
...
</source>

Have again a look at the different variables and '''double check them again'''
<source lang='bash'>
echo "Machine Name = ${machinename}"
echo "VM Pool = ${vmpool}"
echo "VM Type = ${vmtype}"
echo "Backup date = ${backupdate}"
echo "Disk Image 1 = ${diskimage1}"
echo "Disk Image 2 = ${diskimage2}"
...
</source>

Copy all these files to the retain location:
<source lang='bash'>
currentdate=`date --utc +'%Y%m%dT%H%M%SZ'`
mkdir -p /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.ldif.${backupdate} /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/
</source>



''' Now you are entering the critical part. You won't be able to undo the following steps'''

Check if there is a difference between the current LDAP entry and the one from the backup
<source lang='bash'>
domain="<DOMAIN>" # For example domain="stoney-cloud.org"
ldapbase="<LDAPBASE>" # For expample ldapbase="dc=stoney-cloud,dc=org"
ldapsearch -H ldaps://ldapm.${domain} -b "sstVirtualMachine=${machinename},ou=virtual machines,ou=virtualization,ou=services,${ldapbase}" -s sub -x -LLL -o ldif-wrap=no -D "cn=Manager,${ldapbase}" -W "(objectclass=*)" > /tmp/${machinename}.ldif
diff -Naur /tmp/${machinename}.ldif /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>
and '''edit the file at the retain location''' according to your needs.

If there are no differences (or the differences are not important) you can skip the following step. Otherwise use the [https://cloud.stepping-stone.ch/phpldapadmin PhpLdapAdmin] to delete the machine from the LDAP directory (do not forget to delete the dhcp entry <code>dn: cn=<MACHINE-NAME>,ou=virtual machines,cn=192.168.140.0,cn=config-01,ou=dhcp,ou=networks,ou=virtualization,ou=services,dc=stoney-cloud,dc=org</code>). Then add the LDIF (the one you just edited) to the LDAP (first do some general replacement)
<source lang='bash'>
sed -i\
-e 's/snapshotting/finished/'\
-e '/member.*/d'\
/var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}

/usr/bin/ldapadd -H "ldaps://ldapm.${domain}" -x -D "cn=Manager,${ldapbase}" -W -f /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}/${machinename}.ldif.${backupdate}
</source>

Undefine the machine
<source lang='bash'>
virsh undefine ${machinename}
</source>

Copy all the disk images from the backup location back to their original location
<source lang='bash'>
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage1}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage1}
cp -p /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${diskimage2}.${backupdate} /var/virtualization/${vmtype}/${vmpool}/${diskimage2}
...
</source>

And restore the domain from the state file from the backup location with the XML from the retain location (the one you might have edited)
<source lang='bash'>
virsh restore /var/backup/virtualization/${vmtype}/${vmpool}/${machinename}/${backupdate}/${machinename}.state.${backupdate}
</source>

Now the machine should be up and running again. Continuing where it was stopped when taking the backup.

If everything is OK, you can cleanup the created files and directories
<source lang='bash'>
rm -rf /var/virtualization/retain/${vmtype}/${vmpool}/${machinename}/${currentdate}
rm /tmp/${machinename}.ldif
</source>

== Next steps ==

[[Category: stoney conductor]]

stoney cloud: Multi-Node Installation

2013-09-06T09:28:46Z

David:

== Specialized Installation ==
=== Primary-Master-Node (vm-node-01) ===
If you configured a additional Backup Volume on the Storage Nodes, you want to mount them now in the VM-Node.

Log into the Primary-Master-Node and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node

==== Manual Steps ====
Install a transfer user.

<code>
lvcreate -L 60G -n transfer local0

mkfs.xfs -L "OSBD_transfe" /dev/local0/transfer

cat << EOF >> /etc/fstab

LABEL=OSBD_transfe /home/transfer xfs noatime,nodev,nosuid,noexec 0 2
EOF

mount /home/transfer
adduser -d transfer
# passwd transfer

</code>

[[Category:Documentation]]

stoney cloud: Multi-Node Installation

2013-09-06T07:44:29Z

David:

stoney cloud: Multi-Node Installation

2013-09-06T07:42:09Z

David: Created page with " Category:Documentation"

[[Category:Documentation]]

stoney backup: Server set-up

2013-08-28T14:12:33Z

David: /* sshd keys */

stoney backup: Server set-up

2013-08-28T14:12:12Z

David: /* sshd_config */