Changes

Install the [[stoney_backup:_prov-backup-rsnapshot | prov-backup-rsnasphot ]] daemon script using the package manager:

The global configuration for the provisioning daemon (which was installed with the first provisioning module and the <code>sys-apps/sst-provisioning</code> package) applies to all provisioning modules running on the server. This configuration therefore contains information about the provisioning daemon itself and no information at all about the specific modules. /etc/Provisioning/Global.conf

SERVER = ldaps://ldapm.stoney-cloudtombstone.org

Install the backup utils (multiple scripts which help you to manage and monitor your backup server and backup accounts) using the package manager. For more information about the scripts please see the [[stoney_backup:_Server_set-up#stoney_backup_Service_Software _Service_Software | stoney backup Service Software]] sectionpage.

Please refer to the configuration sections for the different scripts in [[stoney_backup:_Server_set-up#stoney_backup_Service_Software _Service_Software | stoney backup Service Software]]. = stoney backup Service Software =The stoney backup Service comes along with multiple scripts which help you to manage and monitor your backup server and accounts:  * '''writeAccountSize.pl''': Calculates the used disk space (backup and iterations) for a given account and writes the corresponding values to:** The LDAP backend (used by the selfcare webinterface to display quota information):*** Backup space used (sstBackupSize): The disk space the account uses for the backup itself (disk space used under the <code>incoming</code> folder of the users chroot-home directory)*** Snapshot space used (sstIncrementSize): The disk space the account uses for the iterations (disk space under the <code>.snapshot</code> folder of the users chroot-home directory)** The file <code>etc/backupSize</code> of the accounts chroot (used by the Sepiola Online Backup client): * '''snapshot.pl''': Is used for ... == Write Backup Account Size ==TBD: PKL (bitte bei allen Scripts angeben, aus welchem Päckli/ebuild es stammt). If you have already installed the [[Backup_(Server_Setup)#rsnapshot | rsnapshot]] script, you also have the writeAccountSize script. Otherwise follow [[Backup_(Server_Setup)#rsnapshot | these instructions (installation only)]] === Write Backup Account Size Configuration === /etc/backup-utils/writeAccountSize.conf<pre>[Global]INCOMING_DIRECTORY = /incomingACCOUNT_SIZE_FILE = /etc/backupSizeSNAPSHOTS = 1 [Syslog]SYSLOG = rsnapshot [Directory]LDAP_SERVER = ldaps://ldapm.tombstone.chLDAP_PORT = 636LDAP_BIND_DN = cn=Manager,dc=stoney-cloud,dc=orgLDAP_BIND_PW = <password>LDAP_BASE_DN = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=orgLDAP_PERSON_BASE = ou=people,dc=stoney-cloud,dc=orgLDAP_RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=orgLDAP_EMAIL_ATTRIBUTE = mail [Notification]EMAIL_SENDER = stepping stone GmbH Supprt <support@stepping-stone.ch>EMAIL_ALERT_THRESHOLD = 85 Salutation_Default_de-CH = Liebe Kundin / Lieber KundeSalutation_m_de-CH = Sehr geehrter HerrSalutation_f_de-CH = Sehr geehrte FrauSalutation_Default_en-GB = Dear customerSalutation_m_en-GB = Dear Mr.Salutation_f_en-GB = Dear Mrs. [MAIL]host = mail.stepping-stone.chport = 587username = support@stepping-stone.chpassword = <password></pre> === Write Backup Account Size Test === /usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d<pre>Debug modus was turned on Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incomingDebug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming existsTotal Quota: 1048576 kilobytesTotal used Space: 0 kilobytesIncoming Size: 0 kilobytesDebug sub getSnapshotsSize: $localUsedQuota: 0Debug sub getSnapshotsSize: $localSnapshotsSize: 0Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSizeDebug: wrote 1024 0 0 to /var/backup/080/4000080/etc/backupSizeDEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 0 DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0 Alert Threshold: 85 %Calculated value: 0</pre> Now write some data (200 megaytes in this example) into the users incoming directory and then execute the script again: dd if=/dev/zero of=/var/backup/080/4000080/home/4000080/incoming/test.zeros bs=1024k count=200 chown 4000080:4000080 /var/backup/080/4000080/home/4000080/incoming/test.zeros /usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d <pre>Debug modus was turned on Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incomingDebug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming existsTotal Quota: 1048576 kilobytesTotal used Space: 204800 kilobytesIncoming Size: 204800 kilobytesDebug sub getSnapshotsSize: $localUsedQuota: 204800Debug sub getSnapshotsSize: $localSnapshotsSize: 0Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSizeDebug: wrote 1024 200 0 to /var/backup/080/4000080/etc/backupSizeDEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200 DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0 Alert Threshold: 85 %Calculated value: 19.53125</pre>Everything seems to be working fine! == Snapshots ==We use rsnapshot - remote filesystem snapshot utility for the actual snapshots and a handful of wrapper scripts, that do things like:* Read the users and their settings from the LDAP directory.* Execute rsnapshot according to the users settings.* Write the backup quotas backup (incoming), iterations (.snapshots) and free space to the users local backupSize file and update the LDAP directory.* Inform the reseller, customer or user (depending on the settings in the LDAP directory) via mail, if the quota limit has been reached.* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time. === rsnaphot configuration directory ===The users individual rsnapshot configurations are stored under <code>/etc/rsnapshot</code>. Please make sure, that the directory exists: ls -al /etc | grep rsnapshot  drwx------ 2 root root 64 30. Aug 20:20 rsnapshot If not, create it: mkdir /etc/rsnapshot chmod 700 /etc/rsnapshot === snapshot.pl Configuration ===The snapshot.pl script is responsible for the execution of rsnapshot according to the users settings. /etc/backup-utils/snapshot.conf <pre>[General]MaxParallelProcesses = 5Rsnapshot_command = /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.%uid% %interval% [LDAP]Host = ldaps://ldapm.tombstone.chPort = 636User = cn=Manager,dc=stoney-cloud,dc=orgPassword = <Password>CA_Path = /etc/ssl/certsAccounts_Base = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org</pre> Legend:* '''%uid%''': The backup account and login uid as a numeric number. For example: 4000205.* '''%interval%''': The backup level to be executed. Possible values are hourly, daily, weekly, monthly and yearly. === snapshot.pl Test ===Before adding the necessary cronjob entries, we need to make sure, that we've configured the snapshot.pl script correctly: /usr/libexec/backup-utils/snapshot.pl --interval daily -d If everything worked as planned, you should receive feedback looking roughly like:<pre>INFO: Starting rsnapshot for interval daily with maximum 5 parallel processes INFO: Executing snapshot for 4000080 INFO: Executing snapshot for 4000079 INFO: Snapshot process for 4000079 finished in 0.18 seconds with status 0 INFO: Snapshot process for 4000080 finished in 0.19 seconds with status 0 INFO: rsnapshot for all backups done. Took 0.24 seconds</pre> Just to make sure, that everything did work out fine, execute <code>writeAccountSize.pl</code> againg: /usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d<pre>Debug modus was turned on Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incomingDebug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming existsTotal Quota: 1048576 kilobytesTotal used Space: 409600 kilobytesIncoming Size: 204800 kilobytesDebug sub getSnapshotsSize: $localUsedQuota: 409600Debug sub getSnapshotsSize: $localSnapshotsSize: 204800Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSizeDebug: wrote 1024 200 200 to /var/backup/080/4000080/etc/backupSizeDEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200 DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 209715200 Alert Threshold: 85 %Calculated value: 39.0625</pre> As you can see, the total used space has risen to 39.0625. === Cronjobs ===After making sure, that everything worked as planned, you can update your crontab entry: crontab -e<pre>...# Rsnapshot for all users30 22 * * * /usr/libexec/backup-utils/snapshot.pl --interval daily15 22 * * sun /usr/libexec/backup-utils/snapshot.pl --interval weekly00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly...</pre> == schedule warning ==To install the new schedule warning script you have to execute the following commands:<pre>cd /var/work/git clone --recursive https://github.com/stepping-stone/backup-surveillance.gitcd backup-surveillance/bin/ln -s ../perl-utils/lib/PerlUtil/ PerlUtil</pre> === Configuration === vi /var/work/backup-surveillance/etc/config.conf<pre>[XML]SCHEDULE_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/scheduler.xmlSCHEDULE_XSD = %configpath%/../etc/schema/scheduler_schema.xsdBACKUP_ENDED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupEnded.xmlBACKUP_ENDED_XSD = %configpath%/../etc/schema/backupended_schema.xsdBACKUP_STARTED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupStarted.xmlBACKUP_STARTED_XSD = %configpath%/../etc/schema/backupstarted_schema.xsd  [TEMPLATE]Salutation_Default_de-CH = Liebe Kundin / Lieber KundeSalutation_m_de-CH = Sehr geehrter HerrSalutation_f_de-CH = Sehr geehrte FrauSalutation_Default_en-GB = Dear customerSalutation_m_en-GB = Dear Mr.Salutation_f_en-GB = Dear Mrs. [LDAP] SERVER = ldaps://ldapm.tombstone.chPORT = 636DEBUG = 1 ADMIN_DN = cn=Manager,dc=foss-cloud,dc=org ADMIN_PASSWORD = <Password> BACKUP_BASE = ou=accounts,ou=backup,ou=services,dc=foss-cloud,dc=orgPEOPLE_BASE = ou=people,dc=foss-cloud,dc=orgRESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=foss-cloud,dc=orgSCOPE = sub [MAIL]mailTo = host = mail.stepping-stone.chport = 587username = password = from =  </pre>