stoney backup: Service Software - Revision history

Pat: /* writeAccountSize.pl */

2014-06-26T12:15:27Z

writeAccountSize.pl

← Older revision		Revision as of 14:15, 26 June 2014
Line 13:		Line 13:
	== writeAccountSize.pl ==		== writeAccountSize.pl ==
	This script is installed to <code>/usr/libexec/backup-utils/writeAccountSize.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:		This script is installed to <code>/usr/libexec/backup-utils/writeAccountSize.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
	~~@PAT: Please add a link to the current Git repository. --[[User:Chrigu\|Chrigu]] ([[User talk:Chrigu\|talk]]) 10:15, 19 June 2014 (CEST)~~
	* Calculates the used disk space (backup and iterations) for a given account and writes the corresponding values to:		* Calculates the used disk space (backup and iterations) for a given account and writes the corresponding values to:
	*# The LDAP backend (used by the selfcare webinterface to display quota information):		*# The LDAP backend (used by the selfcare webinterface to display quota information):

Pat: /* rsnapshot wrapper bash scripts */

2014-06-26T07:28:46Z

rsnapshot wrapper bash scripts

@@ Line 738: / Line 738: @@
 </pre>
 == rsnapshot wrapper bash scripts ==
 There are two rsnapshot wrapper scripts (these are bash scripts), they are installed by the <code>sys-apps/sst-backup-utils</code> package:

Pat: /* Cronjobs */

2014-06-26T07:06:45Z

Cronjobs

← Older revision		Revision as of 09:06, 26 June 2014
Line 352:		Line 352:

	As you can see, the total used space has risen to 39.0625.		As you can see, the total used space has risen to 39.0625.

	~~=== Cronjobs ===~~
	~~After making sure, that everything worked as planned, you can update your crontab entry:~~
	~~crontab -e~~
	~~<pre>~~
	~~...~~
	~~# Rsnapshot for all users~~
	~~30 22 * * * /usr/libexec/backup-utils/snapshot.pl --interval daily~~
	~~15 22 * * sun /usr/libexec/backup-utils/snapshot.pl --interval weekly~~
	~~00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly~~
	~~...~~
	~~</pre>~~

	* TBD: Maybe this is not optimal if there is a lot of data to rotate. In this case, it might be that for example weekly and daily snapshot both run at the same time what might lead to strange results.
	* Workarounds:
	** Simple/short version: Instead one could create another wrapper script which is called everyday and does some simple logic:
	** Is today the first day of a month?
	*** Yes: Is today sunday?
	**** Yes: Execute monthly then weekly then daily rsnapshots using the snapshot.pl script (but wait for each interval to finish before starting the next)
	**** No: Execute monthly then daily rsnapshots using the snapshot.pl script (but wait for monthly interval to finish before starting the daily)
	*** No: Is today sunday?
	**** Yes: Execute weekly then daily rsnapshots using the snapshot.pl script (but wait for weekly interval to finish before starting the daily)
	**** No: Execute daily rsnapshots using the snapshot.pl script
	** Complex/long version: Adapt the snapshot.pl script and call it every day without interval parameter. The script does the logic described above.
	~~This avoids the above mentioned collision~~

	~~(Complex) / short version: rule via cronjobs:~~
	~~<pre>~~
	~~# Rsnapshot for all users~~
	~~30 22 * * 1-6 [ $( date +\%d ) -gt 1 ] && /usr/libexec/backup-utils/snapshot.pl --interval daily~~
	~~15 22 * * 7 [ $( date +\%d ) -gt 1 ] && /usr/libexec/backup-utils/snapshot.pl --interval weekly && /usr/libexec/backup-utils/snapshot.pl --interval daily~~
	~~00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly && /usr/libexec/backup-utils/snapshot.pl --interval weekly && /usr/libexec/backup-utils/snapshot.pl --interval daily~~
	~~...~~
	~~</pre>~~

	== scheduleWarning.pl ==		== scheduleWarning.pl ==

Pat: /* stoney backup Service Software */

2014-06-19T15:14:58Z

stoney backup Service Software

← Older revision		Revision as of 17:14, 19 June 2014
Line 9:		Line 9:
	* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.		* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.

	All these scripts are installed to <code>/usr/libexec/backup-utils/</code> by the <code>sys-apps/sst-backup-utils</code> package (see [[stoney_backup:_Server_set-up#backup_utils \| backup-utils]]).		All these scripts are installed to <code>/usr/libexec/backup-utils/</code> by the <code>sys-apps/sst-backup-utils</code> package (see [[stoney_backup:_Server_set-up#backup_utils \| backup-utils]]). The source code can be found on [https://github.com/stepping-stone/backup-utils github]

	== writeAccountSize.pl ==		== writeAccountSize.pl ==

Pat: /* stoney backup Service Software */

2014-06-19T10:05:38Z

stoney backup Service Software

← Older revision		Revision as of 12:05, 19 June 2014
Line 9:		Line 9:
	* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.		* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.

	All these scripts are installed to <code>/usr/libexec/backup-utils/code> by the <code>sys-apps/sst-backup-utils</code> package (see [[stoney_backup:_Server_set-up#backup_utils \| backup-utils]]).		All these scripts are installed to <code>/usr/libexec/backup-utils/</code> by the <code>sys-apps/sst-backup-utils</code> package (see [[stoney_backup:_Server_set-up#backup_utils \| backup-utils]]).

	== writeAccountSize.pl ==		== writeAccountSize.pl ==

Pat: /* stoney backup Service Software */

2014-06-19T10:05:23Z

stoney backup Service Software

← Older revision		Revision as of 12:05, 19 June 2014
Line 9:		Line 9:
	* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.		* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.

	All these scripts are installed to <code>/usr/libexec/backup-utils~~/writeAccountSize.pl<~~/code> by the <code>sys-apps/sst-backup-utils</code> package (see [[stoney_backup:_Server_set-up#backup_utils \| backup-utils ]]).		All these scripts are installed to <code>/usr/libexec/backup-utils/code> by the <code>sys-apps/sst-backup-utils</code> package (see [[stoney_backup:_Server_set-up#backup_utils \| backup-utils]]).

	== writeAccountSize.pl ==		== writeAccountSize.pl ==

Pat: /* stoney backup Service Software */

2014-06-19T10:04:58Z

stoney backup Service Software

← Older revision		Revision as of 12:04, 19 June 2014
Line 8:		Line 8:
	* Inform the reseller, customer or user (depending on the settings in the LDAP directory) via mail, if the quota limit has been reached.		* Inform the reseller, customer or user (depending on the settings in the LDAP directory) via mail, if the quota limit has been reached.
	* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.		* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.

			All these scripts are installed to <code>/usr/libexec/backup-utils/writeAccountSize.pl</code> by the <code>sys-apps/sst-backup-utils</code> package (see [[stoney_backup:_Server_set-up#backup_utils \| backup-utils ]]).

	== writeAccountSize.pl ==		== writeAccountSize.pl ==

Pat: Created page with "= stoney backup Service Software = The stoney backup Service comes along with multiple scripts which help you to manage and monitor your backup server and accounts: We use r..."

2014-06-19T10:01:50Z

Created page with "= stoney backup Service Software = The stoney backup Service comes along with multiple scripts which help you to manage and monitor your backup server and accounts: We use r..."

New page

= stoney backup Service Software =
The stoney backup Service comes along with multiple scripts which help you to manage and monitor your backup server and accounts:

We use rsnapshot - remote filesystem snapshot utility for the actual snapshots and a handful of wrapper scripts, that do things like:
* Read the users and their settings from the LDAP directory.
* Execute rsnapshot according to the users settings.
* Write the backup quotas backup (incoming), iterations (.snapshots) and free space to the users local backupSize file and update the LDAP directory.
* Inform the reseller, customer or user (depending on the settings in the LDAP directory) via mail, if the quota limit has been reached.
* Depending on the users settings in the LDAP directory, warning mail will be sent to the reseller, customer or user, if a backup was not executed on time.

== writeAccountSize.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeAccountSize.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
@PAT: Please add a link to the current Git repository. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 10:15, 19 June 2014 (CEST)
* Calculates the used disk space (backup and iterations) for a given account and writes the corresponding values to:
*# The LDAP backend (used by the selfcare webinterface to display quota information):
*#* Backup space used (sstBackupSize): The disk space the account uses for the backup itself (disk space used under the <code>incoming</code> folder of the users chroot-home directory)
*#* Snapshot space used (sstIncrementSize): The disk space the account uses for the iterations (disk space under the <code>.snapshot</code> folder of the users chroot-home directory)
*# The file <code>etc/backupSize</code> of the accounts chroot (used by the Sepiola Online Backup client):
* Checks if the user and/or reseller must be notified that there is no more disk space left for the given account
** Checks if the notification flag was passed, if not no notification will be triggered
** Calculates the used disk space (backup and iterations) in percentage
** Reads the notification threshold value from the LDAP backed
** If the disk space used (in percentage) is bigger than the value retrieved from the LDAP backend start the [[stoney_cloud:_Notification_Architecture | notification]] process with
*** Product: Given account UID
*** Service: Backup
*** Problem: Quota
* Pod documentation:
<pre>
NAME
writeAccountSize.pl

DESCRIPTION
This Script gets quota information from filesystem, size of incoming and
snapshots directories, write the data to a file and the LDAP backend and
sends an e-mail message for each account that is over quota to users
e-mail address (from ldap directory) if notification flag is passed.

The configuration file for this script is stored in the backup-utils
configuration directory (/etc/backup-utils/) and is called
writeAccountSize.conf.

The script needs access to the quota program to get quota information.
The script needs ldap access to get users e-mail address and quota
information. The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

-C configfile
The configuration file.

-U uid
The user id.

-n notification
Start notification process if quota threshold is reached

-d debug
Turns the debug mode on.

-h help
This online help.

USAGE
writeAccountSize.pl -U uid [-C configuration file ] []-d debug] [-h
help] [-n]

CREATED
2009-04-16 michael.rhyner@stepping-stone.ch created

VERSION
2009-04-16 michael.rhyner@stepping-stone.ch created
2009-04-30 michael.rhyner@stepping-stone.ch changed position based quota
output parsing with correctly parsed elements
2009-06-15 michael.rhyner@stepping-stone.ch added over quota check and
sending e-mail
2009-06-16 michael.rhyner@stepping-stone.ch renamed script and make it
more general usable (e.g. for online backup, online storage, ...)
2009-06-17 michael.rhyner@stepping-stone.ch changed mail message to read
from a text file instead from configuration parameter
2009-06-18 michael.rhyner@stepping-stone.ch corrected wrong regex to
weed out the asterisk (*) in getQuotaSize
2009-06-19 michael.rhyner@stepping-stone.ch corrected wrong evaluation
success from subroutines and avoid message output when not in debug mode
2009-06-22 michael.rhyner@stepping-stone.ch getQuotaSize: return
immediately if no quota was set
2009-06-24 michael.rhyner@stepping-stone.ch alert when used certain
percentge of allowed space instead of more than allowed space
2009-06-26 michael.rhyner@stepping-stone.ch values are presented in
Gigabytes within notification message
2009-07-23 michael.rhyner@stepping-stone.ch corrected wrong syslog
severities for errors
2009-07-24 michael.rhyner@stepping-stone.ch made e-mail address
available within message body
2013-08-19 pat.klaey@stepping-stone.ch write quota values also to the
LDAP

INCORPORATED CODE
Incorporate code with use:

warnings;
strict;
Config::IniFiles;
Getopt::Std;
Sys::Syslog;
File::Basename;
Text::Template;
POSIX;
Notification;
PerlUtil::Logging;
PerlUtil::LDAPUtil;
</pre>

=== Configuration ===
/etc/backup-utils/writeAccountSize.conf
<pre>
[Global]
INCOMING_DIRECTORY = /incoming
ACCOUNT_SIZE_FILE = /etc/backupSize
SNAPSHOTS = 1

[Syslog]
SYSLOG = rsnapshot

[Directory]
LDAP_SERVER = ldaps://ldapm.tombstone.ch
LDAP_PORT = 636
LDAP_BIND_DN = cn=Manager,dc=stoney-cloud,dc=org
LDAP_BIND_PW = <password>
LDAP_BASE_DN = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
LDAP_PERSON_BASE = ou=people,dc=stoney-cloud,dc=org
LDAP_RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=org
LDAP_EMAIL_ATTRIBUTE = mail

[Notification]
EMAIL_SENDER = stepping stone GmbH Supprt <support@stepping-stone.ch>
EMAIL_ALERT_THRESHOLD = 85

Salutation_Default_de-CH = Liebe Kundin / Lieber Kunde
Salutation_m_de-CH = Sehr geehrter Herr
Salutation_f_de-CH = Sehr geehrte Frau
Salutation_Default_en-GB = Dear customer
Salutation_m_en-GB = Dear Mr.
Salutation_f_en-GB = Dear Mrs.
</pre>

=== Tests ===
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d
<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 0 kilobytes
Incoming Size: 0 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 0
Debug sub getSnapshotsSize: $localSnapshotsSize: 0
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 0 0 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 0

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0

Alert Threshold: 85 %
Calculated value: 0
</pre>

Now write some data (200 megaytes in this example) into the users incoming directory and then execute the script again:
dd if=/dev/zero of=/var/backup/080/4000080/home/4000080/incoming/test.zeros bs=1024k count=200
chown 4000080:4000080 /var/backup/080/4000080/home/4000080/incoming/test.zeros
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d

<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 204800 kilobytes
Incoming Size: 204800 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 204800
Debug sub getSnapshotsSize: $localSnapshotsSize: 0
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 200 0 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 0

Alert Threshold: 85 %
Calculated value: 19.53125
</pre>
Everything seems to be working fine!

== snapshot.pl ==
This script is installed to <code>/usr/libexec/backup-utils/snapshot.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* Read interval parameter value passed
** The interval parameter value can be daily, weekly, monthly (or yearly)
* Find all backup active accounts for which the rsnapshot command must be executed (depending on the given interval)
** Filter to find these accounts: <code>(&(sstBackupInterval<INTERVAL>=*)(sstIsActive=TRUE))</code> for example for the daily rsnapshot the filter is <code>(&(sstBackupIntervalDaily=*)(sstIsActive=TRUE))</code>
*** In other words this means: Get me all acounts that have for <code>sstBackupInterval<INTERVAL></code> a value defined AND <code>sstIsActive</code> is set to "TRUE"
* According to the interval given and the account UID calculate the rsnapshot command for all these accounts
** For example
*** Account UID: 4000000
*** Interval: daily
*** Resulting rsnapshot command: /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.4000000 daily (if you use the [[#snapshot.pl_Configuration | configuration]] below)
* Execute all these commands
** Use controlled parallel execution, you can specify how many commands can be executed in parallel (see [[#snapshot.pl_Configuration | configuration]] below)
* Pod documentation:
<pre>
NAME
snapshot.pl

DESCRIPTION
This script gets all active online backup accounts from the LDAP backend
for which the rsnapshot process for the given interval must be executed.
According to these accounts and the given interval, the commands to be
executed are generated and finally executed. The commands can be
executed in parallel, however there is a limit defined in the
configuration file which limits the amount of parallel running
processes.

USAGE
./snapshot.pl --interval interval [--debug] [--help]

OPTIONS
--interval/-i interval
Specifies the rsnapshot interval (can be hourly, daily, weekly,
monthly or yearly)

--debug/-d
Turns on debug mode

--help/-h
Shows this help

CREATED
2012-03-19 pat.klaey@stepping-stone.ch created

VERSION
2012-03-19 pat.klaey@stepping-stone.ch created

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
Sys::Syslog;
PerlUtil::Logging;
PerlUtil::LDAPUtil;
File::Basename;
Parallel::ForkManager;
Time::Stopwatch;
</pre>

=== Configuration ===
There are two things to do in this step:
* Configure the rsnapshot root directory
* Configure the snapshot.pl script itself

==== rsnaphot configuration directory ====
The users individual rsnapshot configurations are stored under <code>/etc/rsnapshot</code>. Please make sure, that the directory exists:
ls -al /etc | grep rsnapshot

drwx------ 2 root root 64 30. Aug 20:20 rsnapshot

If not, create it:
mkdir /etc/rsnapshot
chmod 700 /etc/rsnapshot

==== snapshot.pl Configuration ====
The snapshot.pl script is responsible for the execution of rsnapshot according to the users settings.
/etc/backup-utils/snapshot.conf
<pre>
[General]
MaxParallelProcesses = 5
Rsnapshot_command = /usr/bin/nice -n 19 /usr/bin/rsnapshot -c /etc/rsnapshot/rsnapshot.conf.%uid% %interval%

[LDAP]
Host = ldaps://ldapm.tombstone.ch
Port = 636
User = cn=Manager,dc=stoney-cloud,dc=org
Password = <Password>
CA_Path = /etc/ssl/certs
Accounts_Base = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
</pre>

Legend: At runtime the following placeholders are replaced as follows
* '''%uid%''': The backup account and login uid as a numeric number. For example: 4000205.
* '''%interval%''': The backup level to be executed. Possible values are hourly, daily, weekly, monthly and yearly.

=== Tests ===
Before adding the necessary cronjob entries, we need to make sure, that we've configured the snapshot.pl script correctly:
/usr/libexec/backup-utils/snapshot.pl --interval daily -d

If everything worked as planned, you should receive feedback looking roughly like:
<pre>
INFO: Starting rsnapshot for interval daily with maximum 5 parallel processes

INFO: Executing snapshot for 4000080

INFO: Executing snapshot for 4000079

INFO: Snapshot process for 4000079 finished in 0.18 seconds with status 0

INFO: Snapshot process for 4000080 finished in 0.19 seconds with status 0

INFO: rsnapshot for all backups done. Took 0.24 seconds
</pre>

Just to make sure, that everything did work out fine, execute <code>writeAccountSize.pl</code> again:
/usr/libexec/backup-utils/writeAccountSize.pl -U 4000080 -d
<pre>
Debug modus was turned on

Debug sub checkUsersHomeDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersHomeDirectory: The $localUsersHomeDirectory /var/backup/080/4000080/home/4000080 exists

Debug sub checkUsersIncomingDirectory: $localUsersHomeDirectory: /var/backup/080/4000080/home/4000080
Debug sub checkUsersIncomingDirectory: $localUsersIncomingDirectory: /incoming
Debug sub checkUsersIncomingDirectory: $localIncomingPath: /var/backup/080/4000080/home/4000080/incoming

Debug sub checkUsersIncomingDirectory: The $localIncomingPath /var/backup/080/4000080/home/4000080/incoming exists
Total Quota: 1048576 kilobytes
Total used Space: 409600 kilobytes
Incoming Size: 204800 kilobytes
Debug sub getSnapshotsSize: $localUsedQuota: 409600
Debug sub getSnapshotsSize: $localSnapshotsSize: 204800
Debug writeAccountSize: Working on /var/backup/080/4000080/etc/backupSize
Debug: wrote 1024 200 200 to /var/backup/080/4000080/etc/backupSize
DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstBackupSize => 209715200

DEBUG: Successfully executed the following modifications for entry uid=4000080,ou=accounts,ou=backup,ou=services,o=stepping-stone,c=ch: sstIncrementSize => 209715200

Alert Threshold: 85 %
Calculated value: 39.0625
</pre>

As you can see, the total used space has risen to 39.0625.

=== Cronjobs ===
After making sure, that everything worked as planned, you can update your crontab entry:
crontab -e
<pre>
...
# Rsnapshot for all users
30 22 * * * /usr/libexec/backup-utils/snapshot.pl --interval daily
15 22 * * sun /usr/libexec/backup-utils/snapshot.pl --interval weekly
00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly
...
</pre>

* TBD: Maybe this is not optimal if there is a lot of data to rotate. In this case, it might be that for example weekly and daily snapshot both run at the same time what might lead to strange results.
* Workarounds:
** Simple/short version: Instead one could create another wrapper script which is called everyday and does some simple logic:
** Is today the first day of a month?
*** Yes: Is today sunday?
**** Yes: Execute monthly then weekly then daily rsnapshots using the snapshot.pl script (but wait for each interval to finish before starting the next)
**** No: Execute monthly then daily rsnapshots using the snapshot.pl script (but wait for monthly interval to finish before starting the daily)
*** No: Is today sunday?
**** Yes: Execute weekly then daily rsnapshots using the snapshot.pl script (but wait for weekly interval to finish before starting the daily)
**** No: Execute daily rsnapshots using the snapshot.pl script
** Complex/long version: Adapt the snapshot.pl script and call it every day without interval parameter. The script does the logic described above.
This avoids the above mentioned collision

(Complex) / short version: rule via cronjobs:
<pre>
# Rsnapshot for all users
30 22 * * 1-6 [ $( date +\%d ) -gt 1 ] && /usr/libexec/backup-utils/snapshot.pl --interval daily
15 22 * * 7 [ $( date +\%d ) -gt 1 ] && /usr/libexec/backup-utils/snapshot.pl --interval weekly && /usr/libexec/backup-utils/snapshot.pl --interval daily
00 22 1 * * /usr/libexec/backup-utils/snapshot.pl --interval monthly && /usr/libexec/backup-utils/snapshot.pl --interval weekly && /usr/libexec/backup-utils/snapshot.pl --interval daily
...
</pre>

== scheduleWarning.pl ==
This script is installed to <code>/usr/libexec/backup-utils/scheduleWarning.pl</code> by the <code>sys-apps/sst-backup-utils</code> package and does the following:
* The basic task of this script is simple: For the given account
** Check if the planned backups were started
*** If not, start the [[stoney_cloud:_Notification_Architecture | notification]] process with
**** Product: Given account UID
**** Service: Backup
**** Problem: Schedule
** Check if the planned backups finished successfully
*** If not, start the [[stoney_cloud:_Notification_Architecture | notification]] process with
**** Product: Given account UID
**** Service: Backup
**** Problem: Unsuccessful

As the backup clients distributed by stepping stone GmbH upload metadata XML before (scheduling information and start time) and after (end time and backup status) the actual backup, the scheduleWarning.pl script is able to verify whether or not a planned backup has been executed and whether or not the backup was successful.

* Pod documentation
<pre>
NAME
scheduleWarning.pl

DESCRIPTION
This script tests whether a planed backup was successful or not. There
are two different typs of failure.

1. The Backup did not start
The first type of error is that a backup is scheduled for YYYY-MM-DD
at HH:MM, but the backup don't start at this specified time. A
possible reason could be that the computer was shut down.

2.The backup was not successful
The second type of error is that a backup started as scheduled but
did not finish successfully. There are different reasons for this
error.

If a backup wasn't successful the script checks which type of error
occured. It reads the XML files which are stored on the server and
compares the given information. If the error is detected, the script
stats the norification process with the information of the error.

If a user has more than one computer backed-up, the script tests one
computer after the other. The mail(s) sent by the programm also
contain(s) the information which computer is affected.

OPTION
-U uid
The -U option is required to run the script, it indicates for which
uid the script is executed.

USAGE
scheduleWarning.pl [-U user]

CREATED
2010-04-14 created Pat Kläy <pat.klaey@stepping-stone.ch>

VERSION
2010-04-14 v0.01, created pkl
2010-08-24 v0.02, modified pkl
New using Net::SMTP::TLS to send mails

2013-09-13 v0.03, modified pat.klaey@stepping-stone.ch
Changes to use the script with the new backup infrastructure (read
more information from LDAP, use Notification lib to send mails)

USES
strict;
warnings;
XML::Simple;
Config::IniFiles;
XML::Validator::Schema;
Date::Calc qw(:all);
Date::Manip;
Schedule::Cron::Events;
DateTime::Format::Strptime;
Sys::Syslog;
XML::SAX::ParserFactory;
Getopt::Std;
MIME::Base64;
Authen::SASL;
Net::LDAPS;
Net::SMTP::TLS;
Cwd 'abs_path';
PerlUtil::Logging;
PerlUtil::LDAPUtil;
</pre>

=== Configuration ===
/etc/backup-utils/scheduleWarning.conf
<pre>
[Backup]
CHROOT_DIRECTORY = /var/backup/%lastthree%/%user%

[XML]
SCHEDULE_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/scheduler.xml
SCHEDULE_XSD = /etc/backup-utils/schema/scheduler_schema.xsd
BACKUP_ENDED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupEnded.xml
BACKUP_ENDED_XSD = /etc/backup-utils/schema/backupended_schema.xsd
BACKUP_STARTED_FILE = %homeDirectory%/incoming/%computerName%/.sepiola_backup/backupStarted.xml
BACKUP_STARTED_XSD = /etc/backup-utils/schema/backupstarted_schema.xsd

[TEMPLATE]
Salutation_Default_de-CH = Liebe Kundin / Lieber Kunde
Salutation_m_de-CH = Sehr geehrter Herr
Salutation_f_de-CH = Sehr geehrte Frau
Salutation_Default_en-GB = Dear customer
Salutation_m_en-GB = Dear Mr.
Salutation_f_en-GB = Dear Mrs.

[LDAP]

SERVER = ldaps://ldapm.tombstone.ch
PORT = 636
DEBUG = 1

ADMIN_DN = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <Password>

BACKUP_BASE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
PEOPLE_BASE = ou=people,dc=stoney-cloud,dc=org
RESELLER_BASE = ou=reseller,ou=configuration,ou=backup,ou=services,dc=stoney-cloud,dc=org
SCOPE = sub

</pre>

== checkBackups.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. It is a wrapper around the above mentioned <code>writeAccountSize.pl</code> and <code>scheduleWarning.pl</code> scripts (if you configure it accordingly).
* As mentioned the script is a wrapper for calling the quota and scheduler information scripts:
** It gets all active backup accounts from the LDAP directory
** According to the CLI parameters passed it calles for all these accounts:
*** The quota checking script (this script can be defined in the configuration file, see [[stoney_backup:_Server_set-up#Configuration_6 | configuration]] section below)
*** The scheduler checking script (this script can be defined in the configuration file, see [[stoney_backup:_Server_set-up#Configuration_6 | configuration]] section below)
* Pod documentation:
<pre>
NAME
checkBackups.pl

DESCRIPTION
This script processes all active backup accounts and checks (according
to the command line options) the quota and/or the scheduled backups for
the given accounts.

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

OPTIONS
--schedule/-s
Checks schedule information by calling the defined script

--quota/-s
Checks the quota values by calling the defined script

--notify/-n
Passes the notify flag to the called scripts

USAGE
./checkBackups.pl [--quota] [--schedule] [--notify]

./checkBackups.pl --quota
Checks and writes the quota for all active backup accounts. The user
and/or reseller will NOT be informed if any quota threshold is
reached

./checkBackups.pl --quota --notify
Checks and writes the quota for all active backup accounts. The user
and/or reseller will be informed if any quota threshold is reached.

./checkBackups.pl --schedule
Checkes all active backup account if the scheduled backup was
executed and successful. The --schedule option will allways trigger
a notification mail to the user and/or reseller.

./checkBackups.pl --schedule --quota
Checks both, quota and schedule, for all active backup accounts.
Only schedule will trigger notification mails.

./checkBackups.pl --schedule --quota --notify
Checks both, quota and schedule, for all active backup accounts.
Both, quota and schedule, will trigger notification mails.

CREATED
2013-09-17 pat.klaey@stepping-stone.ch created

VERSION
2013-09-17 pat.klaey@stepping-stone.ch created
2013-11-19 pat.klaey@stepping-stone.ch Added options to be able to check
quota, schedule or both in one run

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
Sys::Syslog;
Cwd 'abs_path';
File::Basename;
PerlUtil::LDAPUtil;
PerlUtil::Logging;
</pre>

=== Configuration ===
In the configuration you have to define two simple things:
# Which script to call for quota checks and scheduler checks
# How to access the LDAP backend
/etc/backup-utils/checkBackups.conf
<pre>
[Scripts]
CheckQuotaScript = /usr/libexec/backup-utils/writeAccountSize.pl
CheckScheduleScript = /usr/libexec/backup-utils/scheduleWarning.pl

[LDAP]
Server = ldaps://ldapm.tombstone.ch
Port = 636
Username = cn=Manager,dc=stoney-cloud,dc=org
Password = <PASSWORD>
AccountBase = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
</pre>

==== Cronjobs ====
As you can pass the notification flag to the scripts, you can call the quota script say all 10 minutes. This makes sense if followed the configuration section, as the writeAccountSize.pl script writes the quota values to the LDAP backend which will then be visible in the selfcare webinterface. Like that you have a more or less up to date quota representation in the selfcare webinterface. On the other side, you don't want to send notification mails to the user every 10 minutes. It might be enough to send them once a day, so the cronjobs may look like:
<pre>
# Check backups
15 12 * * * /usr/libexec/backup-utils/checkBackups.pl --schedule --quota --notify

# Write quota values every 10 minutes
*/10 * * * * /usr/libexec/backup-utils/checkBackups.pl --quota
</pre>

== ResellerBackupBilling.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. The pod documentation says all what needs to be said about this script.
* Pod documentation
<pre>
NAME
ResellerBackupBilling.pl

DESCRIPTION
This script processes the reseller with the passed UID and creates a
very simplistic backup-billing overview which then is sent to the
address specified in the configuration file. The billing overview
contains all backup accounts which belong to the given reseller as well
as their quota. According to the billing plan specified in the
configuration file, the script also calculates the price for each backup
account.

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

OPTIONS
--reseller/-r UID
Process the reseller with this UID

--debug/-d
Turn on debug mode

--help/-h
Display this help

USAGE
./ResellerBackupBilling.pl --reseller UID [--debug] [--help]

./ResellerBackupBilling.pl --reseller 2000000
This processes the reseller with the UID 200000 and sends the
billing overview to the address specified in the configuration file

CREATED
2014-01-24 pat.klaey@stepping-stone.ch created

VERSION
2014-01-24 pat.klaey@stepping-stone.ch created

INCORPORATED CODE
Incorporated code with use:

warnings;
strict;
Getopt::Long;
PerlUtil::LDAPUtil;
PerlUtil::Logging;
PerlUtil::Mail;
Number::Format;
File::Basename;
Cwd 'abs_path';
</pre>
=== Configuration ===
You have to configure basically three things:
# How to access the LDAP directory (similar to the other backup-util scripts)
# Define a pricing schema
#* Define a base price for an account
#* Define a GB price
#* The total price for each account will finall be: <code>base price + ( quota in GB * GB price )</code>
# How to send the mail
#* You need to enter a valid (outgoing) mail server and corresponding port
#* User and password to authenticate on the mail server
#* A valid recipient (typically the company’s backoffice address) and sender

/etc/backup-utils/ResellerBackupBilling.conf
<pre>
[LDAP]
Server = ldaps://ldapm.tombstone.ch
Port = 636
Username = cn=Manager,dc=stoney-cloud,dc=org
Password = <Password>
AccountBase = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
CustomerBase = ou=customers,dc=stoney-cloud,dc=org
ResellerBase = ou=reseller,dc=stoney-cloud,dc=org

[Pricing]
AccountBasePrice = 5.00
AccountGBPrice = 0.50

[Mail]
Server = <Mail-Server>
Port = 587
Username = <Sender-Email-Address>
Password = <Password>
To = <Backoffice>
From = Billing Script <E-Mail-Address>
</pre>

== writeDate.pl ==
This script is installed to <code>/usr/libexec/backup-utils/writeDate.pl</code> by the <code>sys-apps/sst-backup-utils</code> package. This is a very simple script, it simply writes the current date and time to a file on the backup server before the rsnapshots rotates the backup. This date is used by the Sepiola Online Backup Client to display the iterations and their dates.
* Pod documentation:
<pre>
NAME
writeDate.pl

DESCRIPTION
This script writes the current date and time into the file for each
backup before the rsnapshots rotates the backup. This information is
used by the Sepiola Online Backup Client.

The configuration file for this script is stored under
/etc/backup-utils/writeDate.conf

The script uses syslog for logging purposes.

Command Line Interface (CLI) parameters:

-C configfile
The configuration file.

-U uid
The user id.

-d debug
Turns the debug mode on.

-h help
This online help.

USAGE
writeDate.pl [-C configuration_file] [-U uid] [-d debug] [-h help]

CREATED
2007-09-16 michael.eichenberger@stepping-stone.ch created

VERSION
2013-09-01 michael.eichenberger@stepping-stone.ch updated to reflect
changes in the new backup environment

2007-09-16 michael.eichenberger@stepping-stone.ch created

INCORPORATED CODE
Incorporate code with use:

warnings;
strict;
Config::IniFiles;
Getopt::Std;
Sys::Syslog;
</pre>

=== Configuration ===
/etc/backup-utils/writeDate.conf
<pre>
[Global]
BACKUP_DIRECTORY = /incoming
BACKUP_BACKUP_TIME_FILE = /.sepiola_backup/backupTime

[Syslog]
SYSLOG = rsnapshot
</pre>

== rsnapshot wrapper bash scripts ==
There are two rsnapshot wrapper scripts (these are bash scripts), they are installed by the <code>sys-apps/sst-backup-utils</code> package:
* <code>/usr/libexec/backup-utils/rsnapshotPreExecWrapper.sh</code>
** This script is executed before the rsnapshot process and simply calls the [[stoney_backup:_Server_set-up#writeDate.pl | writeDat.pl ]] script.
* <code>/usr/libexec/backup-utils/rsnapshotPostExecWrapper.sh</code>
** This script is executed after the rsnapshot process and simply calls the [[stoney_backup:_Server_set-up#writeAccountSize.pl | writeAccountSize.pl ]] script.

'''Note:''' These scripts do not need any configuration.