Difference between revisions of "stoney backup: prov-backup-rsnapshot"

From stoney cloud
Jump to: navigation, search
[unchecked revision][unchecked revision]
(→‎Requirements)
 
(14 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
= Overview =
 
= Overview =
−
This pages contains all necessary information one needs to know about the prov-backup-rsnapshot daemon.
+
This pages contains all necessary information one needs to know about the prov-backup-rsnapshot provisioning daemon. This daemon provisions the stoney backup account according to what was requested in the self-service web interface. Changes performed on the web interface (for example creating a new stoney backup account) will be written to the LDAP directory. The changes in the LDAP directory are spotted by the prov-backup-rsnapshot daemon, which then executes the appropriate action (for example creating directories and setting quota values for the new stoney backup account) and change the LDAP again. This change in the LDAP can be made visible in the web interface. This way the user can see, that the new stoney backup account is ready for use.
 +
 
 +
= Source Code =
 +
The source code is located in our GitHub Repository:
 +
 
 +
https://github.com/stepping-stone/prov-backup-rsnapshot
 +
 
 +
== Exit Codes ==
 +
The following list defines the return codes and their meaning for the ...:
 +
<pre>
 +
use constant
 +
{
 +
    SUCCESS                                => 0,
 +
    UNDEFINED_ERROR                        => 1,
 +
    USER_NOT_PRESENT_IN_BACKEND            => 2,
 +
    PERL_VERSION_NOT_STATISFIED            => 3,
 +
    CANNOT_CHECK_HOME_DIR_PRESENCE          => 4,
 +
    CANNOT_CREATE_HOME_DIR                  => 5,
 +
    CANNOT_SET_QUOTA                        => 6,
 +
    CANNOT_CREATE_RSNAPSHOT_CONFIGURATION  => 7,
 +
};
 +
</pre>
  
 
= Configuration =
 
= Configuration =
 +
(Note: Comments starting with /* are not in the configuration file, they are only in the wiki to add some additional information)
 +
 
<pre>
 
<pre>
 
# Copyright (C) 2013 stepping stone GmbH
 
# Copyright (C) 2013 stepping stone GmbH
Line 29: Line 52:
 
#
 
#
  
−
 
+
/* If you want, you can override the log information from the global configuration file this might be useful for debugging */
 
[Global]
 
[Global]
 
# If true the script logs every information to the log-file.
 
# If true the script logs every information to the log-file.
Line 43: Line 66:
 
LOG_ERR = 1
 
LOG_ERR = 1
  
−
ENVIRONMENT = pat-GT60
+
/* Specify the hosts fully qualified domain name. This name will be used to perform some checks and also appear in the information and error mails */
 +
ENVIRONMENT = <FQDN>
 
   
 
   
 
[Database]
 
[Database]
 
BACKEND = LDAP
 
BACKEND = LDAP
−
SERVER = ldaps://ldapm.tombstone.ch
+
SERVER = ldaps://ldapm.tombstone.org
 
PORT = 636
 
PORT = 636
−
ADMIN_USER = cn=Manager,dc=foss-cloud,dc=org
+
ADMIN_USER = cn=Manager,dc=stoney-cloud,dc=org
 
ADMIN_PASSWORD = <PASSWORD>
 
ADMIN_PASSWORD = <PASSWORD>
−
SERVICE_SUBTREE = ou=accounts,ou=backup,ou=services,dc=foss-cloud,dc=org
+
SERVICE_SUBTREE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
−
COOKIE_FILE = /home/pat/repos/stepping-stone/prov-backup-rsnapshot/prov-backup-rsnapshot/etc/Provisioning/Backup/rsnapshot.cookie
+
COOKIE_FILE = /etc/Provisioning/Backup/rsnapshot.cookie
 
DEFAULT_COOKIE = rid=001,csn=
 
DEFAULT_COOKIE = rid=001,csn=
−
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(objectClass=*))
+
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(sstProvisioningState=0))
  
 +
/* Specifies the service itself. As it is the prov-backup-rsnapshot module, the SERVICE is "Backup" and the TYPE is "Rsnapshot".
 +
* The MODUS is as usual selfcare and the TRANSPORTAPI is LocalCLI. This is because the daemon is running on the same host as the
 +
* backup accounts are provisioned and the commands can be executed on this host using the cli.
 +
* For more information about MODUS and TRANSPORTAPI see https://int.stepping-stone.ch/wiki/provisioning.pl#Service_Konfiguration
 +
*/
 
[Service]
 
[Service]
 
MODUS = selfcare
 
MODUS = selfcare
Line 62: Line 91:
 
TYPE = Rsnapshot
 
TYPE = Rsnapshot
  
−
SYSLOG = Backup-Rsnapshot
+
SYSLOG = prov-backup-rsnapshot
  
 +
/* For the TRANSPORTAPI LocalCLI there is no gateway required because there is no connection to establish. So set HOST, USER and
 +
* DSA_FILE to whatever you want. Don't leave it blank, otherwise the provisioning daemon would log some error messages saying
 +
* these attributes are empty
 +
*/
 
[Gateway]
 
[Gateway]
 
HOST = localhost
 
HOST = localhost
Line 69: Line 102:
 
DSA_FILE = none
 
DSA_FILE = none
  
 +
/* Information about the backup itself (how to setup everything). Note that the %uid% int the RSNAPSHOT_CONFIG_FILE parameter will
 +
* be replaced by the accounts UID. The script CREATE_CHROOT_CMD was installed with the prov-backup-rsnapshot module, so do not
 +
* change this parameter. The quota parameters (SET_QUOTA_CMD, MOUNTPOINT, QUOTA_FILE, PROJECTS_FILE and PROJID_FILE) represent
 +
* the quota setup as described on http://wiki.stoney-cloud.org/index.php/stoney_backup:_Server_set-up#Quota. If you followed this
 +
* manual, you can copy-paste them into your configuration file, otherwise adapt them according to your quota setup.
 +
*/
 
[Backup]
 
[Backup]
 
RSNAPSHOT_CONFIG_FILE = /etc/rsnapshot/rsnapshot.conf.%uid%
 
RSNAPSHOT_CONFIG_FILE = /etc/rsnapshot/rsnapshot.conf.%uid%
−
SET_QUOTA_CMD = /usr/sbin/setquota
+
SET_QUOTA_CMD = /usr/sbin/xfs_quota
−
CREATE_CHROOT_CMD = /root/createDummyBackupDirectory.sh
+
CREATE_CHROOT_CMD = /usr/libexec/createBackupDirectory.sh
−
MOUNTPOINT = /
+
MOUNTPOINT = /var
 
QUOTA_FILE = /etc/backupSize
 
QUOTA_FILE = /etc/backupSize
 
PROJECTS_FILE = /etc/projects
 
PROJECTS_FILE = /etc/projects
Line 170: Line 209:
 
...
 
...
 
</source>
 
</source>
−
 
−
= Source Code =
 
−
The source code is located in our GitHub Repository:
 
−
 
−
https://github.com/stepping-stone/prov-backup-rsnapshot
 
−
 
−
== Exit Codes ==
 
−
The following list defines the return codes and their meaning for the ...:
 
−
<pre>
 
−
use constant
 
−
{
 
−
    SUCCESS                                => 0,
 
−
    UNDEFINED_ERROR                        => 1,
 
−
    USER_NOT_PRESENT_IN_BACKEND            => 2,
 
−
    PERL_VERSION_NOT_STATISFIED            => 3,
 
−
    CANNOT_CHECK_HOME_DIR_PRESENCE          => 4,
 
−
    CANNOT_CREATE_HOME_DIR                  => 5,
 
−
    CANNOT_SET_QUOTA                        => 6,
 
−
    CANNOT_CREATE_RSNAPSHOT_CONFIGURATION  => 7,
 
−
};
 
−
</pre>
 
  
 
= Testing =
 
= Testing =
 
== Requirements ==
 
== Requirements ==
−
A working test server set up following the [[stoney backup: Server set-up]] guide.
+
A working test server set up following the [[stoney_backup:_Server_set-up#prov-backup-rsnapshot | stoney backup: Server set-up]] guide.
−
 
+
−
@PKL: Please use full paths ... For example:
+
−
 
+
−
Replace existing node-integration binaries:
+
−
cd /var/work/node-integration
+
−
ln -sf $PWD/bin/fc-node-configuration  /usr/sbin/fc-node-configuration
+
−
ln -sf $PWD/bin/node-integration-check /usr/libexec/foss-cloud/node-integration-check
+
−
 
+
−
<pre>
+
−
mkdir -p /var/work
+
−
cd /var/work
+
−
git clone --recursive https://github.com/stepping-stone/prov-backup-rsnapshot.git
+
−
cd /var/work/prov-backup-rsnapshot/Provisioning/etc/Provisioning/
+
−
ln -s ../../../etc/Provisioning/Backup/ Backup
+
−
cd /var/work/prov-backup-rsnapshot/Provisioning/lib/Provisioning/
+
−
ln -s ../../../lib/Provisioning/Backup/ Backup
+
−
</pre>
+
−
 
+
−
A test configuration
+
−
<pre>
+
−
cd /var/work/prov-backup-rsnapshot/Provisioning/etc/Provisioning/Backup/
+
−
cp -p Rsnapshot.conf.template Rsnapshot_test.conf
+
−
vi Rsnapshot_test.conf
+
−
</pre>
+
−
Update the Database section accordingly
+
  
 
== Test the daemon ==
 
== Test the daemon ==
−
* The daemon is running on the Backup-Server VM (kvm-0036)
+
* The daemon is running on the Backup-Server VM
 
** Open a terminal and login to the Backup-Server VM
 
** Open a terminal and login to the Backup-Server VM
 
** <pre>tail -f /var/log/syslog/provisioning.log</pre>
 
** <pre>tail -f /var/log/syslog/provisioning.log</pre>
Line 228: Line 221:
 
** Make sure the directory /root/data exists, if not create it
 
** Make sure the directory /root/data exists, if not create it
 
** Create an online backup account by executing the test script:  
 
** Create an online backup account by executing the test script:  
−
** <pre>/var/work/prov-backup-rsnapshot/bin/createBackupAccountLDIF.pl --uid 3724300</pre>
+
** <pre>/usr/libexec/createBackupDirectory.sh --uid 3724300</pre>
 
** The test script currently supports the following options:  
 
** The test script currently supports the following options:  
 
*** --uid <UID>: Mandatory option, the backup accounts uid
 
*** --uid <UID>: Mandatory option, the backup accounts uid
Line 238: Line 231:
 
** Load the generated LDIF (you find them under /root/data) step by step into the LDAP and observe what the deamon is doing in the first terminal:  
 
** Load the generated LDIF (you find them under /root/data) step by step into the LDAP and observe what the deamon is doing in the first terminal:  
 
*** <pre>/root/loadOpenLDAPsingleLDIF.sh /root/data/01_add_3724300.ldif</pre>
 
*** <pre>/root/loadOpenLDAPsingleLDIF.sh /root/data/01_add_3724300.ldif</pre>
 +
**** After this step, the account with the given uid should be created and you should be able to login via ssh using the uid and the password "gugus":
 +
***** <pre>ssh <UID>@<Server></pre>
 
*** <pre>/root/loadOpenLDAPsingleLDIF.sh /root/data/02_modify_3724300.ldif</pre>
 
*** <pre>/root/loadOpenLDAPsingleLDIF.sh /root/data/02_modify_3724300.ldif</pre>
 +
**** After this step, the quota should be reduced by 50% (from 1GB to 500MB)
 +
***** To verify use the report_xfs quota command as mentioned in [[stoney_backup:_Server_set-up#Modifying_a_Project_.28Directory.29_Quota | server setup]]
 
*** <pre>/root/loadOpenLDAPsingleLDIF.sh /root/data/03_delete_3724300.ldif</pre>
 
*** <pre>/root/loadOpenLDAPsingleLDIF.sh /root/data/03_delete_3724300.ldif</pre>
 +
**** After this step, the account should be removed, the login via ssh should no longer be possible. The LDAP entry however is still present.
 +
***** <pre>ssh <UID>@<Server></pre>
 
*** <pre>/root/loadOpenLDAPsingleLDIF.sh /root/data/04_remove_3724300.ldif</pre>
 
*** <pre>/root/loadOpenLDAPsingleLDIF.sh /root/data/04_remove_3724300.ldif</pre>
 +
**** After this step, the LDAP entry should be removed
  
 
= Links =
 
= Links =
  
−
[[Category:stoney backup]][[Category:Provisioning Modules]]
+
[[Category: stoney backup]]
 +
[[Category: Provisioning Modules]]

Latest revision as of 16:43, 1 November 2019

Overview

This pages contains all necessary information one needs to know about the prov-backup-rsnapshot provisioning daemon. This daemon provisions the stoney backup account according to what was requested in the self-service web interface. Changes performed on the web interface (for example creating a new stoney backup account) will be written to the LDAP directory. The changes in the LDAP directory are spotted by the prov-backup-rsnapshot daemon, which then executes the appropriate action (for example creating directories and setting quota values for the new stoney backup account) and change the LDAP again. This change in the LDAP can be made visible in the web interface. This way the user can see, that the new stoney backup account is ready for use.

Source Code

The source code is located in our GitHub Repository:

https://github.com/stepping-stone/prov-backup-rsnapshot

Exit Codes

The following list defines the return codes and their meaning for the ...: 

use constant
{
    SUCCESS                                 => 0,
    UNDEFINED_ERROR                         => 1,
    USER_NOT_PRESENT_IN_BACKEND             => 2,
    PERL_VERSION_NOT_STATISFIED             => 3,
    CANNOT_CHECK_HOME_DIR_PRESENCE          => 4,
    CANNOT_CREATE_HOME_DIR                  => 5,
    CANNOT_SET_QUOTA                        => 6,
    CANNOT_CREATE_RSNAPSHOT_CONFIGURATION   => 7,
};

Configuration

(Note: Comments starting with /* are not in the configuration file, they are only in the wiki to add some additional information) 

# Copyright (C) 2013 stepping stone GmbH
#                    Switzerland
#                    http://www.stepping-stone.ch
#                    support@stepping-stone.ch
#
# Authors:
#  Pat Kläy <pat.klaey@stepping-stone.ch>
#  
# Licensed under the EUPL, Version 1.1.
#
# You may not use this work except in compliance with the
# Licence.
# You may obtain a copy of the Licence at:
#
# http://www.osor.eu/eupl
#
# Unless required by applicable law or agreed to in
# writing, software distributed under the Licence is
# distributed on an "AS IS" basis,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied.
# See the Licence for the specific language governing
# permissions and limitations under the Licence.
#

/* If you want, you can override the log information from the global configuration file this might be useful for debugging */
[Global]
# If true the script logs every information to the log-file.
LOG_DEBUG = 1

# If true the script logs additional information to the log-file.
LOG_INFO = 1

#If true the script logs warnings to the log-file.
LOG_WARNING = 1

#If true the script logs errors to the log-file.
LOG_ERR = 1

/* Specify the hosts fully qualified domain name. This name will be used to perform some checks and also appear in the information and error mails */
ENVIRONMENT = <FQDN>
 
[Database]
BACKEND = LDAP
SERVER = ldaps://ldapm.tombstone.org
PORT = 636
ADMIN_USER = cn=Manager,dc=stoney-cloud,dc=org
ADMIN_PASSWORD = <PASSWORD>
SERVICE_SUBTREE = ou=accounts,ou=backup,ou=services,dc=stoney-cloud,dc=org
COOKIE_FILE = /etc/Provisioning/Backup/rsnapshot.cookie
DEFAULT_COOKIE = rid=001,csn=
SEARCH_FILTER = (&(entryCSN>=%entryCSN%)(sstProvisioningState=0))

/* Specifies the service itself. As it is the prov-backup-rsnapshot module, the SERVICE is "Backup" and the TYPE is "Rsnapshot".
 * The MODUS is as usual selfcare and the TRANSPORTAPI is LocalCLI. This is because the daemon is running on the same host as the
 * backup accounts are provisioned and the commands can be executed on this host using the cli.
 * For more information about MODUS and TRANSPORTAPI see https://int.stepping-stone.ch/wiki/provisioning.pl#Service_Konfiguration
 */
[Service]
MODUS = selfcare
TRANSPORTAPI = LocalCLI
SERVICE = Backup
TYPE = Rsnapshot

SYSLOG = prov-backup-rsnapshot

/* For the TRANSPORTAPI LocalCLI there is no gateway required because there is no connection to establish. So set HOST, USER and
 * DSA_FILE to whatever you want. Don't leave it blank, otherwise the provisioning daemon would log some error messages saying
 * these attributes are empty 
 */
[Gateway]
HOST = localhost
USER = provisioning
DSA_FILE = none

/* Information about the backup itself (how to setup everything). Note that the %uid% int the RSNAPSHOT_CONFIG_FILE parameter will
 * be replaced by the accounts UID. The script CREATE_CHROOT_CMD was installed with the prov-backup-rsnapshot module, so do not
 * change this parameter. The quota parameters (SET_QUOTA_CMD, MOUNTPOINT, QUOTA_FILE, PROJECTS_FILE and PROJID_FILE) represent 
 * the quota setup as described on http://wiki.stoney-cloud.org/index.php/stoney_backup:_Server_set-up#Quota. If you followed this
 * manual, you can copy-paste them into your configuration file, otherwise adapt them according to your quota setup.
 */
[Backup]
RSNAPSHOT_CONFIG_FILE = /etc/rsnapshot/rsnapshot.conf.%uid%
SET_QUOTA_CMD = /usr/sbin/xfs_quota
CREATE_CHROOT_CMD = /usr/libexec/createBackupDirectory.sh
MOUNTPOINT = /var
QUOTA_FILE = /etc/backupSize
PROJECTS_FILE = /etc/projects
PROJID_FILE = /etc/projid

Communication with LDAP

The daemon "communicates" with the LDAP to let other processes know its current state. The important attributes therefore are sstProvisioningMode and sstProvisioningState in the given Backup-Account entry. The sequence is defined for adding, modifying or deleting an entry. The preconditions for all three modi are:

  • sstProvisioningMode is set to add, modify or delete
  • sstProvisioningState is set to 0
  • sstProvisioningDate is set to 0 or the current date (format: YYYYMMDD)

Add

Precondition

...
sstProvisioningMode: add
sstProvisioningState: 0
sstProvisioningDate: 0
...

Step 1

...
sstProvisioningMode: adding
sstProvisioningState: 0
sstProvisioningDate: 0
...

Now the prov-backup-rsnapshot daemon works on the given entry.

Step 2

As soon as the prov-backup-rsnapshot daemon finished working on the given entry: 

...
sstProvisioningMode: added
sstProvisioningState: YYYYMMDDThhmmssZ
sstProvisioningDate: 0
...

Modify

Precondition

...
sstProvisioningMode: modify
sstProvisioningState: 0
sstProvisioningDate: 0
...

Step 1

...
sstProvisioningMode: modifying
sstProvisioningState: 0
sstProvisioningDate: 0
...

Now the prov-backup-rsnapshot daemon works on the given entry.

Step 2

As soon as the prov-backup-rsnapshot daemon has finished working on the given entry: 

...
sstProvisioningMode: modified
sstProvisioningState: YYYYMMDDThhmmssZ
sstProvisioningDate: 0
...

Delete

Precondition

...
sstIsActive: FALSE
sstProvisioningMode: delete
sstProvisioningState: 0
sstProvisioningDate: 0
...

Step 1

...
sstProvisioningMode: deleting
sstProvisioningState: 0
sstProvisioningDate: 0
...

Now the prov-backup-rsnapshot daemon works on the given entry.

Step 2

As soon as the prov-backup-rsnapshot daemon finished working on the given entry: 

...
sstProvisioningMode: deleted
sstProvisioningState: YYYYMMDDThhmmssZ
sstProvisioningDate: 0
...

Testing

Requirements

A working test server set up following the stoney backup: Server set-up guide.

Test the daemon

  • The daemon is running on the Backup-Server VM
    • Open a terminal and login to the Backup-Server VM 
    • tail -f /var/log/syslog/provisioning.log
  • Open a second terminal and login to the Backup-Server VM
    • Make sure the directory /root/data exists, if not create it
    • Create an online backup account by executing the test script: 
    • /usr/libexec/createBackupDirectory.sh --uid 3724300
    • The test script currently supports the following options:
      • --uid <UID>: Mandatory option, the backup accounts uid
      • --reseller <UID>: Optional, sets sstBelongsToResellerUID (default 4000000)
      • --customer <UID>: Optional, sets sstBelongsToCustomerUID (default 4000001)
      • --people <UID>: Optional, sets sstBelongsToPersonUID (default 4000002)
      • --givenname <Name>: Optional, sets the givenname attribute (default "Givenname")
      • --surname <Name>: Optional, sets the sn attribute (defualt "Surname")
    • Load the generated LDIF (you find them under /root/data) step by step into the LDAP and observe what the deamon is doing in the first terminal: 
      • /root/loadOpenLDAPsingleLDIF.sh /root/data/01_add_3724300.ldif
        • After this step, the account with the given uid should be created and you should be able to login via ssh using the uid and the password "gugus": 
          • ssh <UID>@<Server>
      • /root/loadOpenLDAPsingleLDIF.sh /root/data/02_modify_3724300.ldif
        • After this step, the quota should be reduced by 50% (from 1GB to 500MB)
          • To verify use the report_xfs quota command as mentioned in server setup 
      • /root/loadOpenLDAPsingleLDIF.sh /root/data/03_delete_3724300.ldif
        • After this step, the account should be removed, the login via ssh should no longer be possible. The LDAP entry however is still present. 
          • ssh <UID>@<Server>
      • /root/loadOpenLDAPsingleLDIF.sh /root/data/04_remove_3724300.ldif
        • After this step, the LDAP entry should be removed

Links

Retrieved from "http://wiki.stoney-cloud.org/mediawiki/index.php?title=stoney_backup:_prov-backup-rsnapshot&oldid=4998"