Changes
== System Overview ==
The [http://www.foss-cloud.org/ FOSS-Cloud] builds upon various standard open source components and can be run on commodity hardware. The final Multi-Node Setup consists of the following components:
* One [[Primary-Master-Node]] with an OpenLDAP Directory Server for the storage of the FOSS-Cloud user and service related data with the web based management [[VM-Manager]] interface and the Linux kernel based virtualization technology.
* One [[VM-Node]] with the Linux kernel based virtualization technology.
* Two [[Storage-Node | Storage-Nodes]] configured as a replicated and distributed data storage service based on [http://www.gluster.org/ GlusterFS].
The components communicate with each other over a standard Ethernet based IPv4 network.
=== Prerequisites ===
The following items and conditions are required to be able to setup a FOSS-Cloud environment:
* Dedicated Hardware (4 Servers) which fulfil the following requirements:
** 64-Bit Intel with VT-Technologie (AMD is not tested at the moment).
** 8 Gigabyte Memory (more is better).
** 147 Gigabyte up to 2 Terabyte disks.
*** For all four Nodes ([[VM-Node | VM-Nodes]] and [[Storage-Node | Storage-Nodes]]) two 147 Gigabyte Disks, configured as RAID1, are enough for the [http://www.gentoo.org/ Gentoo Linux] operating system.
*** Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.
** Two physical Ethernet Interfaces, which support the same bandwidth (for example 1 Gigabit/s).
* Two Gigabit layer-2 switches supporting [http://en.wikipedia.org/wiki/IEEE_802.3ad IEEE 802.3ad] (dynamic link aggregation), [http://en.wikipedia.org/wiki/IEEE_802.1Q IEEE 802.1Q] (VLAN tagging) and stacking (optional, but recommended).
* Experience with Linux environments especially with [http://www.gentoo.org Gentoo Linux].
* Good experience with IP networking, because the Switches need to be configured manually (dynamic link aggregation and VLAN tagging).
=== Limitations ===
During the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system, the first two physical Ethernet Interfaces are automatically configured as a logical interface (bond0) and the four tagged VLANs are set up. If more the two physical Ethernet Interfaces are to be included in to the logical interface (bond0) or the physical Ethernet Interfaces have different bandwidths (for example 1 Gigabit/s and 10 Gigabit/s), the final logical interface (bond0) needs to be configured manually after the installation of the [http://www.gentoo.org/ Gentoo Linux] operating system.
Only the first two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] are set up automatically. More [[Storage-Node | Storage-Nodes]] need to be integrated manually.
=== Network Overview ===
As stated before, a minimal multi node FOSS-Cloud environment consist of two FOSS-Cloud and GlusterFS nodes.
It is highly recommended to use IEEE 802.3ad link aggregation (bonding, trunking etc.) over two network cards and attach them as one logical link to the access switch.
It is out of scope of this document on how to configure the switches for stacking, link aggregation and VLAN tagging, consult the respective user manual.
There are two scenarios on how to connect the nodes to the network.
==== Network Overview: Physical Layer Scenario 1 (recommended) ====
The preferred solution is to use two switches which are stackable (supporting link aggregation over two switches). Connect the nodes and switches as illustrated below. Thus eliminating the single point of failure as presented in scenario 2.
<pre>
+--------------------+ +--------------------+
| | | |
| foss-cloud-node-01 | | foss-cloud-node-02 |
| | | |
+--------------------+ +--------------------+
bond0 | \ / | bond0
| \ / |
| \ / |
| \/ | ___
| /\ | ___( )___
| / \ | __( )__
| / \ | _( )_
+-------------+ +-------------+ _( )_
| switch-01 |===| switch-02 |-------------------(_ Corporate LAN/WAN _)
+-------------+ +-------------+ (_ _)
| \ / | (__ __)
| \ / | (___ ___)
| \ / | (___)
| \/ |
| /\ |
| / \ |
bond0 | / \ | bond0
+--------------------+ +--------------------+
| | | |
| gluster-node-01 | | gluster-node-02 |
| | | |
+--------------------+ +--------------------+
</pre>
==== Network Overview: Physical Layer Scenario 2 (use at your own risk) ====
If theres only one switch available (or the switches aren't stackable) connect the nodes as illustrated below. As you can see, the switch is as a single point of failure.
<pre>
+--------------------+ +--------------------+
| | | |
| foss-cloud-node-01 | | foss-cloud-node-02 |
| | | |
+--------------------+ +--------------------+ ___
bond0 \\ // bond0 ___( )___
\\ // __( )__
\\ // _( )_
+-------------+ _( )_
| switch-01 |-----------------------------(_ Corporate LAN/WAN _)
+-------------+ (_ _)
// \\ (__ __)
// \\ (___ ___)
bond0 // \\ bond0 (___)
+--------------------+ +--------------------+
| | | |
| gluster-node-01 | | gluster-node-02 |
| | | |
+--------------------+ +--------------------+
</pre>
==== Network overview: Logical layer ====
The ideal FOSS-Cloud environment is based on four logical separated VLANs (virtual LANs):
* '''admin''': Administrative network, used for administration and monitoring purposes.
* '''data''': Data network, used for GlusterFS traffic.
* '''int''': Internal network, used for internal traffic such as LDAP, libvirt and more.
* '''pub''': Public network, used for accessing the VM-Manager webinterface, Spice traffic and internet access.
Each of the above VLANs hold dedicated services and separates them from each other. This documentation assumes, that the four VLANs are present and the following IP networks are available:
{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|VLAN name
!align="left"|VLAN ID
!align="left"|Network prefix
!align="left"|Default Gateway address
!align="left"|Broadcast address
!align="left"|Domain name
!align="left"|VIP
|-
|admin
|110
|10.1.110.0/24
| --
|10.1.110.255
|admin.example.com
|
|-
|data
|120
|10.1.120.0/24
| --
|10.1.120.255
|data.example.com
|
|-
|int
|130
|10.1.130.0/24
| --
|10.1.130.255
|int.example.com
|10.1.130.10
|-
|pub
|140
|192.168.140.0/24
|192.168.140.1
|192.168.140.255
|example.com
|192.168.140.10
|-
|}
The IP allocation of the nodes will be assumed as stated in the table below:
{| border="1" style="border-collapse: collapse; font-size:100%;" width=100%
!align="left"|Node name
!align="left"|Admin address (VLAN 110)
!align="left"|Data address (VLAN 120)
!align="left"|Int address (VLAN 130)
!align="left"|Pub address (VLAN 140)
|-
|-
|gluster-node-01
|10.1.110.11
|10.1.120.11
|10.1.130.11
|192.168.140.11
|-
|-
|gluster-node-02
|10.1.110.12
|10.1.120.12
|10.1.130.12
|192.168.140.12
|-
|-
|foss-cloud-node-01
|10.1.110.13
|10.1.120.13
|10.1.130.13
|192.168.140.13
|-
|-
|foss-cloud-node-02
|10.1.110.14
|10.1.120.14
|10.1.130.14
|192.168.140.14
|-
|}
We'll also presume, we have the following Domain Name Servers:
* Domain Name Server 1: '''192.168.140.1'''
== Base Installation ==
All nodes are based on the [http://www.gentoo.org/ Gentoo Linux] operating system.
=== BIOS Set Up Checklist ===
* Enable the "reboot-after-power-loss" option (if your BIOS supports it).
* Make sure, that you have the newest BIOS (BMC and perhaps SCSI firmware) version.
* Make sure, you've disabled halt on post errors (or similar) or enable keyboard-less operation (if your BIOS supports it).
=== RAID Set Up ===
Create a RAID1 volume. This RAID-Set is used for the Operating System. Please be aware, the the current FOSS-Cloud only supports 147 Gigabyte up to 2 Terabyte disks for this first RAID-Set.
Optional: For the two [http://www.gluster.org/ GlusterFS] [[Storage-Node | Storage-Nodes]] we recommend a second RAID-Set configured as [http://en.wikipedia.org/wiki/RAID_6#RAID_6 RAID6]-Set with battery backup.
=== Node Installation ===
The first step of Semi-Automatic Multi-Node Set Up is the same for all four Nodes. In this documentation we presume, that you stick to the naming convention mentioned above. After the Base Installation of the Nodes, the following daemons will be running:
* '''crond''': Crond, to execute scheduled commands
* '''ntpd''': Network Time Protocol daemon, keeps the time synced with the Time from Servers in the LAN or WAN.
* '''sshd''': OpenSSH SSH daemon, used for remote access and remote administration.
* '''syslogd''': System Logging, keeps track of messages from the System and the Applications.
* '''udevd''': Linux dynamic device management, that manage events, symlinks and permissions of devices.
=== Skipping Checks ===
To skip checks, type '''no''' when asked:
Do you want to start the installation?
yes or no?: '''no'''
Then manually restart the FOSS-Cloud installer with the desired options:
/mnt/cdrom/foss-cloud-installer -c
Options:
-c: Skip CPU requirement checks
-m: Skip memory requirement checks
-s: Skip CPU and memory requirement checks
==== First Storage-Node (gluster-node-01) ====
# Insert the FOSS-Cloud CD and boot the server.
# Answer the questions as follows (the bold values are examples can be set through the administrator and are variable, according to the local setup):
## Global Section
### Confirm that you want to start? '''yes'''
### Choose a Node-Type: '''Storage-Node'''
### Choose a Block-Device: '''sda'''
### Confirm to erase all (from a previous installation): '''yes'''
### Confirm to continue with the given the Partition-Scheme: '''yes'''
### Choose the network interfaces to bond together.
#### Device #0: '''eth0'''
#### Device #1: '''eth1'''
### Node-Name: '''gluster-node-01'''
## pub-VLAN Section
### VLAN ID: '''140'''
### Domain Name: '''example.com'''
### IP Address: '''192.168.140.11'''
### Netmask: '''24'''
### Broadcast: '''192.168.140.255'''
### Confirm the pub-VLAN Section with '''yes'''
## admin-VLAN Section
### VLAN ID: '''110'''
### Domain Name: '''admin.example.com'''
### IP Address: '''10.1.110.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.110.255'''
### Confirm the admin-VLAN Section with '''yes'''
## data-VLAN Section
### VLAN ID: '''120'''
### Domain Name: '''data.example.com'''
### IP Address: '''10.1.120.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.120.255'''
### Confirm the data-VLAN Section with '''yes'''
## int-VLAN Section
### VLAN ID: '''130'''
### Domain Name: '''int.example.com'''
### IP Address: '''10.1.130.11'''
### Netmask: '''24'''
### Broadcast: '''10.1.130.255'''
### Confirm the int-VLAN Section with '''yes'''
### Enter the default Gateway: '''192.168.140.1'''
### Enter the primary DNS-Server: '''192.168.140.1'''
### Omit configuring a second DNS-Server with '''no'''
## Confirm the listed configuration with '''yes'''
## Confirm to reboot with '''yes'''
# Make sure, that you boot from the first harddisk and not from the installation medium again.
# Continue with [[Multi-Node Installation#Specialized_Installation|specializing your Node]]
==== Second Storage-Node (gluster-node-02) ====
# Insert the FOSS-Cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.
==== Primary-Master-Node (foss-cloud-node-01) ====
# Insert the FOSS-Cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.
==== Secondary-Master-Node (foss-cloud-node-02) ====
# Insert the FOSS-Cloud CD and boot the server.
# Answer the questions.
# Reboot the Server and make sure, that you boot from the first harddisk.
== Specialized Installation ==
==== First Storage-Node (gluster-node-01) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].
Log into the first [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-storage-node
For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.
==== Second Storage-Node (gluster-node-02) ====
Before running the node configuration script, you may want to create a [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#Before_node_configuration_script | additional Backup Volume on Storage Node]].
Log into the second [[Storage-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-storage-node
For more information about the script and what it does, please visit the [[node-configuration]] script page.
==== Primary-Master-Node (foss-cloud-node-01) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the VM-Node]].
Log into the [[Primary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type primary-master-node
The FOSS-Cloud uses virtual ip addresses (VIPs) for failover purposes. Therefore you need to configure [http://www.pureftpd.org/project/ucarp ucarp].
Confirm that you want to run the script.
Do you really want to proceed with configuration of the primary-master-node?
yes or no (default: no): '''yes'''
Enter the VIP (virtual IP) for the public interface. The apache will listen on this VIP because it is listening on the public interface (if you followed this documentation the VIP for the public interface is 192.168.140.10).
Please enter the VIP for the pub-interface (VLAN 140)
(default: 192.168.140.10): '''192.168.140.10'''
Enter the VIP (virtual IP) for the internal interace. The LDAP will listen on this VIP because it is listening on the internal interface (if you followed this documentation the VIP for the internal interface is 10.1.130.10).
Please enter the VIP for the int-interface (VLAN 130)
(default: 10.1.130.10): '''10.1.130.10'''
Enter the RIP (real IP) for the secondary-master-node on the internal interface. The RIP is needed to keep the LDAP directories synchronized (if you followed this documentation the RIP for the secondary-master-node on the internal interface is 10.1.130.14)
Please enter the IP for the int-interface (VLAN 130) of the
secondary-master-node (default: 10.1.130.14): '''10.1.130.14'''
The script now tests the network configuration and
In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node. Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the primary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':
IP address (default: 10.1.110.11): '''10.1.110.11'''
Username (default: root): '''root'''
Password for root: '''**********'''
In order to mount the gluster-filesystem, you need to connect via ssh to the secondary-storage-node. Enter the IP for the secondary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.12). Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root) and the corresponding password.
Please enter the following information for the secondary Storage-Node with the OpenSSH daemon listening on the VLAN with the name 'admin' and with the VLAN ID '110':
IP address (default: 10.1.110.12): '''10.1.110.12'''
Username (default: root): '''root'''
Password for root: '''**********'''
* Configure the LDAP directory:
** Define the password for the LDAP-Superuser (cn=Manager,dc=foss-cloud,dc=org)
** Currently the user for the prov-backup-kvm daemon is the LDAP-Superuser so enter the same password again
** Define the password for the LDAP-dhcp user (cn=dhcp,ou=services,ou=administration,dc=foss-cloud,dc=org)
** Enter all necessary information for the FOSS-Cloud administrator (User1)
*** Given name
*** Surname
*** Gender
*** E-mail
*** Language
*** Password
* Finally enter the domain name which will correspond to the public VIP (default is foss-cloud.example.tld)
For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.
==== Secondary-Master-Node (foss-cloud-node-02) ====
If you configured a additional Backup Volume on the Storage Nodes, you want to [[Additional_Local_Backup_Volume_on_the_Storage-Nodes#On_the_VM-Nodes | mount them now in the Secondary-Master-Node]].
Log into the [[Secondary-Master-Node]] and execute the node-configuration script as follows:
/usr/sbin/fc-node-configuration --node-type secondary-master-node
* In order to get the configuration from the primary-master-node, we need to access it via ssh
** Enter the IP for the primary-master-node on the admin interface (if you followed this documentation it is 10.1.130.13)
** Enter the username (if you followd the default setup it is root)
** Enter the users password.
* In order to mount the gluster-filesystem, you need to connect via ssh to the primary-storage-node, so enter the following information:
** Enter the IP for the primary-storage-node on the admin interface (if you followed this documentation the IP for the primary-storage-node on the admin interface is 10.1.110.11)
** Enter the a valid username which exists on the primary-storage-node (if you followed this documentation it is root)
** Enter the users password
* Repeat the same procedure for the secondary-storage-node (if you followed this documentation the IP is 10.1.110.12)
* Enter the LDAP-Superuser password you defined during the [[Multi-Node Installation#Primary-Master-Node (foss-cloud-node-01)_2 | primary-master-node]] installation
For more information about the script and what it does, please visit the [[fc-node-configuration]] script page.
== Links ==
= Old Documentation =
== Specialized Installation ==
=== Primary-Master-Node (vm-node-01) ===
[[Category:stoney cloud]]
[[Category:Documentation]]
[[Category:Installation]]
