stoney-cloud.org - User contributions [en]

User:Lucas/Gentoo Install Notes

2014-02-23T13:10:52Z

Lucas:

* hack <code>/usr/local/scripts/netfilter/local/chains/vms/kvm_0231_chain.sh</code> on host to allow gateway conns
* first hd is /dev/vda
* default gentoo handbook install with lvm setup on vda3 and one large lv_root
* install lvm2 so you can build a lvm initramfs
** if you skip this you will have tons of fun loading lvm in the initramfs shell: <code>lvm vgscan --mknodes && lvm lvchange -a ly vg01/lv_root</code>
* kernel build with: <code>genkernel --install --lvm --menuconfig all</code> (do not use <code>--virtio</code>, activate them in menuconfig instead, I had heaps of fun hunting down all the modules)
** actually <code>genkernel --install --lvm --kernel-config=/root/kernel.config all</code> since lazy me hates using a ui
** the --virtio switch seems screwed due to some oldconfig changes with the <code>VIRTIO_MMIO</code> system, but i haven't looked into that more
* remember to also set <code>GRUB_CMDLINE_LINUX="dolvm"</code> in <code>/etc/default/grub</code> (as i said before, a ton of fun)
* more things to install on new machines: <code>emerge dev-vcs/git vim</code>
* now for puppet: <code>USE="augeas vim-syntax" emerge puppet</code>
* before using puppet: <code>emerge eix && eix-update</code>
* clone puppet tree: <code>git clone https://github.com/purplehazech/purplehazech-orcatamer.git /etc/puppet/environments/development</code>
* install librarian: <code>gem19 install librarian-puppet</code>
* load puppet modules: <code>cd /etc/puppet/environments/development && librarian-puppet install</code>
* workaround some TODOs: <code>ln -s /etc/puppet/environments/development/ /vagrant && ulimit -n 2048 && emerge dev-ruby/rgen --autounmask-write && dispatch-conf && emerge dev-ruby/rgen</code>
* test if puppet is useable: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'notify{"test":}' --pluginsync</code>
* run puppet like so to find the first batch of stuff to fix: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync --noop</code>
* let puppet rip: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync </code>
* after running the last command until all the errors where fixed i can try to run in agent mode: <code>puppet agent --test --server=`hostname -f`</code>
** i still need to figure out why the <code>--server</code> flag is needed at this stage, somehow the agent is consulting DNS rather than <code>/etc/hosts</code>

now for some hacking that i did to test some concepts:
* setup openldap tooling: emerge openldap
* search for machine: <code>ldapsearch -D 'cn=Manager,dc=stoney-cloud,dc=org' -w admin '(&(objectClass=sstVirtualizationVirtualMachine)(sstNetworkHostname=kvm-0231))'</code>
* open ldap port in fw: <code>ldap_pub_out="10.1.130.13"</code> and <code>openTcpPortOut "${chains_out[pub]}" "$ldap_pub_out" "636"</code>
** i also need to configure <code>ldaps_int_in="${ip_int[vm-test-02]} ${ip_int[vm-test-03]} 192.168.140.136"</code> in <code>/usr/local/scripts/netfilter/local/chains/vm-test-01/vm-test-01_chain.sh</code> for the above to work.
** after all the above i can still not connect from my node to the ldap server. I'll have ot get the iptables gurus on board to solve this. We need more documentation on the setup if a as simple dev should be able to change this. At some point I might even consider puppetizing th eiptables config.

== TODOs ==
* refactor role and profile things into proper modules and use proper puppet:// data urls
* figure out why the betagarden overlay needs <code>ulimit -n 2048</code> to clone
* install rgen for puppet parser future at some sensible part of bootstrapping
* figure out what going on here: <code>Feb 22 22:30:01 vm-test-01 ulogd[30493]: p_kvm-0231_0_in Denied dst:: IN=vmbr0 OUT=vmbr0 MAC=01:00:5e:00:00:12:00:00:5e:00:01:03:08:00 SRC=192.168.140.2 DST=224.0.0.18 LEN=56 TOS=10 PREC=0x00 TTL=255 ID=33458 DF PROTO=112 MARK=0 </code>
* get rid of <code>/vargant</code> hard-deps.
* make git with USE="curl"

User:Lucas/Gentoo Install Notes

2014-02-23T12:07:35Z

Lucas:

* hack <code>/usr/local/scripts/netfilter/local/chains/vms/kvm_0231_chain.sh</code> on host to allow gateway conns
* first hd is /dev/vda
* default gentoo handbook install with lvm setup on vda3 and one large lv_root
* install lvm2 so you can build a lvm initramfs
** if you skip this you will have tons of fun loading lvm in the initramfs shell: <code>lvm vgscan --mknodes && lvm lvchange -a ly vg01/lv_root</code>
* kernel build with: <code>genkernel --install --lvm --menuconfig all</code> (do not use <code>--virtio</code>, activate them in menuconfig instead, I had heaps of fun hunting down all the modules)
** actually <code>genkernel --install --lvm --kernel-config=/root/kernel.config all</code> since lazy me hates using a ui
** the --virtio switch seems screwed due to some oldconfig changes with the <code>VIRTIO_MMIO</code> system, but i haven't looked into that more
* remember to also set <code>GRUB_CMDLINE_LINUX="dolvm"</code> in <code>/etc/default/grub</code> (as i said before, a ton of fun)
* more things to install on new machines: <code>emerge dev-vcs/git vim</code>
* now for puppet: <code>USE="augeas vim-syntax" emerge puppet</code>
* before using puppet: <code>emerge eix && eix-update</code>
* clone puppet tree: <code>git clone https://github.com/purplehazech/purplehazech-orcatamer.git /etc/puppet/environments/development</code>
* install librarian: <code>gem19 install librarian-puppet</code>
* load puppet modules: <code>cd /etc/puppet/environments/development && librarian-puppet install</code>
* workaround some TODOs: <code>ln -s /etc/puppet/environments/development/ /vagrant && mkdir /usr/local/portage && touch /usr/local/portage/make.conf && ulimit -n 2048 && emerge sudo && emerge dev-ruby/rgen --autounmask-write && dispatch-conf && emerge dev-ruby/rgen</code>
* test if puppet is useable: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'notify{"test":}' --pluginsync</code>
* run puppet like so to find the first batch of stuff to fix: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync --noop</code>
* let puppet rip: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync </code>
* after running the last command until all the errors where fixed i can try to run in agent mode: <code>puppet agent --test --server=`hostname -f`</code>
** i still need to figure out why the <code>--server</code> flag is needed at this stage, somehow the agent is consulting DNS rather than <code>/etc/hosts</code>

now for some hacking that i did to test some concepts:
* setup openldap tooling: emerge openldap
* search for machine: <code>ldapsearch -D 'cn=Manager,dc=stoney-cloud,dc=org' -w admin '(&(objectClass=sstVirtualizationVirtualMachine)(sstNetworkHostname=kvm-0231))'</code>
* open ldap port in fw: <code>ldap_pub_out="10.1.130.13"</code> and <code>openTcpPortOut "${chains_out[pub]}" "$ldap_pub_out" "636"</code>
** i also need to configure <code>ldaps_int_in="${ip_int[vm-test-02]} ${ip_int[vm-test-03]} 192.168.140.136"</code> in <code>/usr/local/scripts/netfilter/local/chains/vm-test-01/vm-test-01_chain.sh</code> for the above to work.
** after all the above i can still not connect from my node to the ldap server. I'll have ot get the iptables gurus on board to solve this. We need more documentation on the setup if a as simple dev should be able to change this. At some point I might even consider puppetizing th eiptables config.

== TODOs ==
* replace silly headers in orcatamer with block chars with something that most tools dont bork on (ie. some ascii art)
** I removed this on Puppetfile and Modulefile to get librarian to run
* use github https URLs through out, they are simply proxy friendlier everywhere
* refactor role and profile things into proper modules and use proper puppet:// data urls
* dont' depend on /usr/local/portage/make.conf
* figure out why the betagarden overlay needs <code>ulimit -n 2048</code> to clone
* figure out why layman-add from betagarden needs sudo
* install rgen for puppet parser future at some sensible part of bootstrapping
* figure out what going on here: <code>Feb 22 22:30:01 vm-test-01 ulogd[30493]: p_kvm-0231_0_in Denied dst:: IN=vmbr0 OUT=vmbr0 MAC=01:00:5e:00:00:12:00:00:5e:00:01:03:08:00 SRC=192.168.140.2 DST=224.0.0.18 LEN=56 TOS=10 PREC=0x00 TTL=255 ID=33458 DF PROTO=112 MARK=0 </code>

User:Lucas/Gentoo Install Notes

2014-02-22T21:49:03Z

Lucas:

* hack <code>/usr/local/scripts/netfilter/local/chains/vms/kvm_0231_chain.sh</code> on host to allow gateway conns
* first hd is /dev/vda
* default gentoo handbook install with lvm setup on vda3 and one large lv_root
* install lvm2 so you can build a lvm initramfs
** if you skip this you will have tons of fun loading lvm in the initramfs shell: <code>lvm vgscan --mknodes && lvm lvchange -a ly vg01/lv_root</code>
* kernel build with: <code>genkernel --install --lvm --menuconfig all</code> (do not use <code>--virtio</code>, activate them in menuconfig instead, I had heaps of fun hunting down all the modules)
** actually <code>genkernel --install --lvm --kernel-config=/root/kernel.config all</code> since lazy me hates using a ui
** the --virtio switch seems screwed due to some oldconfig changes with the <code>VIRTIO_MMIO</code> system, but i haven't looked into that more
* remember to also set <code>GRUB_CMDLINE_LINUX="dolvm"</code> in <code>/etc/default/grub</code> (as i said before, a ton of fun)
* more things to install on new machines: <code>emerge dev-vcs/git vim</code>
* now for puppet: <code>USE="augeas vim-syntax" emerge puppet</code>
* before using puppet: <code>emerge eix && eix-update</code>
* clone puppet tree: <code>git clone https://github.com/purplehazech/purplehazech-orcatamer.git /etc/puppet/environments/development</code>
* install librarian: <code>gem19 install librarian-puppet</code>
* load puppet modules: <code>cd /etc/puppet/environments/development && librarian-puppet install</code>
* workaround some TODOs: <code>ln -s /etc/puppet/environments/development/ /vagrant && mkdir /usr/local/portage && touch /usr/local/portage/make.conf && ulimit -n 2048 && emerge sudo && emerge dev-ruby/rgen --autounmask-write && dispatch-conf && emerge dev-ruby/rgen</code>
* test if puppet is useable: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'notify{"test":}' --pluginsync</code>
* run puppet like so to find the first batch of stuff to fix: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync --noop</code>
* let puppet rip: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync </code>
* after running the last command until all the errors where fixed i can try to run in agent mode: <code>puppet agent --test --server=`hostname -f`</code>
** i still need to figure out why the <code>--server</code> flag is needed at this stage, somehow the agent is consulting DNS rather than <code>/etc/hosts</code>

now for some hacking that i did to test some concepts:
* setup openldap tooling: emerge openldap
* search for machine: <code>ldapsearch -D 'cn=Manager,dc=stoney-cloud,dc=org' -w admin '(&(objectClass=sstVirtualizationVirtualMachine)(sstNetworkHostname=kvm-0231))'</code>
* open ldap port in fw: <code>ldap_pub_out="10.1.130.13"</code> and <code>openTcpPortOut "${chains_out[pub]}" "$ldap_pub_out" "636"</code>
** i also need to configure <code>ldaps_int_in="${ip_int[vm-test-02]} ${ip_int[vm-test-03]} 192.168.140.136"</code> in <code>/usr/local/scripts/netfilter/local/chains/vm-test-01/vm-test-01_chain.sh</code> for the above to work.

== TODOs ==
* replace silly headers in orcatamer with block chars with something that most tools dont bork on (ie. some ascii art)
** I removed this on Puppetfile and Modulefile to get librarian to run
* use github https URLs through out, they are simply proxy friendlier everywhere
* refactor role and profile things into proper modules and use proper puppet:// data urls
* dont' depend on /usr/local/portage/make.conf
* figure out why the betagarden overlay needs <code>ulimit -n 2048</code> to clone
* figure out why layman-add from betagarden needs sudo
* install rgen for puppet parser future at some sensible part of bootstrapping
* figure out what going on here: <code>Feb 22 22:30:01 vm-test-01 ulogd[30493]: p_kvm-0231_0_in Denied dst:: IN=vmbr0 OUT=vmbr0 MAC=01:00:5e:00:00:12:00:00:5e:00:01:03:08:00 SRC=192.168.140.2 DST=224.0.0.18 LEN=56 TOS=10 PREC=0x00 TTL=255 ID=33458 DF PROTO=112 MARK=0 </code>

User:Lucas/Gentoo Install Notes

2014-02-22T21:31:15Z

Lucas:

* hack <code>/usr/local/scripts/netfilter/local/chains/vms/kvm_0231_chain.sh</code> on host to allow gateway conns
* first hd is /dev/vda
* default gentoo handbook install with lvm setup on vda3 and one large lv_root
* install lvm2 so you can build a lvm initramfs
** if you skip this you will have tons of fun loading lvm in the initramfs shell: <code>lvm vgscan --mknodes && lvm lvchange -a ly vg01/lv_root</code>
* kernel build with: <code>genkernel --install --lvm --menuconfig all</code> (do not use <code>--virtio</code>, activate them in menuconfig instead, I had heaps of fun hunting down all the modules)
** actually <code>genkernel --install --lvm --kernel-config=/root/kernel.config all</code> since lazy me hates using a ui
** the --virtio switch seems screwed due to some oldconfig changes with the <code>VIRTIO_MMIO</code> system, but i haven't looked into that more
* remember to also set <code>GRUB_CMDLINE_LINUX="dolvm"</code> in <code>/etc/default/grub</code> (as i said before, a ton of fun)
* more things to install on new machines: <code>emerge dev-vcs/git vim</code>
* now for puppet: <code>USE="augeas vim-syntax" emerge puppet</code>
* before using puppet: <code>emerge eix && eix-update</code>
* clone puppet tree: <code>git clone https://github.com/purplehazech/purplehazech-orcatamer.git /etc/puppet/environments/development</code>
* install librarian: <code>gem19 install librarian-puppet</code>
* load puppet modules: <code>cd /etc/puppet/environments/development && librarian-puppet install</code>
* workaround some TODOs: <code>ln -s /etc/puppet/environments/development/ /vagrant && mkdir /usr/local/portage && touch /usr/local/portage/make.conf && ulimit -n 2048 && emerge sudo && emerge dev-ruby/rgen --autounmask-write && dispatch-conf && emerge dev-ruby/rgen</code>
* test if puppet is useable: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'notify{"test":}' --pluginsync</code>
* run puppet like so to find the first batch of stuff to fix: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync --noop</code>
* let puppet rip: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync </code>
* after running the last command until all the errors where fixed i can try to run in agent mode: <code>puppet agent --test --server=`hostname -f`</code>
** i still need to figure out why the <code>--server</code> flag is needed at this stage, somehow the agent is consulting DNS rather than <code>/etc/hosts</code>

now for some hacking that i did to test some concepts:
* setup openldap tooling: emerge openldap
* search for machine: <code>ldapsearch -D 'cn=Manager,dc=stoney-cloud,dc=org' -w admin '(&(objectClass=sstVirtualizationVirtualMachine)(sstNetworkHostname=kvm-0231))'</code>

== TODOs ==
* replace silly headers in orcatamer with block chars with something that most tools dont bork on (ie. some ascii art)
** I removed this on Puppetfile and Modulefile to get librarian to run
* use github https URLs through out, they are simply proxy friendlier everywhere
* refactor role and profile things into proper modules and use proper puppet:// data urls
* dont' depend on /usr/local/portage/make.conf
* figure out why the betagarden overlay needs <code>ulimit -n 2048</code> to clone
* figure out why layman-add from betagarden needs sudo
* install rgen for puppet parser future at some sensible part of bootstrapping
* figure out what going on here: <code>Feb 22 22:30:01 vm-test-01 ulogd[30493]: p_kvm-0231_0_in Denied dst:: IN=vmbr0 OUT=vmbr0 MAC=01:00:5e:00:00:12:00:00:5e:00:01:03:08:00 SRC=192.168.140.2 DST=224.0.0.18 LEN=56 TOS=10 PREC=0x00 TTL=255 ID=33458 DF PROTO=112 MARK=0 </code>

User:Lucas/Gentoo Install Notes

2014-02-22T21:06:07Z

Lucas:

* hack <code>/usr/local/scripts/netfilter/local/chains/vms/kvm_0231_chain.sh</code> on host to allow gateway conns
* first hd is /dev/vda
* default gentoo handbook install with lvm setup on vda3 and one large lv_root
* install lvm2 so you can build a lvm initramfs
** if you skip this you will have tons of fun loading lvm in the initramfs shell: <code>lvm vgscan --mknodes && lvm lvchange -a ly vg01/lv_root</code>
* kernel build with: <code>genkernel --install --lvm --menuconfig all</code> (do not use <code>--virtio</code>, activate them in menuconfig instead, I had heaps of fun hunting down all the modules)
** actually <code>genkernel --install --lvm --kernel-config=/root/kernel.config all</code> since lazy me hates using a ui
** the --virtio switch seems screwed due to some oldconfig changes with the <code>VIRTIO_MMIO</code> system, but i haven't looked into that more
* remember to also set <code>GRUB_CMDLINE_LINUX="dolvm"</code> in <code>/etc/default/grub</code> (as i said before, a ton of fun)
* more things to install on new machines: <code>emerge dev-vcs/git vim</code>
* now for puppet: <code>USE="augeas vim-syntax" emerge puppet</code>
* before using puppet: <code>emerge eix && eix-update</code>
* clone puppet tree: <code>git clone https://github.com/purplehazech/purplehazech-orcatamer.git /etc/puppet/environments/development</code>
* install librarian: <code>gem19 install librarian-puppet</code>
* load puppet modules: <code>cd /etc/puppet/environments/development && librarian-puppet install</code>
* workaround some TODOs: <code>ln -s /etc/puppet/environments/development/ /vagrant && mkdir /usr/local/portage && touch /usr/local/portage/make.conf && ulimit -n 2048 && emerge sudo && emerge dev-ruby/rgen --autounmask-write && dispatch-conf && emerge dev-ruby/rgen</code>
* test if puppet is useable: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'notify{"test":}' --pluginsync</code>
* run puppet like so to find the first batch of stuff to fix: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync --noop</code>
* let puppet rip: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync </code>
* after running the last command until all the errors where fixed i can try to run in agent mode: <code>puppet agent --test --server=`hostname -f`</code>
** i still need to figure out why the <code>--server</code> flag is needed at this stage, somehow the agent is consulting DNS rather than <code>/etc/hosts</code>

== TODOs ==
* replace silly headers in orcatamer with block chars with something that most tools dont bork on (ie. some ascii art)
** I removed this on Puppetfile and Modulefile to get librarian to run
* use github https URLs through out, they are simply proxy friendlier everywhere
* refactor role and profile things into proper modules and use proper puppet:// data urls
* dont' depend on /usr/local/portage/make.conf
* figure out why the betagarden overlay needs <code>ulimit -n 2048</code> to clone
* figure out why layman-add from betagarden needs sudo
* install rgen for puppet parser future at some sensible part of bootstrapping

User:Lucas/Gentoo Install Notes

2014-02-22T18:37:21Z

Lucas:

* hack <code>/usr/local/scripts/netfilter/local/chains/vms/kvm_0231_chain.sh</code> on host to allow gateway conns
* first hd is /dev/vda
* default gentoo handbook install with lvm setup on vda3 and one large lv_root
* install lvm2 so you can build a lvm initramfs
** if you skip this you will have tons of fun loading lvm in the initramfs shell: <code>lvm vgscan --mknodes && lvm lvchange -a ly vg01/lv_root</code>
* kernel build with: <code>genkernel --install --lvm --menuconfig all</code> (do not use <code>--virtio</code>, activate them in menuconfig instead, I had heaps of fun hunting down all the modules)
** actually <code>genkernel --install --lvm --kernel-config=/root/kernel.config all</code> since lazy me hates using a ui
** the --virtio switch seems screwed due to some oldconfig changes with the <code>VIRTIO_MMIO</code> system, but i haven't looked into that more
* remember to also set <code>GRUB_CMDLINE_LINUX="dolvm"</code> in <code>/etc/default/grub</code> (as i said before, a ton of fun)
* more things to install on new machines: <code>emerge dev-vcs/git vim</code>
* now for puppet: <code>USE="augeas vim-syntax" emerge puppet</code>
* before using puppet: <code>emerge eix && eix-update</code>
* clone puppet tree: <code>git clone https://github.com/purplehazech/purplehazech-orcatamer.git /etc/puppet/environments/development</code>
* install librarian: <code>gem19 install librarian-puppet</code>
* load puppet modules: <code>cd /etc/puppet/environments/development && librarian-puppet install</code>
* workaround some TODOs: <code>ln -s /etc/puppet/environments/development/ /vagrant && mkdir /usr/local/portage && touch /usr/local/portage/make.conf && ulimit -n 2048 && emerge sudo && emerge dev-ruby/rgen --autounmask-write && dispatch-conf && emerge dev-ruby/rgen</code>
* test if puppet is useable: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'notify{"test":}' --pluginsync</code>
* run puppet like so to find the first batch of stuff to fix: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync --noop</code>
* let puppet rip: <code>puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/development/manifests/ -e 'include ::role::puppet::master' --pluginsync </code>
* after running the last command until all the errors where fixed i can try to run in agent mode: <code>puppet agent --test --noop</code>

== TODOs ==
* replace silly headers in orcatamer with block chars with something that most tools dont bork on (ie. some ascii art)
** I removed this on Puppetfile and Modulefile to get librarian to run
* use github https URLs through out, they are simply proxy friendlier everywhere
* refactor role and profile things into proper modules and use proper puppet:// data urls
* dont' depend on /usr/local/portage/make.conf
* figure out why the betagarden overlay needs <code>ulimit -n 2048</code> to clone
* figure out why layman-add from betagarden needs sudo
* install rgen for puppet parser future at some sensible part of bootstrapping

User:Lucas/Gentoo Install Notes

2014-02-22T18:16:37Z

Lucas:

User:Lucas/Gentoo Install Notes

2014-02-22T18:09:24Z

Lucas:

User:Lucas/Gentoo Install Notes

2014-02-22T18:07:30Z

Lucas:

User:Lucas/Gentoo Install Notes

2014-02-22T18:05:18Z

Lucas:

User:Lucas/Gentoo Install Notes

2014-02-22T17:47:31Z

Lucas:

User:Lucas/Gentoo Install Notes

2014-02-22T17:40:44Z

Lucas:

User:Lucas/Gentoo Install Notes

2014-02-22T17:37:04Z

Lucas:

User:Lucas/Gentoo Install Notes

2014-02-22T17:35:29Z

Lucas:

User:Lucas/Gentoo Install Notes

2014-02-22T17:33:10Z

Lucas:

* hack /usr/local/scripts/netfilter/local/chains/vms/kvm_0231_chain.sh on host to allow gateway conns
* first hd is /dev/vda
* default gentoo handbook install with lvm setup on vda3 and one large lv_root
* install lvm2 so you can build a lvm initramfs
** if you skip this you will have tons of fun loading lvm in the initramfs shell: lvm vgscan --mknodes && lvm lvchange -a ly vg01/lv_root
* kernel build with: genkernel --install --lvm --menuconfig all (do not use --virtio, activate them in menuconfig instead, I had heaps of fun hunting down all the modules)
** actually genkernel --install --lvm --kernel-config=/root/kernel.config since lazy me hates using a ui
** the --virtio switch seems screwed due to some oldconfig changes with the VIRTIO_MMIO system, but i haven't looked into that more
* remember to also set GRUB_CMDLINE_LINUX="dolvm" in /etc/default/grub (as i said before, a ton of fun)
* more things to install on new machines: emerge dev-vcs/git vim
* now for puppet: USE="augeas vim-syntax" emerge puppet
* before using puppet: emerge eix && eix-update
* clone puppet tree: git clone https://github.com/purplehazech/purplehazech-orcatamer.git /etc/puppet/environments/development
* install librarian: gem19 install librarian-puppet
* load puppet modules: cd /etc/puppet/environments/development && librarian-puppet install
* run puppet like so to find the first batch of stuff to fix: puppet apply --environment=development --modulepath=/etc/puppet/environments/development/modules/:/etc/puppet/environments/developmen
t/manifests/ -e 'include ::role::puppet::master' --noop

== TODOs ==
* [ ] replace silly headers with block chars with something that most tools dont bork on (ie. some ascii art)
** I removed this on Puppetfile and Modulefile to get librarian to run
* use github https URLs through out, they are simply proxy friendlier everywhere
* refactor role and profile things into proper modules and use proper puppet:// data urls

User:Lucas/Gentoo Install Notes

2014-02-22T17:13:19Z

Lucas: Created page with "* hack /usr/local/scripts/netfilter/local/chains/vms/kvm_0231_chain.sh on host to allow gateway conns * first hd is /dev/vda * default gentoo handbook install with lvm setup o..."

User:Lucas

2014-02-22T13:33:55Z

Lucas:

Itsa Mee :)
== Things ==
* [[User:Lucas/Gentoo Install Notes]]

Gentoo Infrastructure

2014-02-06T17:27:05Z

Lucas: /* build orchestration */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build farm proposal ==
The build farm consists of a system of multiple vms to build binary packages for multiple environments, architectures and build profiles.

* Git webhook on internal gitlab install pushes changes to jenkins master.
* Jenkins master dishes out jobs to jenkins slave machines for needed architecture and build profile.
* Jenkins slaves only get used once and wipe/reprovision themselves after master has stored build artefacts.
* We have build-slave templates available for each architecture/build profile combo.
* Upon use those get provisioned to the needed environment using puppet.
* All of this is set up using puppet and fully automated, even building of new build-slave templates and the whole releng on those.
* The build farm also keeps old templates and stable boxes on hold so it can use them to build differentials.
* Artefacts slaves will be producing:
** "vagrant"-style boot boxes
** full binpkg repos for a given env/arch/build profile combo
** stage3 balls for each arch/build profile
** stage4 balls for each environment
** build logs
** <code>/var/db/pkg</code>
** puppet report data
** test results and code analysis results
* When we come to continuos deployment the jenkins master will also be able to trigger puppet when merges to master happen.
* This rolls out releases to the sub-system that was signed off by a merge to a master branch (see branching strategy in git proposal).

=== Links ===

==== build orchestration ====
* [http://mesos.apache.org/ Apache Mesos] cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. Can run for instance Jenkins.

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their [https://github.com/Sabayon/build build system "Matter"] might be of interest, it seems to automate large parts of tracking gentoo portage with its tinderbox subsystem
** sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
** is in dire need of DRY: [https://github.com/jedi4ever/veewee/pull/690] to make it worth forking
* [http://blinkeye.ch/dokuwiki/doku.php/projects/mkstage4 mkstage4]
** aimed at creating backup stage4 tarballs of gentoo systems
** written in bash
** pretty simple, might come in handy as automation tool

==== kernel ====

* at the moment we build tarballs for the kernel+initramfs and the modules using <code>genkernel</code> and have a separate ebuild which installs them
* ideally we would like to have an ebuild which takes the kernel sources (like the ebuild for <code>sys-kernel/gentoo-source</code> does), builds it according to some default configuration or a user configuration if available (<code>savedconfig.eclass</code>) and then installs the kernel and the modules as well as some minimal headers+configuration to build other packages requiring the sources to be present
* TODO: check whether dracut has some advantages regarding module loading over genkernel-generated initramfs

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* adhere to git-flow for all the things. Automate said usage as far as possible.
{|- class="wikitable"
! colspan=4 | git-flow branching
|-
! Branch
! Environment
! Merge from
! Description
|-
| <code>master</code>
| production
| <code>release/</code> or <code>hotfix/</code>
| Released code with a <code>git tag</code> for each merge.
|-
| <code>release/v0.0.0</code>
| staging
| <code>develop</code>
| Contains final releasing work like updating versioning and changelog. This is where we keep semver concerns in check if they where not taken care of already.
|-
| <code>hotfix/v0.0.0</code>
| staging
| <code>master</code>
| Only for critically urgent fixes. In most cases doing a release from <code>develop</code> is preferred.
|-
| <code>develop</code>
| development
| <code>feature/</code> or <code>master</code>
| Only feature branches that are ready for production should get merged here. <code>master</code> gets merged here after each merge to it. Merging is done with pull requests and review.
|-
| <code>feature/featurename</code>
| development
| <code>develop</code>
| New features get implemented here until they are considered ready for production and merged to <code>develop</code>.
|-
| <code>support/v0.0.0</code>
| LTS
|
| Marked experimental in most implementations and unused for now.
|}

* Install gitlab on a vm and integrate external mirrors from github and ldap users from stoney-ldap.
** keep repo of public mirrors in hieradata so we can configure them from puppet.
** each organisation in stoney-ldap automatically gets a private project in gitlab.
* Configure web hook intrastructure and integrate with continuous integration system.
* Make continuous integration show feedback back in gitlab.
** check for <code>git annotate</code> support or use img badges.

'''On organization projects in gitlab'''
* Each project comes with default repos.
{| class="wikitable"
! Repo
! Description
|-
| <code>puppet</code>
| Set up using a template, contains a Puppetfile and Puppetfile.lock and a hieradata directory.
|-
| <code>role</code>
| Read only copy of global role module for reference.
|-
| <code>profile</code>
| Read only copy of global profile module for reference.
|}
* Everything in the latter two modules is configurable through hieradata in the first repo.
* The default setup automatically updates <code>role</code> and <code>profile</code> when they get new merges.
* A software agent (ci) regularly clones <code>develop</code>, does a full build and pushes the results back to <code>feature/tinderbox</code>
* This agent autmatically creates pull requests if tinderbox builds did not fail.
* Org leaders may then merge these PRs and bake them into a local release.
* Some kind of UI helps them do this without much technical knowledge.
* More repos may be added by the customer.
* project organizations are private, per customer.

== Links ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood
* [https://github.com/sag47/gitlab-mirrors gitlab-mirrors] is a companion app to gitlab for adding readonly mirror repos to gitlab. We might consider hacking it to not use <code>git remote prune</code>.
* [http://www.javacodegeeks.com/2014/01/git-flow-with-jenkins-and-gitlab.html git-flow with jenkins and gitlab]
* [https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin gitlab hook for jenkins]

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

stoney orchestra: Roadmap

2014-02-02T21:24:41Z

Lucas:

== backlog ==

=== unsorted ===
These might be rather epic and need breaking down until they are sensible user-stories.

* investigate, document and implement hooks and git-scripts needed for continuous-(integration|development|.*)
* evaluate and decide between puppet-librarian, r10k or byoc solution for applying Puppetfile/Modulefile
* develop and establish "frontend" tooling like git-* scripts or puppet-rake tasks for devs and admins
* use mcollective for orchestration when provisioning new services thru puppet
* early bootstrap of puppet master during stoney cloud install

== links ==

* Architecture
** [[Gentoo_Infrastructure#Puppet_proposal]]
** [http://www.craigdunn.org/2012/05/239/ Craig Dunn's Blog: Designing Puppet – Roles and Profiles]
* Puppet Modules
** [http://forge.puppetlabs.com/gentoo/portage gentoo/portage puppet module]
** [http://forge.puppetlabs.com/purplehazech/syslogng syslogng module]
** [http://forge.puppetlabs.com/purplehazech/ccache ccache module] ;)

[[Category:stoney orchestra]][[Category:Roadmap]]

Main Page

2014-02-02T21:18:56Z

Lucas: make the infrastructure part be more marketing like and contain links to moar

<div style="float:left; width:67%">
The stoney cloud wiki acts as a collection of all aspects of a high availability cloud infrastructure. It is roughly divided into the following sections:
* '''Infrastructure''': The basis of the whole ecosystem.
* '''stoney cloud''': The actual cloud with a simple multi-tenant web based interface.
* '''Self-Service Modules''': Extension modules to expand the functionality of the stoney cloud.
<br />
</div>
<div style="float:right; width:33%">
== Quick Links ==
* [http://packages.stoney-cloud.org/stoney-cloud/pre-releases/1.2/iso/ stoney cloud: Download]
* [[stoney cloud: Demo-System Installation]]
* [[stoney cloud: Single-Node Installation]]
* [[stoney cloud: Multi-Node Installation]]
* [[stoney cloud: Upgrade]]
</div>
<div style="clear:both; height:0px; line-height:0px;"></div>

<div style="float:left; width:33%">

== Infrastructure ==
The stoney cloud builds upon the [http://www.gentoo.org/ Gentoo] Linux Distribution and is dependent on infrastructure projects like:
* Build Server
* Binary Package Server
* Mirror Server
* Puppet Server
While this infrastructure will serve stoney cloud in many ways, it may also be used to build cloud independent systems. We want you to be able to deploy desktops or more specialized gear using the same stack the infrastructure projects use to build services and manage machines for the cloud.

Currently some [[Gentoo Infrastructure|requirements and proposals]] have been imported from the RaBe Wiki and are being fleshed out and merged into this wiki proper.
== stoney cloud Wiki Write Access ==
If you would like write access to the stoney cloud Wiki, please send us a [mailto:support@stoney-cloud.org?Subject=stoney%20cloud%20Wiki%3A%20Request%20Account&body=Dear%20stoney%20cloud%20Support%20Team%0A%0AI%20hereby%20request%20a%20write%20access%20account%20to%20the%20stoney%20cloud%20Wiki.%0A%0AName%3A%20%0ASurname%3A%20%0AE-Mail%3A%0A%0AMany%20thanks%20in%20advance! request].
</div>

<div style="float:left; width:34%;">
== stoney cloud ==
The stoney cloud is an expandable multi-tenant web based Open Source Cloud management solution with service providers as it's target audience.

*[[:Category:stoney core|stoney core]]: Main framework responsible for shared functionality:
** [[:Category:REST_API|REST API]], which serves as a data and business logic abstraction layer and uses JSON as the primary data interchange format.
** User management, rights and roles.
** A consistent look and feel between modules.
** Internationalization.

*[[:Category:stoney conductor|stoney conductor]]: Responsible for:
** Storage allocation
** Network configuration
** Virtual machine profiles
** Virtual machine templates
** Virtual machines
** Virtual machine snapshots
** Virtual machine full backups
</div>

<div style="float:right; width:33%">
== Self-Service Modules ==
=== Existing ===
* [[:Category:stoney backup|stoney backup]]: On-line backup service for desktops, servers and virtual machines.

=== Work in progress ===
* [[:Category:stoney vm|stoney vm]]: Simplified sub set of the [[:Category:stoney conductor|stoney conductor]] functionality.
* [[:Category:stoney mail|stoney mail]]: Mail service with optional group-ware (based on Open-Xchange).

=== Planned ===
* [[:Category:stoney monitor|stoney monitor]]: Monitoring (with Zabbix).
* [[:Category:stoney orchestra|stoney orchestra]]: Configuration Management (with Puppet).
* [[:Category:stoney box|stoney box]]: An on-line storage service (will support WebDAV via HTTPS and SFTP, later CIFS as well).
* [[:Category:stoney web|stoney web]]: Web & Database hosting service (based on Apache and MariaDB).
</div>
<div style="clear:both; height:0px; line-height:0px;"></div>

__NOTOC__
__NOEDITSECTION__

Hiera Example

2014-02-01T15:24:02Z

Lucas:

<pre>
---
:backends:
- ldap
- yaml
- json
:ldap:
:url:ldaps://ldap.stoney-cloud.org:636/
:binddn:cn=Manager,dc=stoney-cloud,dc=org
:bindpw:secret
:basedn:dc=stoney-cloud,dc=org
:yaml:
:datadir: /etc/puppet/hieradata
:json:
:datadir: /etc/puppet/hieradata
:hierarchy:
- "ou=virtual machines,ou=services?sub?(&(sstNetworkHostName=%{::hostname})(sstNetworkDomainName=%{::domainname}))"
- "ou=software stack,ou=configuration?sub?(uid=%{::rzUid})"
- "%{::clientcert}"
- "%{::custom_location}"
- common
</pre>

Notes:
* This is an example of how a hiera config file might look with an mock ldap backend. The backend in question still needs to be found or written.
* mapping from a DN to directory structure would be nice, so we would rather have to write: ''ou=virtual machines/ou=services'' instead to be compatible with the already existing yaml/json backends or something different entirely
* this needs to take into consideration that puppet expects keys to be defined in a way to enable implicit parameter injection in parameterized classes
* existing ldap backends for hiera: [https://github.com/hunner/hiera-ldap], [http://forge.ircam.fr/p/hiera-ldap-backend/]
* we should probably aim at integrating this in hiera-2 with regards to ARM-8 and the already imlemented ARM-9
* I'm not convinced that we should not just grab all this data from the [[stoney_core:_REST_API]] and use that as an integration point for puppet.

[[Category: Documentation]]

Hiera Example

2014-02-01T15:14:50Z

Lucas:

<pre>
---
:backends:
- ldap
- yaml
- json
:ldap:
:url:ldaps://ldap.stoney-cloud.org:636/
:binddn:cn=Manager,dc=stoney-cloud,dc=org
:bindpw:secret
:basedn:dc=stoney-cloud,dc=org
:yaml:
:datadir: /etc/puppet/hieradata
:json:
:datadir: /etc/puppet/hieradata
:hierarchy:
- "ou=virtual machines,ou=services?sub?(&(sstNetworkHostName=%{::hostname})(sstNetworkDomainName=%{::domainname}))"
- "ou=software stack,ou=configuration?sub?(uid=%{::rzUid})"
- "%{::clientcert}"
- "%{::custom_location}"
- common
</pre>

Notes:
* This is an example of how a hiera config file might look with an mock ldap backend. The backend in question still needs to be found or written.
* mapping from a DN to directory structure would be nice, so we would rather have to write: ''ou=virtual machines/ou=services'' instead to be compatible with the already existing yaml/json backends or something different entirely
* this needs to take into consideration that puppet expects keys to be defined in a way to enable implicit parameter injection in parameterized classes
* existing ldap backends for hiera: [https://github.com/hunner/hiera-ldap], [http://forge.ircam.fr/p/hiera-ldap-backend/]
* we should probably aim at integrating this in hiera-2 with regards to ARM-8 and the already imlemented ARM-9
* I'm not convienced that we should not just grab all this data from the [[stoney_core:_REST_API]] and use that as an integration point fpr puppet.

[[Category: Documentation]]

stoney orchestra: Requirements

2014-02-01T11:54:34Z

Lucas: /* Requirements */

<noinclude>
== Overview ==
== Requirements ==
</noinclude>
* Support for all three environments (development, staging and production)
* Version controlled via Git
* ENC and hiera support with data from ldap
* Puppet recipes for
** installing, updating, removing and (re-)configuring specific software belonging to an application stack (see [[#Build_host_requirements|build host]]).
** (re-)configuring software belonging to a system stack
** Updating the system stack (<code>emerge @system</code>) aka system update.
** installing, updating and removing of kernel packages (including the handling of the ensuing reboot)
* use best-of-breed tools like hiera and augeas (this might mean targeting 3.3.x due to module data support in [https://github.com/puppetlabs/armatures/blob/master/arm-9.data_in_modules/index.md ARM-9])
* Use a sane prexisting puppet architecture concept
<noinclude>
[[Category:stoney orchestra]]
[[Category:Requirements]]
</noinclude>

Gentoo Infrastructure

2014-01-31T20:09:17Z

Lucas: /* Build farm proposal */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build farm proposal ==
The build farm consists of a system of multiple vms to build binary packages for multiple environments, architectures and build profiles.

* Git webhook on internal gitlab install pushes changes to jenkins master.
* Jenkins master dishes out jobs to jenkins slave machines for needed architecture and build profile.
* Jenkins slaves only get used once and wipe/reprovision themselves after master has stored build artefacts.
* We have build-slave templates available for each architecture/build profile combo.
* Upon use those get provisioned to the needed environment using puppet.
* All of this is set up using puppet and fully automated, even building of new build-slave templates and the whole releng on those.
* The build farm also keeps old templates and stable boxes on hold so it can use them to build differentials.
* Artefacts slaves will be producing:
** "vagrant"-style boot boxes
** full binpkg repos for a given env/arch/build profile combo
** stage3 balls for each arch/build profile
** stage4 balls for each environment
** build logs
** <code>/var/db/pkg</code>
** puppet report data
** test results and code analysis results
* When we come to continuos deployment the jenkins master will also be able to trigger puppet when merges to master happen.
* This rolls out releases to the sub-system that was signed off by a merge to a master branch (see branching strategy in git proposal).

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their [https://github.com/Sabayon/build build system "Matter"] might be of interest, it seems to automate large parts of tracking gentoo portage with its tinderbox subsystem
** sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
** is in dire need of DRY: [https://github.com/jedi4ever/veewee/pull/690] to make it worth forking
* [http://blinkeye.ch/dokuwiki/doku.php/projects/mkstage4 mkstage4]
** aimed at creating backup stage4 tarballs of gentoo systems
** written in bash
** pretty simple, might come in handy as automation tool

==== kernel ====

* at the moment we build tarballs for the kernel+initramfs and the modules using <code>genkernel</code> and have a separate ebuild which installs them
* ideally we would like to have an ebuild which takes the kernel sources (like the ebuild for <code>sys-kernel/gentoo-source</code> does), builds it according to some default configuration or a user configuration if available (<code>savedconfig.eclass</code>) and then installs the kernel and the modules as well as some minimal headers+configuration to build other packages requiring the sources to be present
* TODO: check whether dracut has some advantages regarding module loading over genkernel-generated initramfs

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* adhere to git-flow for all the things. Automate said usage as far as possible.
{|- class="wikitable"
! colspan=4 | git-flow branching
|-
! Branch
! Environment
! Merge from
! Description
|-
| <code>master</code>
| production
| <code>release/</code> or <code>hotfix/</code>
| Released code with a <code>git tag</code> for each merge.
|-
| <code>release/v0.0.0</code>
| staging
| <code>develop</code>
| Contains final releasing work like updating versioning and changelog. This is where we keep semver concerns in check if they where not taken care of already.
|-
| <code>hotfix/v0.0.0</code>
| staging
| <code>master</code>
| Only for critically urgent fixes. In most cases doing a release from <code>develop</code> is preferred.
|-
| <code>develop</code>
| development
| <code>feature/</code> or <code>master</code>
| Only feature branches that are ready for production should get merged here. <code>master</code> gets merged here after each merge to it. Merging is done with pull requests and review.
|-
| <code>feature/featurename</code>
| development
| <code>develop</code>
| New features get implemented here until they are considered ready for production and merged to <code>develop</code>.
|-
| <code>support/v0.0.0</code>
| LTS
|
| Marked experimental in most implementations and unused for now.
|}

* Install gitlab on a vm and integrate external mirrors from github and ldap users from stoney-ldap.
** keep repo of public mirrors in hieradata so we can configure them from puppet.
** each organisation in stoney-ldap automatically gets a private project in gitlab.
* Configure web hook intrastructure and integrate with continuous integration system.
* Make continuous integration show feedback back in gitlab.
** check for <code>git annotate</code> support or use img badges.

'''On organization projects in gitlab'''
* Each project comes with default repos.
{| class="wikitable"
! Repo
! Description
|-
| <code>puppet</code>
| Set up using a template, contains a Puppetfile and Puppetfile.lock and a hieradata directory.
|-
| <code>role</code>
| Read only copy of global role module for reference.
|-
| <code>profile</code>
| Read only copy of global profile module for reference.
|}
* Everything in the latter two modules is configurable through hieradata in the first repo.
* The default setup automatically updates <code>role</code> and <code>profile</code> when they get new merges.
* A software agent (ci) regularly clones <code>develop</code>, does a full build and pushes the results back to <code>feature/tinderbox</code>
* This agent autmatically creates pull requests if tinderbox builds did not fail.
* Org leaders may then merge these PRs and bake them into a local release.
* Some kind of UI helps them do this without much technical knowledge.
* More repos may be added by the customer.
* project organizations are private, per customer.

== Links ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood
* [https://github.com/sag47/gitlab-mirrors gitlab-mirrors] is a companion app to gitlab for adding readonly mirror repos to gitlab. We might consider hacking it to not use <code>git remote prune</code>.
* [http://www.javacodegeeks.com/2014/01/git-flow-with-jenkins-and-gitlab.html git-flow with jenkins and gitlab]
* [https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin gitlab hook for jenkins]

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-31T20:07:04Z

Lucas: /* Build farm proposal */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build farm proposal ==
The build farm consists of a system of multiple vms to build binary packages for multiple environments, architectures and build profiles.

* Git webhook on internal gitlab install pushes changes to jenkins master.
* Jenkins master dishes out jobs to jenkins slave machines for needed architecture and build profile.
* Jenkins slaves only get used once and wipe/reprovision themselves after master has stored build artefacts.
* We have build-slave templates available for each architecture/build profile combo.
* Upon use those get provisioned to the needed environment using puppet.
* All of this is set up using puppet and fully automated, even building of new build-slave templates and the whole releng on those.
* The build farm also keeps old templates and stable boxes on hold so it can use them to build differentials.
* Artefacts slaves will be producing:
** "vagrant"-style boot boxes
** full binpkg repos for a given env/arch/build profile combo
** stage3 balls for each arch/build profile
** stage4 balls for each environment
** build logs
** <code>/var/db/pkg</code>
** puppet report data
** test results and code analysis results
* When we come to continuos deployment the jenkins master will also be able to trigger puppet when merges to master happen. Thus rolling out releases to the sub-system that was signed off by a merge to a master branch.

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their [https://github.com/Sabayon/build build system "Matter"] might be of interest, it seems to automate large parts of tracking gentoo portage with its tinderbox subsystem
** sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
** is in dire need of DRY: [https://github.com/jedi4ever/veewee/pull/690] to make it worth forking
* [http://blinkeye.ch/dokuwiki/doku.php/projects/mkstage4 mkstage4]
** aimed at creating backup stage4 tarballs of gentoo systems
** written in bash
** pretty simple, might come in handy as automation tool

==== kernel ====

* at the moment we build tarballs for the kernel+initramfs and the modules using <code>genkernel</code> and have a separate ebuild which installs them
* ideally we would like to have an ebuild which takes the kernel sources (like the ebuild for <code>sys-kernel/gentoo-source</code> does), builds it according to some default configuration or a user configuration if available (<code>savedconfig.eclass</code>) and then installs the kernel and the modules as well as some minimal headers+configuration to build other packages requiring the sources to be present
* TODO: check whether dracut has some advantages regarding module loading over genkernel-generated initramfs

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* adhere to git-flow for all the things. Automate said usage as far as possible.
{|- class="wikitable"
! colspan=4 | git-flow branching
|-
! Branch
! Environment
! Merge from
! Description
|-
| <code>master</code>
| production
| <code>release/</code> or <code>hotfix/</code>
| Released code with a <code>git tag</code> for each merge.
|-
| <code>release/v0.0.0</code>
| staging
| <code>develop</code>
| Contains final releasing work like updating versioning and changelog. This is where we keep semver concerns in check if they where not taken care of already.
|-
| <code>hotfix/v0.0.0</code>
| staging
| <code>master</code>
| Only for critically urgent fixes. In most cases doing a release from <code>develop</code> is preferred.
|-
| <code>develop</code>
| development
| <code>feature/</code> or <code>master</code>
| Only feature branches that are ready for production should get merged here. <code>master</code> gets merged here after each merge to it. Merging is done with pull requests and review.
|-
| <code>feature/featurename</code>
| development
| <code>develop</code>
| New features get implemented here until they are considered ready for production and merged to <code>develop</code>.
|-
| <code>support/v0.0.0</code>
| LTS
|
| Marked experimental in most implementations and unused for now.
|}

* Install gitlab on a vm and integrate external mirrors from github and ldap users from stoney-ldap.
** keep repo of public mirrors in hieradata so we can configure them from puppet.
** each organisation in stoney-ldap automatically gets a private project in gitlab.
* Configure web hook intrastructure and integrate with continuous integration system.
* Make continuous integration show feedback back in gitlab.
** check for <code>git annotate</code> support or use img badges.

'''On organization projects in gitlab'''
* Each project comes with default repos.
{| class="wikitable"
! Repo
! Description
|-
| <code>puppet</code>
| Set up using a template, contains a Puppetfile and Puppetfile.lock and a hieradata directory.
|-
| <code>role</code>
| Read only copy of global role module for reference.
|-
| <code>profile</code>
| Read only copy of global profile module for reference.
|}
* Everything in the latter two modules is configurable through hieradata in the first repo.
* The default setup automatically updates <code>role</code> and <code>profile</code> when they get new merges.
* A software agent (ci) regularly clones <code>develop</code>, does a full build and pushes the results back to <code>feature/tinderbox</code>
* This agent autmatically creates pull requests if tinderbox builds did not fail.
* Org leaders may then merge these PRs and bake them into a local release.
* Some kind of UI helps them do this without much technical knowledge.
* More repos may be added by the customer.
* project organizations are private, per customer.

== Links ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood
* [https://github.com/sag47/gitlab-mirrors gitlab-mirrors] is a companion app to gitlab for adding readonly mirror repos to gitlab. We might consider hacking it to not use <code>git remote prune</code>.
* [http://www.javacodegeeks.com/2014/01/git-flow-with-jenkins-and-gitlab.html git-flow with jenkins and gitlab]
* [https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin gitlab hook for jenkins]

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-31T19:52:17Z

Lucas: /* Build host proposal */ -> build farm

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build farm proposal ==
The build farm consists out of a system of multiple vms to build binary packages for multiple environments, architectures and build profiles.

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their [https://github.com/Sabayon/build build system "Matter"] might be of interest, it seems to automate large parts of tracking gentoo portage with its tinderbox subsystem
** sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
** is in dire need of DRY: [https://github.com/jedi4ever/veewee/pull/690] to make it worth forking
* [http://blinkeye.ch/dokuwiki/doku.php/projects/mkstage4 mkstage4]
** aimed at creating backup stage4 tarballs of gentoo systems
** written in bash
** pretty simple, might come in handy as automation tool

==== kernel ====

* at the moment we build tarballs for the kernel+initramfs and the modules using <code>genkernel</code> and have a separate ebuild which installs them
* ideally we would like to have an ebuild which takes the kernel sources (like the ebuild for <code>sys-kernel/gentoo-source</code> does), builds it according to some default configuration or a user configuration if available (<code>savedconfig.eclass</code>) and then installs the kernel and the modules as well as some minimal headers+configuration to build other packages requiring the sources to be present
* TODO: check whether dracut has some advantages regarding module loading over genkernel-generated initramfs

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* adhere to git-flow for all the things. Automate said usage as far as possible.
{|- class="wikitable"
! colspan=4 | git-flow branching
|-
! Branch
! Environment
! Merge from
! Description
|-
| <code>master</code>
| production
| <code>release/</code> or <code>hotfix/</code>
| Released code with a <code>git tag</code> for each merge.
|-
| <code>release/v0.0.0</code>
| staging
| <code>develop</code>
| Contains final releasing work like updating versioning and changelog. This is where we keep semver concerns in check if they where not taken care of already.
|-
| <code>hotfix/v0.0.0</code>
| staging
| <code>master</code>
| Only for critically urgent fixes. In most cases doing a release from <code>develop</code> is preferred.
|-
| <code>develop</code>
| development
| <code>feature/</code> or <code>master</code>
| Only feature branches that are ready for production should get merged here. <code>master</code> gets merged here after each merge to it. Merging is done with pull requests and review.
|-
| <code>feature/featurename</code>
| development
| <code>develop</code>
| New features get implemented here until they are considered ready for production and merged to <code>develop</code>.
|-
| <code>support/v0.0.0</code>
| LTS
|
| Marked experimental in most implementations and unused for now.
|}

* Install gitlab on a vm and integrate external mirrors from github and ldap users from stoney-ldap.
** keep repo of public mirrors in hieradata so we can configure them from puppet.
** each organisation in stoney-ldap automatically gets a private project in gitlab.
* Configure web hook intrastructure and integrate with continuous integration system.
* Make continuous integration show feedback back in gitlab.
** check for <code>git annotate</code> support or use img badges.

'''On organization projects in gitlab'''
* Each project comes with default repos.
{| class="wikitable"
! Repo
! Description
|-
| <code>puppet</code>
| Set up using a template, contains a Puppetfile and Puppetfile.lock and a hieradata directory.
|-
| <code>role</code>
| Read only copy of global role module for reference.
|-
| <code>profile</code>
| Read only copy of global profile module for reference.
|}
* Everything in the latter two modules is configurable through hieradata in the first repo.
* The default setup automatically updates <code>role</code> and <code>profile</code> when they get new merges.
* A software agent (ci) regularly clones <code>develop</code>, does a full build and pushes the results back to <code>feature/tinderbox</code>
* This agent autmatically creates pull requests if tinderbox builds did not fail.
* Org leaders may then merge these PRs and bake them into a local release.
* Some kind of UI helps them do this without much technical knowledge.
* More repos may be added by the customer.
* project organizations are private, per customer.

== Links ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood
* [https://github.com/sag47/gitlab-mirrors gitlab-mirrors] is a companion app to gitlab for adding readonly mirror repos to gitlab. We might consider hacking it to not use <code>git remote prune</code>.
* [http://www.javacodegeeks.com/2014/01/git-flow-with-jenkins-and-gitlab.html git-flow with jenkins and gitlab]
* [https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin gitlab hook for jenkins]

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-31T19:50:07Z

Lucas: /* git hosting proposal */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their [https://github.com/Sabayon/build build system "Matter"] might be of interest, it seems to automate large parts of tracking gentoo portage with its tinderbox subsystem
** sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
** is in dire need of DRY: [https://github.com/jedi4ever/veewee/pull/690] to make it worth forking
* [http://blinkeye.ch/dokuwiki/doku.php/projects/mkstage4 mkstage4]
** aimed at creating backup stage4 tarballs of gentoo systems
** written in bash
** pretty simple, might come in handy as automation tool

==== kernel ====

* at the moment we build tarballs for the kernel+initramfs and the modules using <code>genkernel</code> and have a separate ebuild which installs them
* ideally we would like to have an ebuild which takes the kernel sources (like the ebuild for <code>sys-kernel/gentoo-source</code> does), builds it according to some default configuration or a user configuration if available (<code>savedconfig.eclass</code>) and then installs the kernel and the modules as well as some minimal headers+configuration to build other packages requiring the sources to be present
* TODO: check whether dracut has some advantages regarding module loading over genkernel-generated initramfs

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* adhere to git-flow for all the things. Automate said usage as far as possible.
{|- class="wikitable"
! colspan=4 | git-flow branching
|-
! Branch
! Environment
! Merge from
! Description
|-
| <code>master</code>
| production
| <code>release/</code> or <code>hotfix/</code>
| Released code with a <code>git tag</code> for each merge.
|-
| <code>release/v0.0.0</code>
| staging
| <code>develop</code>
| Contains final releasing work like updating versioning and changelog. This is where we keep semver concerns in check if they where not taken care of already.
|-
| <code>hotfix/v0.0.0</code>
| staging
| <code>master</code>
| Only for critically urgent fixes. In most cases doing a release from <code>develop</code> is preferred.
|-
| <code>develop</code>
| development
| <code>feature/</code> or <code>master</code>
| Only feature branches that are ready for production should get merged here. <code>master</code> gets merged here after each merge to it. Merging is done with pull requests and review.
|-
| <code>feature/featurename</code>
| development
| <code>develop</code>
| New features get implemented here until they are considered ready for production and merged to <code>develop</code>.
|-
| <code>support/v0.0.0</code>
| LTS
|
| Marked experimental in most implementations and unused for now.
|}

* Install gitlab on a vm and integrate external mirrors from github and ldap users from stoney-ldap.
** keep repo of public mirrors in hieradata so we can configure them from puppet.
** each organisation in stoney-ldap automatically gets a private project in gitlab.
* Configure web hook intrastructure and integrate with continuous integration system.
* Make continuous integration show feedback back in gitlab.
** check for <code>git annotate</code> support or use img badges.

'''On organization projects in gitlab'''
* Each project comes with default repos.
{| class="wikitable"
! Repo
! Description
|-
| <code>puppet</code>
| Set up using a template, contains a Puppetfile and Puppetfile.lock and a hieradata directory.
|-
| <code>role</code>
| Read only copy of global role module for reference.
|-
| <code>profile</code>
| Read only copy of global profile module for reference.
|}
* Everything in the latter two modules is configurable through hieradata in the first repo.
* The default setup automatically updates <code>role</code> and <code>profile</code> when they get new merges.
* A software agent (ci) regularly clones <code>develop</code>, does a full build and pushes the results back to <code>feature/tinderbox</code>
* This agent autmatically creates pull requests if tinderbox builds did not fail.
* Org leaders may then merge these PRs and bake them into a local release.
* Some kind of UI helps them do this without much technical knowledge.
* More repos may be added by the customer.
* project organizations are private, per customer.

== Links ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood
* [https://github.com/sag47/gitlab-mirrors gitlab-mirrors] is a companion app to gitlab for adding readonly mirror repos to gitlab. We might consider hacking it to not use <code>git remote prune</code>.
* [http://www.javacodegeeks.com/2014/01/git-flow-with-jenkins-and-gitlab.html git-flow with jenkins and gitlab]
* [https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin gitlab hook for jenkins]

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-31T18:22:35Z

Lucas: /* "stage4"/box/iso building */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their [https://github.com/Sabayon/build build system "Matter"] might be of interest, it seems to automate large parts of tracking gentoo portage with its tinderbox subsystem
** sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
** is in dire need of DRY: [https://github.com/jedi4ever/veewee/pull/690] to make it worth forking
* [http://blinkeye.ch/dokuwiki/doku.php/projects/mkstage4 mkstage4]
** aimed at creating backup stage4 tarballs of gentoo systems
** written in bash
** pretty simple, might come in handy as automation tool

==== kernel ====

* at the moment we build tarballs for the kernel+initramfs and the modules using <code>genkernel</code> and have a separate ebuild which installs them
* ideally we would like to have an ebuild which takes the kernel sources (like the ebuild for <code>sys-kernel/gentoo-source</code> does), builds it according to some default configuration or a user configuration if available (<code>savedconfig.eclass</code>) and then installs the kernel and the modules as well as some minimal headers+configuration to build other packages requiring the sources to be present
* TODO: check whether dracut has some advantages regarding module loading over genkernel-generated initramfs

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-11T21:41:34Z

Lucas: /* "stage4"/box/iso building */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their [https://github.com/Sabayon/build build system "Matter"] might be of interest, it seems to automate large parts of tracking gentoo portage with its tinderbox subsystem
** sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
** is in dire need of DRY: [https://github.com/jedi4ever/veewee/pull/690] to make it worth forking

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-11T14:59:16Z

Lucas: /* package building */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their [https://github.com/Sabayon/build build system "Matter"] might be of interest, it seems to automate large parts of tracking gentoo portage with its tinderbox subsystem
** sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-11T14:56:04Z

Lucas: /* package building */ some sabayon stuff

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.
* [https://wiki.sabayon.org/?title=En:Entropy entropy] is sabayons portage replacement, it focuses on binaries due to sabayon being a binary distribution
** their build system "Matter" might be of interest (although I haven;t found any info), also sabayon has <code>kernel-switcher</code> for updating kernels
** kernel ebuilds live [https://github.com/Sabayon/sabayon-distro/tree/master/sys-kernel/linux-sabayon here] and probably rely on the [https://github.com/Sabayon/sabayon-distro/blob/master/eclass/sabayon-kernel.eclass sabayon-kernel eclass].

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-11T10:37:39Z

Lucas: /* Build host proposal */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===

==== build orchestration ====

==== package building ====
* [http://www.chromium.org/chromium-os/developer-guide/chromite-shell-quick-start chromite] build utility from chromium os ([https://chromium.googlesource.com/chromiumos/chromite/ source repo])
** as far as I recall chromium os does highly parallel building making their build really fast with a slight trade of in long termn stability (ie. build might fail due to dependencies being built out of oder),
** the [http://www.chromium.org/chromium-os/developer-guide chromium os developer guide] might also be of interest, among other things it shows that google do split the build into a package building part and an image creation part.

==== "stage4"/box/iso building ====
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-11T10:19:21Z

Lucas: /* Links */

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
**** The <code>net-analyzer/zabbix</code> is definitely a good example, I don't want to install and maintain MySQL, Apache, PHP, snmpd (including all the deps) etc. on hosts which just need a Zabbix agent. I would also like to pragmatically avoid unused deps, in order to minimize reverse-updates and security updates (which must be provided nonetheless if the software is in use or not). --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 13:20, 10 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box
** currently packer and packer-warehouse do not seem capable of building gentoo machines out of the box, I tested this with osx/virtualbox using gentoo stage3 and portage snapshots [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)
* [https://github.com/jedi4ever/veewee veewee] vagrant box builder (builds stage4 images in a manner similar to packer
** has support for a massive amount of guest os types
*** installs puppet/chef using gem due to the oldish versions in gentoo (and probably elsewhere)
** supports kvm and others as host os
** while testing with osx/virtualbox I was able to build and export a vagrant box from gentoo stage3 and portage snapshots without any hiccups [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 11:19, 11 January 2014 (CET)

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-09T18:52:35Z

Lucas: /* Binary package requirements */ my 2cts on the "different USE flags" thing

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
*** Yes and no on this one. We clearly need to keep the list of packages that require this at bare minimum. <code>net-analyzer/zabbix</code> for instance doesn't warrant this, we just won't start the server on non server nodes. Easy as cake. The server code and it's deps wont do any harm on say a desktop or other server box. Even though I can't think of example, I do believe we will be needing this possibility when we encounter packages that need to be built using different profiles for different use cases, things like having a php with-curlwrappers vs one with the curl module sans curlwrappers. The important point I take from this is that creating new profiles with small deviations from our default must be very easy (ie. not much work). Basically we need the infras support for n different build profiles to be fully automated and well documented. [[User:Lucas|Lucas]] ([[User talk:Lucas|talk]]) 19:52, 9 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-09T18:38:43Z

Lucas: /* Install host proposal */ add tools concerned with provisioning and handing control to puppet after

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

=== Links ===
* Tools that run puppet on freshly installed machines (and also do some provisioning)
** [https://forge.puppetlabs.com/puppetlabs/razor puppetlabs razor] bare metal/cloud provisioning tool
** [http://www.vagrantup.com/ vagrant] cloud provisioning aimed at provisioning developer boxes (with virtualbox). Has 3rd party support for various cloud systems. Vagrant might be interesting for creating dev clouds. I've seen this being used on production sites.

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-09T18:33:04Z

Lucas: /* Links */ gentoo example

= Overview =
This article describes how we plan on using gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

== Glossary ==
@TODO We need to clean up some terms already (for instance the portage vs puppet profile thing) A [[:Category:Glossary|glossary]] should help us define term more closely (and stick to the definitions).

; portage profile
: A profile in gentoo portage. Defines either a system or application stack for portage.
; portage build profile
: A profile in gentoo portage. Based of a system profile but used during the build phase of the binary packages used in the final deploy.
; puppet profile
: A puppet profile contains the implementation logic of how to install and configure an aspect of a system.
; stack
: A stack contains a complete and deployable product that may be provisioned and used. Stack have very simple inheritance letting the admin create stack trees based on each other. For instance a Ruby on Rails stack will be based of of a ruby stack which is based off a linux stack.

= Required components =
* Build host(s) for binary packages
* HTTP server for serving binary packages and distfiles (required by the ebuilds)
* Git clone of official portage tree
* Overlay(s)
* Own portage profile(s)
* rsync or Git server for serving the Overlay and the portage profiles
* Stage3 building system
* Puppet for configuration management and software installation
* Git version control for everything (overlays, portage profiles, puppet manifests and scripts/code)
* Install host (PXE boot / TFTP / DHCP)
** emc/puppetlabs [https://github.com/puppetlabs/Razor razor] can do this but needs some work for gentoo
* Automatic base installation script
** also in the scope of razor
* Separation of development, staging and production environments
** tagged and managed in git
* PKI environment (with dedicated sub CAs) for X509 certificates (used for Puppet, server and client certs etc.)
* git web interface (make dotfiles and frozen clones accessible to power-users)
* Central authentication service
* DNS, DHCP and NTP services
* Monitoring and alarming system
* Logging
* versioning for everything (if it is a committable file, use semver on its repo)

== Binary package requirements ==
* Ability to build and install binary packages with the same version but different USE flags. For example, MySQL server package (<code>-minimal</code> and MySQL client & libs package <code>minimal</code>)
** don't go there: this imposes a significant amount of maintenance work and may still break. Rather provide large enough base sets and accept that some packages install too much (you can still disable them at runtime) and build the few deviations from the rule on the servers from source --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:39, 3 January 2014 (CET)
*** Yes, we need to and can go there :-) I agree with you, that we should do this only if necessary, apache for example can be built once and has the ability to turn features (module loading) on/off via its configuration. Other software does not provide such run-time configuration which results in unwanted server-software and dependencies on the installed hosts (<code>net-analyzer/zabbix</code> for example). I clearly do not want to have a dedicated build environment for each of those packages, I would rather see a build env, called minimal for example, which is used to build all those database packages with only lib and clients enabled (use the same env for PostgreSQL, OpenLDAP, MySQL etc.). As stated before, the whole build process needs to be automated, so I don't see a considerable increase of maintenance work coming up here. The dependency problem is mitigated through the fact that we have a frozen portage tree for all our build envs and therefore use the same versions everywhere. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 12:04, 6 January 2014 (CET)
* Providing binary packages for different major (and sometimes minor) versions, for example: <code>dev-db/mysql-5.X.Y</code> and <code>dev-db/mysql-6.X.Y</code>.
* Provide binary packages for pre-compiled Linux kernels and modules (not just a binary package of <code>sys-kernel/gentoo-sources</code>)
** This makes it possible to build stage4 images from binary packages.
** Most likely there will be separate packages for servers and desktops built with different genkernel configs.
* Handle reverse dependency updates and ABI changes

== Build host requirements ==
* Build binary package for all required software
* Support for multiple environments (development, staging and production)
* Support for multiple architectures (such as x86, amd64 etc.)
* Support for multiple build profiles
** system (or base) profile, such as desktop or server (stage3) (all the packages contained within the <code>/etc/portage/make.profile</code> or via <code>emerge @system</code>)
** application profiles, such as php5-app, django-app etc.)
** simple inheritance is used for things like python-app -> django-app
** stacks consist of one system profile and multiple application profiles
** don't do this: Gentoo itself has only a few profiles and even there issues arise when combining them (for example desktop + selinux-hardened) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:40, 3 January 2014 (CET)
*** Those are build-profiles (for example chroots or some sort of overlay-fs) not Gentoo (portage) profiles, we definitely need to clarify those terms ;) --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 20:01, 5 January 2014 (CET)
* All build profiles will use a system profile as their base profile
* Ability to update an existing build profile, without the need to build it from scratch
* Ability to do fully automated clean builds (ie. for new archs or new stacks)
* Ability to automatically update all development profiles on a predefined frequency such as daily, weekly or monthly an be notified about build failures
** [http://jenkins-ci.org/ jenkins ci] can do this using one jenkins master and a least one build slave per architecture.
** Other options would be [https://github.com/travis-ci/travis-ci travis ci] (not ready for in-house use) or [http://cruisecontrol.sourceforge.net/ cruise control]
** Rabe already has a jenkins instance: [http://intranet.rabe.ch/jenkins/]. The instance [[Jenkins-01]] is more or less modern and should be easy to reintegrate with puppet.
* Each build profile stores the built binary packages under a per-defined directory which will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Application build profiles stores only the extra packages within the above directory, packages included in a base profile won't be duplicated.
* Old or no longer supported packages will be removed automatically
* Build a stage 3 tarball, which can be used for the automatic installation via PXE/TFTP.
** must be able to build a stage tarball for each of the available environment-arch-system profile combinations
* Handle reverse dependency updates and ABI changes (aka <code>revdep-rebuild</code>)
* Handle perl and python (maybe more) dependency updates (aka <code>perl-cleaner</code> & <code>python-updater</code>)
* Ability to build kernel and modules

== Portage tree clone requirements ==
* The official portage tree needs to be cloned via Git, which basically enables one to:
** keep the control over portage tree updates
** provide an old version of the tree
** cherry pick updates
*** this should be avoided at all cost since it can lead to various sorts of breakages (ebuild <-> ebuild, ebuild <-> eclass, ebuild <-> profile, eclass <-> profile interaction) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:24, 3 January 2014 (CET)
**** Yes, I agree. Nonetheless, we need the ''possibility'' to do cherry picking, for example to react on zero-day exploits. --[[User:Chrigu|Chrigu]] ([[User talk:Chrigu|talk]]) 19:53, 5 January 2014 (CET)
* Support for a development, staging and production branch
** Ability to automatically sync from upstream
** Easy merge support from one branch to the next ''higher'' one (staging -> production)
* Notification support for new [http://www.gentoo.org/security/en/glsa/index.xml GLSAs] which affect packages within the cloned trees.
** Either via automatic update and merge of <code>/usr/portage/metadata/glsa</code> or via external mechanisms such as consulting the [http://www.gentoo.org/rdf/en/glsa-index.rdf RDF feed].
** Having an inventory by collecting puppet facts allows to check for security updates in a central location --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:31, 3 January 2014 (CET)

== Portage overlay requirements ==
* One Git based portage [http://www.gentoo.org/proj/en/overlays/userguide.xml overlay]
** Contains own [[#Portage_profile_requirements|portage profiles]]
** Contains own or modified ebuilds or legacy ones removed from the official tree
* Support for development, staging and production environment (via Git branches)
* [http://layman.sourceforge.net/ Layman] compatibility
** Portage has now direct repository support (as has cave/paludis) and layman may be omitted --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:32, 3 January 2014 (CET)

== Portage profile requirements ==
* Multiple [http://wiki.gentoo.org/wiki/Profile Portage profiles] stored within the [[#Overlay_requirements|overlay]].
** One for base, desktop and server (maybe more in the future, such as streambox)
*** desktop and server both inherit from the base profile which serves as the lowest common denominator.
* Support for multiple architectures (such as x86 and amd64)
** Avoid definition duplications via parent profile inheriting.
* All the profiles have an official Gentoo profile as their master
* Profiles include only packages belonging to a base system, not an application stack (those will be managed via puppet recipes)
* Profiles can be used to unmask packages required but not belonging to the base system
* Profiles sets all the default values for the client's [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html <code>make.conf</code>], such as USE flags, BINHOSTS, GENTOO_MIRRORS, CFLAGS, CHOST etc.
** '''Warning''': many such variables are not incremental and therefore need duplication of Gentoo base profile variables (requiring that someone tracks changes in those variables) --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:29, 3 January 2014 (CET)
* keep the profiles (and the inheritance structure) as simple as possible, rather duplicate than inherit for small deviations to avoid inheritence issues --[[User:Tiziano|Tiziano]] ([[User talk:Tiziano|talk]]) 14:33, 3 January 2014 (CET)

== Package host requirements ==
* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/ENVIRONMENT/gentoo/ARCH/BUILD-PROFILE-NAME</code>.
*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEM-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)
** Acts as a caching mirror for already downloaded packages from an official mirror
** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients
* Files are served via HTTPS
* Distinguishes between three groups of files
** '''public''': Files which are available to all clients (theoretically even to the entire internet)
** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)
** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer)
* Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
* Old or no longer supported files will be removed automatically
* Can be implemented on the [[#Build_host_requirements|build host]]

== Puppet requirements ==
* moved to [[stoney_orchestra:_Requirements]], included below for reference.

{{:stoney orchestra: Requirements}}

== Install host requirements ==
* Ability to install physical and virtual machines
* Distinguish machines by their Ethernet MAC address
* Provide a PXE/TFTP boot mechanism
* Partition and format the (virtual) harddisks
* Install a stage3 image which was built by the build host
* Bootstrap puppet, enabling it to take over the individual installation and customization.
* Group hosts into
** environments (development, staging and production)
** architectures (such as x86, amd64 etc.)
** portage profiles (system profiles such as desktop and server)
** <s>stacks (comprising a complete product as a service with the underlying infrastructure)</s> this is the task of Puppet --[[Benutzer:Chaf|Chaf]] ([[Benutzer Diskussion:Chaf|Diskussion]]) 09:42, 19. Dez. 2013 (CET)

== Public key infrastructure requirements ==
* Local certificate authority for signing [http://en.wikipedia.org/wiki/X509 X.509] certificates.
* Master certificate authority root certificate which is only used to sign Sub-CA certificates
* Sub certificate authorities used for various cases such as
** Puppet certificates [http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html]
** User certificates
** Client certificates
** Host certificates
* Ability to sign, revoke and extend certificates
* Publish certificate revocation status either via [http://en.wikipedia.org/wiki/Certificate_revocation_list CRL] and/or [http://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol OCSP]
** CRL is not worth the hassle due to it not defining how often the CRL must be consulted. Since we are in the same physical net OCSP should be far superior here (thank to its live checking support). On the other hand puppet does not do OCSP yet (redmine: [http://projects.puppetlabs.com/issues/10111 #110111]) so we might need to implement both or implement OCSP as well as develop our own automated revocation for puppet.
* Choose DNs below <code>dc=rabe,dc=ch</code>
* register a PEN-OID as issued by IANA if custom schema work is required
** Use a @rabe email when requesting a PEN at [http://pen.iana.org/pen/PenApplication.page IANA], last time the @purplehaze.ch was a problem!
* Some of the aforementioned sub-CAs might be implemented as robot CAs with a self service interface (ie for authorized users).
* Consider using [http://en.wikipedia.org/wiki/Certificate_Management_Protocol CMP] or [http://en.wikipedia.org/wiki/Certificate_Management_over_CMS CMC] as an API to signing, revoking et. al.
** Since the underlying RFCs of both these protocols are rather new they are not yet broadly supported.
* Keep local root CA offline!
** Maybe use an old netbook as root CA :P
* Support GPG keys for signing packages

== Git hosting requirements ==
* Public repositories hosted on [http://www.github.com GitHub] (mainly) under the [https://github.com/organizations/radiorabe radiorabe organization] (almost anything which doesn't leak sensitive informations)
* Private repositories hosted on the internal infrastructure
** Accessible via https and a web interface
** contains some repos with uber-private data the gets compartmentalized even further (ie. hiera datafiles in different repos)
* One repository per component
* Daily backup of all repositories
* Branches for development, staging and production
** New features are added to the development branch only and later merged up to staging and production
* Must support pull-requests so we can implement a review process (when pulling through the envs)
** Sing-Offing might also be required
* Adhere to [http://semver.org/ Semantic Versioning] for version/release tags.
** Tag releases as <code>vX.Y.Z</code> those will be automatically appear on GitHub as downloadable tarballs, which can be referenced within the corresponding ebuilds.
** Hit 1.0.0 as soon as code lands on production or earlier
** Commit .lock files when reaching 1.0.0 where applicable (Gemfile.lock, composer.lock) or earlier if needed
* Must be able to trigger remote events (ie. update master through mcollective after code was promoted to production in a PR)
* Support the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model

== Messaging requirements ==
* I'm talking AMPQ, JMS, STOMP, 0MQ and the likes
** not sure if we need something in this space for the infra
** it could facilitate comms between components
** stuff like mcollective and RadioDNS need something in this space

== Monitoring, logging and alarming system requirements ==
@TODO
* centralized logging is used throughout
** with tools that help find and fix problems and do post mortems
* all systems are always monitored by a full monitoring suite
* the monitoring suite must support alarming users through multiple paths
** alarming should include a fallback strategy and a way to acknowledge alarms
** it must have a easy way to configure scheduled maintenance either before or while the maintenance is undergoing
* monitoring, logging and alarming are all automatically configured during regular provisioning of machines
* alerting uses jabber by default with fallbacks to email and sms-through-gsm depending on the site.

= Implementation proposal =
== Build host proposal ==
The build host consists out of various chroots to build binary packages for multiple environments, architectures and build profiles.

=== Links ===
* [http://packer.io packer.io] can be used to build stage4 (containing a kernel) images and seems to work for gentoo. Packer often gets used to build Vagrant boxes.
** [https://github.com/pierreozoux/packer-warehouse/blob/master/var-files/gentoo/generate_latest.sh gentoo script from packer-warehouse] used with packer to create a minimal gentoo vagrant box

== Portage tree clone proposal ==
== Portage overlay proposal ==
== Portage profile proposal ==
== Package and file mirror proposal ==
== Puppet proposal ==
* Adhere to Craig Dunns [http://www.craigdunn.org/2012/05/239/ architecture] [http://www.slideshare.net/PuppetLabs/roles-talk]
** on the system level (ie for each bar-metal or virtual machine)
*** roles contains the business view (ie. [https://github.com/radiorabe/puppet/blob/master/role/manifests/puppet/master.pp role::puppet::master])
*** profiles the implementation (such as [https://github.com/radiorabe/puppet/blob/master/profile/manifests/puppet/master.pp profile::puppet::master])
** on the architecture level (ie. in the cloud-fabric)
*** roles contains the business view (ie. role::cloud-storage, role::product1)
*** profiles contain the implementation (ie profile::storage-cluster, profile::storage-webinterface-farm)
* Keep profiles, roles (as per craig) and Puppetfile in [https://github.com/radiorabe/puppet github.com/radiorabe/puppet]
** This is where we keep feature/*, develop and master (ie staging) branches
** An internal clone then contains all these + production (what exactly is in prodution, ie. our release schedule is considered sensitive in this implementation)
** This lets us use the [http://nvie.com/posts/a-successful-git-branching-model/ git-flow] branching model with almost no changes (the one change being us gating stuff into production on the closed clone)
** github may use hooks to push content to our internal git when they happen
* All other modules need their own repo and must be published to the puppet module forge
* Use librarian-puppet (or r10k) for composing the final puppet envs
** r10k eschews git submodule support we used in puppet-syslogng but has support for multiple envs out of the box
** librarian-puppet would need to be run once per environment to achieve what r10k does
* provide develop, master and production branches from private repo as puppet environments on master

== Install host proposal ==
* use the existing server on [[tftp-01]] on the RaBe infra as a shortcut
** replace that instance with one native to the infra when it is ready for that
* iPXE [http://ipxe.org/]

== Public key infrastructure proposal ==
* write [[certificate policy]] (in german!)
* hold a key ceremony for the root and level 1
** offline ceremony on an old netbook with centos or similar (not debian, probably not gentoo to make this happen soonish)
** Sign RaBe root cert and level 1 intermediate cert
** store root cert key on 2 sdcards and as 1 printout somewhere safely
** store level 1 intermediate key on sdcards for use by admins
* use level 1 intermediate key to sign level 2 cas as needed
** level 2 robot ca key for puppet (managed by <code>puppet ca</code>)
** level 2 ca for client certs
** level 2 ca for host certs
** more level 2 certs
* use OpenSSL as default software for PKI
** ssl has the largest userbase which should make it easier on new admins
** features that openssl does not implement get used as soon as openssl catches up (ie. [http://cmpforopenssl.sourceforge.net/‎ CMP])

== git hosting proposal ==

* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood

= Links =
* [http://wiki.gentoo.org/wiki/Binary_package_guide Gentoo Binary Package Guide]
* [http://wiki.gentoo.org/wiki/Preserve-libs Gentoo preserve-libs]
* [http://swift.siphos.be/aglara/ A Gentoo Linux Advanced Reference Architecture]
* [http://www.gentoo.org/proj/en/gentoo-alt/prefix/ Gentoo Prefix]
* man pages
** [http://dev.gentoo.org/~zmedico/portage/doc/man/portage.5.html portage(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/emerge.1.html emerge(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/make.conf.5.html make.conf(5)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html ebuild(1)]
** [http://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.5.html ebuild(5)]

[[Category: Infrastructure]]

Gentoo Infrastructure

2014-01-09T18:30:41Z

Lucas: /* Build host proposal */ link and 'splain packer.io

stoney orchestra: Roadmap

2014-01-06T19:34:31Z

Lucas:

== backlog ==

=== unsorted ===
These might be rather epic and need breaking down until they are sensible user-stories.

* investigate, document and implement hooks and git-scripts needed for continuous-(integration|development|.*)
* evaluate and decide between puppet-librarian, r10k or byoc solution for applying Puppetfile/Modulefile
* develop and establish "frontend" tooling like git-* scripts or puppet-rake tasks for devs and admins
* mcollective

== links ==

* Architecture
** [[Gentoo_Infrastructure#Puppet_proposal]]
** [http://www.craigdunn.org/2012/05/239/ Craig Dunn's Blog: Designing Puppet – Roles and Profiles]
* Puppet Modules
** [http://forge.puppetlabs.com/gentoo/portage gentoo/portage puppet module]
** [http://forge.puppetlabs.com/purplehazech/syslogng syslogng module]
** [http://forge.puppetlabs.com/purplehazech/ccache ccache module] ;)

[[Category:stoney orchestra]][[Category:Roadmap]]

stoney orchestra: Roadmap

2014-01-06T19:33:15Z

Lucas:

== backlog ==

=== unsorted ===
These might be rather epic and need breaking down until they are sensible user-stories.

* investigate, document and implement hooks and git-scripts needed for continuous-(integration|development|.*)
* evaluate and decide between puppet-librarian, r10k or byoc solution for applying Puppetfile/Modulefile
* develop and establish "frontend" tooling like git-* scripts or puppet-rake tasks for devs and admins

== links ==

* Architecture
** [[Gentoo_Infrastructure#Puppet_proposal]]
** [http://www.craigdunn.org/2012/05/239/ Craig Dunn's Blog: Designing Puppet – Roles and Profiles]
* Puppet Modules
** [http://forge.puppetlabs.com/gentoo/portage gentoo/portage puppet module]
** [http://forge.puppetlabs.com/purplehazech/syslogng syslogng module]
** [http://forge.puppetlabs.com/purplehazech/ccache ccache module] ;)

[[Category:stoney orchestra]][[Category:Roadmap]]

stoney orchestra: Roadmap

2014-01-06T19:27:27Z

Lucas:

== backlog ==

=== unsorted ===
These might be rather epic and need breaking down until they are sensible user-stories.

* investigate, document and implement hooks and git-scripts needed for continuous-(integration|development|.*)
* evaluate and decide between puppet-librarian, r10k or byoc solution for using Puppetfile/Modulefile
* develop and establish "frontend" tooling like git-* scripts or puppet-rake tasks for devs and admins

== links ==

* Architecture
** [[Gentoo_Infrastructure#Puppet_proposal]]
** [http://www.craigdunn.org/2012/05/239/ Craig Dunn's Blog: Designing Puppet – Roles and Profiles]
* Puppet Modules
** [http://forge.puppetlabs.com/gentoo/portage gentoo/portage puppet module]
** [http://forge.puppetlabs.com/purplehazech/syslogng syslogng module]
** [http://forge.puppetlabs.com/purplehazech/ccache ccache module] ;)

[[Category:stoney orchestra]][[Category:Roadmap]]

stoney orchestra: Roadmap

2014-01-06T19:15:18Z

Lucas:

== Links ==
* Architecture
** [http://www.craigdunn.org/2012/05/239/ Craig Dunn's Blog: Designing Puppet – Roles and Profiles]
* Puppet Modules
** [http://forge.puppetlabs.com/gentoo/portage gentoo/portage puppet module]
** [http://forge.puppetlabs.com/purplehazech/syslogng syslogng module]
** [http://forge.puppetlabs.com/purplehazech/ccache ccache module] ;)

[[Category:stoney orchestra]][[Category:Roadmap]]

stoney orchestra: Roadmap

2014-01-06T19:13:12Z

Lucas:

== Links ==
* [http://forge.puppetlabs.com/gentoo/portage gentoo/portage puppet module]
* [http://www.craigdunn.org/2012/05/239/ Craig Dunn's Blog: Designing Puppet – Roles and Profiles]

[[Category:stoney orchestra]][[Category:Roadmap]]

Gentoo Infrastructure

2014-01-05T21:37:32Z

Lucas: /* Puppet requirements */

stoney orchestra: Requirements

2014-01-05T21:37:02Z

Lucas:

<noinclude>
== Overview ==
== Requirements ==
</noinclude>
* Support for all three environments (development, staging and production)
* Version controlled via Git
* ENC support
* Puppet recipes for
** installing, updating, removing and (re-)configuring specific software belonging to an application stack (see [[#Build_host_requirements|build host]]).
** (re-)configuring software belonging to a system stack
** Updating the system stack (<code>emerge @system</code>) aka system update.
** installing, updating and removing of kernel packages (including the handling of the ensuing reboot)
* use best-of-breed tools like hiera and augeas (this might mean targeting 3.3.x due to module data support in [https://github.com/puppetlabs/armatures/blob/master/arm-9.data_in_modules/index.md ARM-9])
* Use a sane prexisting puppet architecture concept
<noinclude>
[[Category:stoney orchestra]]
[[Category:Requirements]]
</noinclude>

stoney orchestra: Requirements

2014-01-05T21:34:49Z

Lucas: Created page with "== Overview == == Requirements == * Support for all three environments (development, staging and production) * Version controlled via Git * ENC support * Puppet recipes for *..."

== Overview ==
== Requirements ==
* Support for all three environments (development, staging and production)
* Version controlled via Git
* ENC support
* Puppet recipes for
** installing, updating, removing and (re-)configuring specific software belonging to an application stack (see [[#Build_host_requirements|build host]]).
** (re-)configuring software belonging to a system stack
** Updating the system stack (<code>emerge @system</code>) aka system update.
** installing, updating and removing of kernel packages (including the handling of the ensuing reboot)
* use best-of-breed tools like hiera and augeas (this might mean targeting 3.3.x due to module data support in [https://github.com/puppetlabs/armatures/blob/master/arm-9.data_in_modules/index.md ARM-9])
* Use a sane prexisting puppet architecture concept

[[Category:stoney orchestra]]
[[Category:Requirements]]

User:Lucas

2014-01-05T21:31:13Z

Lucas: Created page with "Itsa Mee :)"

Itsa Mee :)

Hiera Example

2014-01-05T21:29:51Z

Lucas: explain a bit what this is

Talk:stoney web: Requirements

2013-12-01T23:18:00Z

Lucas:

== DB Quota? ==

I'm assuming the DB-Quota here is due to the fact that you are also planning on using the MySQL DB for Zabbix data. Otherwise it would seem to be overkill as I can't see stoney web using that much space on mysql. 00:18, 2 December 2013 (CET)

Talk:stoney web: Requirements

2013-12-01T21:46:39Z

Lucas: Created page with "== DB Quota? == I'm assuming the DB-Quota here is due to the fact that you are also planning on using the MySQL DB for Zabbix data. Otherwise it would seem to be overkill as ..."