* Serving files via HTTPS
** Binary packages for all the clients (<code>PORTAGE_BINHOST</code>), which were built by the [[#Build_host_requirements|build host]]
*** Binary packages will be accessible via a HTTP URL such as <code>https://packages.example.com/gentoo/ENVIRONMENT/ARCH/BUILD-PROFILE-NAME/latest</code>. This is a symlink to the current snapshot <code>https://packages.example.com/gentoo/ENVIRONMENT/ARCH/BUILD-PROFILE-NAME/YYYY-MM-DD</code>.*** Clients will have <code>PORTAGE_BINHOST="https://packages.example.com/ENVIRONMENT/gentoo/ARCH/SYSTEMENVIRONMENT/BUILD-PROFILE-NAME https://packages.example.com/ENVIRONMENT/gentoo/ENVIRONMENT/ARCH/APP-STACK-PROFILE-NAME ..."</code> set in their <code>/etc/portage/make.conf</code>.
* Support for all three environments (development, staging and production)
* Possibility to authenticate clients either via HTTP basic auth or client certificates.
== File mirror host requirements ==
* Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>)** Acts as a caching mirror for already downloaded packages from an official mirror** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients* Files are served via HTTPS* Distinguishes between three groups of files** '''public''': Files which are available to all clients (theoretically even to the entire internet)** '''site-local''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public)** '''stack-local''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer) * Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.* Possibility to authenticate clients either via HTTP basic auth or client certificates.* Old or no longer supported files will be removed automatically* Can be implemented on the see [[Mirror Server#Build_host_requirements|build hostRequirements]]
== Puppet requirements ==
= Implementation proposal =
== Build host farm proposal ==The build host farm consists out of various chroots a system of multiple vms to build binary packages for multiple environments, architectures and build profiles. * Git webhook on internal gitlab install pushes changes to jenkins master.* Jenkins master dishes out jobs to jenkins slave machines for needed architecture and build profile.* Jenkins slaves only get used once and wipe/reprovision themselves after master has stored build artefacts.* We have build-slave templates available for each architecture/build profile combo.* Upon use those get provisioned to the needed environment using puppet.* All of this is set up using puppet and fully automated, even building of new build-slave templates and the whole releng on those.* The build farm also keeps old templates and stable boxes on hold so it can use them to build differentials.* Artefacts slaves will be producing:** "vagrant"-style boot boxes** full binpkg repos for a given env/arch/build profile combo** stage3 balls for each arch/build profile** stage4 balls for each environment** build logs** <code>/var/db/pkg</code>** puppet report data** test results and code analysis results* When we come to continuos deployment the jenkins master will also be able to trigger puppet when merges to master happen.* This rolls out releases to the sub-system that was signed off by a merge to a master branch (see branching strategy in git proposal).
=== Links ===
==== build orchestration ====
* [http://mesos.apache.org/ Apache Mesos] cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. Can run for instance Jenkins.
==== package building ====
== git hosting proposal ==
* adhere to git-flow for all the things. Automate said usage as far as possible.
{|- class="wikitable"
! colspan=4 | git-flow branching
|-
! Branch
! Environment
! Merge from
! Description
|-
| <code>master</code>
| production
| <code>release/</code> or <code>hotfix/</code>
| Released code with a <code>git tag</code> for each merge.
|-
| <code>release/v0.0.0</code>
| staging
| <code>develop</code>
| Contains final releasing work like updating versioning and changelog. This is where we keep semver concerns in check if they where not taken care of already.
|-
| <code>hotfix/v0.0.0</code>
| staging
| <code>master</code>
| Only for critically urgent fixes. In most cases doing a release from <code>develop</code> is preferred.
|-
| <code>develop</code>
| development
| <code>feature/</code> or <code>master</code>
| Only feature branches that are ready for production should get merged here. <code>master</code> gets merged here after each merge to it. Merging is done with pull requests and review.
|-
| <code>feature/featurename</code>
| development
| <code>develop</code>
| New features get implemented here until they are considered ready for production and merged to <code>develop</code>.
|-
| <code>support/v0.0.0</code>
| LTS
|
| Marked experimental in most implementations and unused for now.
|}
* Install gitlab on a vm and integrate external mirrors from github and ldap users from stoney-ldap.
** keep repo of public mirrors in hieradata so we can configure them from puppet.
** each organisation in stoney-ldap automatically gets a private project in gitlab.
* Configure web hook intrastructure and integrate with continuous integration system.
* Make continuous integration show feedback back in gitlab.
** check for <code>git annotate</code> support or use img badges.
'''On organization projects in gitlab'''
* Each project comes with default repos.
{| class="wikitable"
! Repo
! Description
|-
| <code>puppet</code>
| Set up using a template, contains a Puppetfile and Puppetfile.lock and a hieradata directory.
|-
| <code>role</code>
| Read only copy of global role module for reference.
|-
| <code>profile</code>
| Read only copy of global profile module for reference.
|}
* Everything in the latter two modules is configurable through hieradata in the first repo.
* The default setup automatically updates <code>role</code> and <code>profile</code> when they get new merges.
* A software agent (ci) regularly clones <code>develop</code>, does a full build and pushes the results back to <code>feature/tinderbox</code>
* This agent autmatically creates pull requests if tinderbox builds did not fail.
* Org leaders may then merge these PRs and bake them into a local release.
* Some kind of UI helps them do this without much technical knowledge.
* More repos may be added by the customer.
* project organizations are private, per customer.
== Links ==
* [http://gitlab.org/ gitlab] seems nice even though is is ruby on rails under the hood
* [https://github.com/sag47/gitlab-mirrors gitlab-mirrors] is a companion app to gitlab for adding readonly mirror repos to gitlab. We might consider hacking it to not use <code>git remote prune</code>.
* [http://www.javacodegeeks.com/2014/01/git-flow-with-jenkins-and-gitlab.html git-flow with jenkins and gitlab]
* [https://wiki.jenkins-ci.org/display/JENKINS/Gitlab+Hook+Plugin gitlab hook for jenkins]
= Links =