Difference between revisions of "Build Server"

From stoney cloud
Jump to: navigation, search
[unchecked revision][unchecked revision]
(→‎Overview)
 
(15 intermediate revisions by 2 users not shown)
Line 3: Line 3:
 
* Packages, which don't need to be built, for example [http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html Oracle's Java SE Development Kit] (JDK), can be fetched from the [[Mirror Server]].
 
* Packages, which don't need to be built, for example [http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html Oracle's Java SE Development Kit] (JDK), can be fetched from the [[Mirror Server]].
 
* Software stacks built upon specialized binary packages, require their own chroot environment. If possible, avoid this.
 
* Software stacks built upon specialized binary packages, require their own chroot environment. If possible, avoid this.
 +
* For reproducibility, Portage and Profiles are frozen with the help of version control system (git in our case). This is only done on the [[Mirror Server]].
 +
* When building packages in a chroot environment, the portage and overlay versions need to set.
  
 +
== Portage Snapshotting ==
 +
 +
To be able to reliably rebuild packages, we are working with a snapshotted/versioned portage tree.
 +
 +
=== Updating the snapshot to latest ===
 +
 +
We are using <code>emerge-webrsync</code> since it features an already determined version, checksum- and gpg-validation.
 +
 +
{{Cmd
 +
|git clone "${PORTAGE_GIT_REPO}" portage
 +
|cd portage
 +
|PORTAGE_RSYNC_EXTRA_OPTS{{=}}"--exclude{{=}}/.git --exclude{{=}}/.gitignore" PORTDIR{{=}}"${PWD}" emerge-webrsync
 +
|output=<pre>
 +
Fetching most recent snapshot ...
 +
Trying to retrieve 20140821 snapshot from http://mirror.switch.ch/mirror/gentoo ...
 +
Fetching file portage-20140821.tar.xz.md5sum ...
 +
Fetching file portage-20140821.tar.xz.gpgsig ...
 +
Fetching file portage-20140821.tar.xz ...
 +
Checking digest ...
 +
Getting snapshot timestamp ...
 +
Syncing local tree ...
 +
 +
</pre>}}
 +
 +
{{Warning|If you get the following instead then there is no newer snapshot available}}
 +
 +
{{Cmd
 +
|PORTAGE_RSYNC_EXTRA_OPTS{{=}}"--exclude{{=}}/.git --exclude{{=}}/.gitignore" PORTDIR{{=}}"${PWD}" emerge-webrsync
 +
|output=<pre>
 +
Fetching most recent snapshot ...
 +
* Latest snapshot date: 20140821
 +
*
 +
* Approximate snapshot timestamp: 1408668300
 +
*        Current local timestamp: 1408667701
 +
*
 +
* The current local timestamp is possibly identical to the
 +
* timestamp of the latest snapshot. In order to force sync, use
 +
* the --revert option or remove the timestamp file located at
 +
* '/run/user/1000/portage/metadata/timestamp.x'.
 +
</pre>}}
 +
 +
Set the tag and push if there was an update:
 +
{{Cmd
 +
|cd portage/
 +
|git add -A
 +
|snapshot_date{{=}}$(date --date{{=}}@$(( $(cut -f 1 -d " " metadata/timestamp.x) - 86400 )) +%Y-%m-%d)
 +
|git commit -m "Update portage tree to ${snapshot_date}"
 +
|git tag -a -m "Tagging ${snapshot_date}" ${snapshot_date}
 +
|git push && git push --tags
 +
|echo "snapshot date: ${snapshot_date}"
 +
}}
 +
 +
The subtraction of 86400 (1 day in seconds) above is to account for the fact that the snapshot for a given day is taken at 00:45 UTC on the following day and the timestamp contained in the tarball therefore is one day ahead (see {{Path|/usr/bin/emerge-webrsync}}).
 +
 +
The meaning of the tag is therefore: contains at least all changes from that day (and possibly a bit more).
 +
 +
== Chroot Setup ==
 +
 +
=== Preparation ===
 +
 +
At least the following is required beforehand to ensure reproducible building:
 +
 +
* a versioned portage tree
 +
* versioned overlays (if any)
 +
 +
=== Setup ===
 +
 +
* Create a staging root directory and extract a stage3 or stage4 tarball
 +
* Mount proc, sys and dev (required for portage to control the tty, determine available storage space, etc.)
 +
* Copy {{Path|/etc/resolv.conf}}
 +
* Checkout the versioned portage tree (since you probably won't have git yet in the tree)
 +
* Enter the chroot
 +
 +
{{RootCmd
 +
|stagingRoot{{=}}/var/staging/staging-base
 +
|mkdir -p "${stagingRoot}"
 +
|tar -xjpf /var/tmp/your-stage4.tbz2 -C "${stagingRoot}"
 +
|mount -t proc none "${stagingRoot}/proc"
 +
|mount --rbind /dev/ "${stagingRoot}/dev/"
 +
|mount --rbind /sys/ "${stagingRoot}/sys/"
 +
|cp -af /etc/resolv.conf "${stagingRoot}/etc/"
 +
|git clone "${PORTAGE_GIT_REPO}" "${stagingRoot}/usr/portage"
 +
|chmod -R a+rX "${stagingRoot}/usr/portage"
 +
|chroot "${stagingRoot}" /usr/bin/env -i HOME{{=}}"/root" TERM{{=}}"${TERM}" /bin/bash --login
 +
|}}
 +
 +
{{Note|We are deliberately using a special command for entering the chroot to avoid leaking environment variables}}
 +
 +
* Install {{Package|app-portage/layman}} and probably {{Package|dev-vcs/git}} to install overlays
 +
* You may want to add a mirror configuration if you have local mirrors:
 +
{{File|/etc/portage/make.conf||<source lang='bash'>
 +
GENTOO_MIRRORS="https://mirror.example.com/public/gentoo"
 +
</source>}}
 +
 +
* If needed, put the mirror for fetch restricted ebuilds in a local mirror definition:
 +
{{File|/etc/portage/mirrors||<pre>
 +
local https://mirror.example.com/restricted/gentoo
 +
</pre>}}
 +
 +
* Add the following to generate binary packages during installation:
 +
{{File|/etc/portage/make.conf||<source lang='bash'>
 +
FEATURES="${FEATURES} buildpkg cgroup ipc-sandbox network-sandbox parallel-fetch"
 +
</source>}}
 +
 +
* Make sure that the repository configuration points to your git mirror as well:
 +
{{File|/etc/portage/repos.conf/gentoo.conf||<source lang='ini'>
 +
[DEFAULT]
 +
main-repo = gentoo
 +
 +
[gentoo]
 +
location = /usr/portage
 +
sync-type = git
 +
sync-uri = https://mirror.example.com/versioned/portage.git
 +
</source>}}
 +
 +
* Rebuild everything:
 +
{{RootCmd|emerge -e world}}
  
 
[[Category:Infrastructure]]
 
[[Category:Infrastructure]]

Latest revision as of 08:51, 1 June 2015

Overview

  • Base chroot environment for the creation of most binary packages which will be copied to the Binary Package Server.
  • Packages, which don't need to be built, for example Oracle's Java SE Development Kit (JDK), can be fetched from the Mirror Server.
  • Software stacks built upon specialized binary packages, require their own chroot environment. If possible, avoid this.
  • For reproducibility, Portage and Profiles are frozen with the help of version control system (git in our case). This is only done on the Mirror Server.
  • When building packages in a chroot environment, the portage and overlay versions need to set.

Portage Snapshotting

To be able to reliably rebuild packages, we are working with a snapshotted/versioned portage tree.

Updating the snapshot to latest

We are using emerge-webrsync since it features an already determined version, checksum- and gpg-validation.

user $ git clone "${PORTAGE_GIT_REPO}" portage
user $ cd portage
user $ PORTAGE_RSYNC_EXTRA_OPTS="--exclude=/.git --exclude=/.gitignore" PORTDIR="${PWD}" emerge-webrsync
Fetching most recent snapshot ...
Trying to retrieve 20140821 snapshot from http://mirror.switch.ch/mirror/gentoo ...
Fetching file portage-20140821.tar.xz.md5sum ...
Fetching file portage-20140821.tar.xz.gpgsig ...
Fetching file portage-20140821.tar.xz ...
Checking digest ...
Getting snapshot timestamp ...
Syncing local tree ...
Warning
If you get the following instead then there is no newer snapshot available
user $ PORTAGE_RSYNC_EXTRA_OPTS="--exclude=/.git --exclude=/.gitignore" PORTDIR="${PWD}" emerge-webrsync 
Fetching most recent snapshot ...
 * Latest snapshot date: 20140821
 * 
 * Approximate snapshot timestamp: 1408668300
 *        Current local timestamp: 1408667701
 * 
 * The current local timestamp is possibly identical to the
 * timestamp of the latest snapshot. In order to force sync, use
 * the --revert option or remove the timestamp file located at
 * '/run/user/1000/portage/metadata/timestamp.x'.

Set the tag and push if there was an update:

user $ cd portage/
user $ git add -A
user $ snapshot_date=$(date --date=@$(( $(cut -f 1 -d " " metadata/timestamp.x) - 86400 )) +%Y-%m-%d)
user $ git commit -m "Update portage tree to ${snapshot_date}"
user $ git tag -a -m "Tagging ${snapshot_date}" ${snapshot_date}
user $ git push && git push --tags
user $ echo "snapshot date: ${snapshot_date}"

The subtraction of 86400 (1 day in seconds) above is to account for the fact that the snapshot for a given day is taken at 00:45 UTC on the following day and the timestamp contained in the tarball therefore is one day ahead (see /usr/bin/emerge-webrsync).

The meaning of the tag is therefore: contains at least all changes from that day (and possibly a bit more).

Chroot Setup

Preparation

At least the following is required beforehand to ensure reproducible building:

  • a versioned portage tree
  • versioned overlays (if any)

Setup

  • Create a staging root directory and extract a stage3 or stage4 tarball
  • Mount proc, sys and dev (required for portage to control the tty, determine available storage space, etc.)
  • Copy /etc/resolv.conf
  • Checkout the versioned portage tree (since you probably won't have git yet in the tree)
  • Enter the chroot
root # stagingRoot=/var/staging/staging-base
root # mkdir -p "${stagingRoot}"
root # tar -xjpf /var/tmp/your-stage4.tbz2 -C "${stagingRoot}"
root # mount -t proc none "${stagingRoot}/proc"
root # mount --rbind /dev/ "${stagingRoot}/dev/"
root # mount --rbind /sys/ "${stagingRoot}/sys/"
root # cp -af /etc/resolv.conf "${stagingRoot}/etc/"
root # git clone "${PORTAGE_GIT_REPO}" "${stagingRoot}/usr/portage"
root # chmod -R a+rX "${stagingRoot}/usr/portage"
root # chroot "${stagingRoot}" /usr/bin/env -i HOME="/root" TERM="${TERM}" /bin/bash --login
Note
We are deliberately using a special command for entering the chroot to avoid leaking environment variables
File/etc/portage/make.conf

GENTOO_MIRRORS="https://mirror.example.com/public/gentoo"
  • If needed, put the mirror for fetch restricted ebuilds in a local mirror definition:
File/etc/portage/mirrors

local https://mirror.example.com/restricted/gentoo
  • Add the following to generate binary packages during installation:
File/etc/portage/make.conf

FEATURES="${FEATURES} buildpkg cgroup ipc-sandbox network-sandbox parallel-fetch"
  • Make sure that the repository configuration points to your git mirror as well:
File/etc/portage/repos.conf/gentoo.conf

[DEFAULT]
main-repo = gentoo
 
[gentoo]
location = /usr/portage
sync-type = git
sync-uri = https://mirror.example.com/versioned/portage.git
  • Rebuild everything:
root # emerge -e world
Retrieved from "https://wiki.stoney-cloud.org/mediawiki/index.php?title=Build_Server&oldid=4607"