Difference between revisions of "Mirror Server"
From stoney cloud
| [unchecked revision] | [unchecked revision] |
(→Fetch restricting) |
|||
| (7 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| + | == Requirements == | ||
| + | |||
| + | * Hosts all the files required to build a package (<code>GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles</code>) | ||
| + | ** Acts as a caching mirror for already downloaded packages from an official mirror, example caching mirror URL https://mirror.example.com/mirror | ||
| + | ** Serves fetch-restricted files (<code>dev-java/oracle-jdk-bin</code> for example), to authorized clients | ||
| + | * Files are served via HTTPS | ||
| + | * Distinguishes between three groups of files | ||
| + | ** '''public''': Files which are available to all clients (theoretically even to the entire internet, added to clients via GENTOO_MIRRORS) | ||
| + | ** '''site-restricted''': Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into [http://www.bettercallsaul.com/ legal troubles] if available to the public, used by clients via a ''local'' entry in <code>/etc/portage/mirrors</code>) | ||
| + | *** A subdirectory named after a site will be created below <code>site-local</code>, for example <code>site-local/example-site</code>. | ||
| + | ** '''role-restricted''': Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer, used by clients via a ''local'' entry in <code>/etc/portage/mirrors</code>) | ||
| + | *** A subdirectory per role will be created below <code>role-restricted</code>, for example <code>role-restricted/mediawiki_mysql</code>. | ||
| + | * Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism. | ||
| + | * Possibility to authenticate clients either via HTTP basic auth or client certificates. | ||
| + | * Old or no longer supported files will be removed automatically | ||
| + | * Can be implemented on the [[#Build_host_requirements|build host]] | ||
| + | |||
== Directory structure == | == Directory structure == | ||
| Line 4: | Line 21: | ||
<pre> | <pre> | ||
| − | + | ├── public | |
| − | + | │ ├── gentoo | |
| − | + | │ │ └── distfiles | |
| − | + | │ └── stepping-stone.ch | |
| − | + | │ └── sepiola | |
| − | + | ├── role-restricted | |
| − | + | │ ├── crm_sugarcrm_mysql | |
| − | + | │ └── wiki_mediawiki_mysql | |
| − | + | └── site-restricted | |
| − | + | └── stepping-stone.ch | |
| − | + | └── gentoo | |
| − | + | └── distfiles | |
</pre> | </pre> | ||
| + | |||
| + | == Packaging notes == | ||
| + | |||
| + | === Fetch restriction === | ||
| + | |||
| + | To avoid passive information disclosure all packages requiring source files from at least one of the restricted locations must be fetch restricted (if possible). Otherwise a client may in some circumstances try to fetch the package from a public mirror, leaving traces in the respective access log. | ||
| + | |||
| + | For Gentoo ebuilds this means: | ||
| + | |||
| + | <source lang='bash'> | ||
| + | RESTRICT="fetch" | ||
| + | </source> | ||
[[Category:Infrastructure]] | [[Category:Infrastructure]] | ||
Latest revision as of 19:28, 22 August 2014
Requirements
- Hosts all the files required to build a package (
GENTOO_MIRRORS=mirror.example.com/public/gentoo/distfiles)- Acts as a caching mirror for already downloaded packages from an official mirror, example caching mirror URL https://mirror.example.com/mirror
- Serves fetch-restricted files (
dev-java/oracle-jdk-binfor example), to authorized clients
- Files are served via HTTPS
- Distinguishes between three groups of files
- public: Files which are available to all clients (theoretically even to the entire internet, added to clients via GENTOO_MIRRORS)
- site-restricted: Files which are only available to authenticated clients belonging to the same infrastructure (for example those which would put us into legal troubles if available to the public, used by clients via a local entry in
/etc/portage/mirrors)- A subdirectory named after a site will be created below
site-local, for examplesite-local/example-site.
- A subdirectory named after a site will be created below
- role-restricted: Files which are only available to authenticated clients belonging to the same infrastructure and the software stack group (private files of a specific customer, used by clients via a local entry in
/etc/portage/mirrors)- A subdirectory per role will be created below
role-restricted, for examplerole-restricted/mediawiki_mysql.
- A subdirectory per role will be created below
- Provides an easy way to let an administrator manually upload new files, for example via WebDAV-CGI, SFTP or a similar mechanism.
- Possibility to authenticate clients either via HTTP basic auth or client certificates.
- Old or no longer supported files will be removed automatically
- Can be implemented on the build host
Directory structure
The visible structure starting from the root directory will be as follows:
├── public
│ ├── gentoo
│ │ └── distfiles
│ └── stepping-stone.ch
│ └── sepiola
├── role-restricted
│ ├── crm_sugarcrm_mysql
│ └── wiki_mediawiki_mysql
└── site-restricted
└── stepping-stone.ch
└── gentoo
└── distfiles
Packaging notes
Fetch restriction
To avoid passive information disclosure all packages requiring source files from at least one of the restricted locations must be fetch restricted (if possible). Otherwise a client may in some circumstances try to fetch the package from a public mirror, leaving traces in the respective access log.
For Gentoo ebuilds this means:
RESTRICT="fetch"
