Difference between revisions of "Modularisation"

From stoney cloud
Jump to: navigation, search
[unchecked revision][checked revision]
(→‎Directory Structure Web Module)
(→‎schema)
 
(40 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
== Overview ==
 +
As the stoney cloud actually tries to act as a collection of all aspects of a high availability cloud infrastructure, we have roughly three main sections:
 +
* '''Infrastructure''': The basis of the whole ecosystem (Build Server, Binary Package Server, Mirror Server, Puppet Server, Operating System).
 +
* '''stoney cloud''': The actual cloud with an easy to use multi-tenant web based interface.
 +
* '''Self-Service Modules''': Modules that expand the functionality of the stoney cloud.
 +
To make the base installation of the stoney cloud as small and simple as possible, everything is as modularised as possible.
 +
 
== Requirements ==
 
== Requirements ==
−
Core
+
=== Infrastructure ===
−
* User, Customer, Reseller
+
The article [[Gentoo Infrastructure]] describes how use gentoo as an infrastructure backbone for creating a complete and modern IT architecture.
−
* Roles and rights
+
−
* Billing
+
−
* Forgot Password
+
−
* ...
+
  
−
Modules: Service/Product specific:
+
=== stoney cloud ===
−
* Online Backup (stoney safe)
+
The main framework called [[:Category:stoney core|stoney core]] is responsible for shared functionality.
−
* Mail Hosting (stoney mail)
+
* The basic installation and configuration of the initial stoney cloud components (OpenLDAP server, Apache web server, GlusterFS, ...).
−
* VM-Manager (stoney conductor)
+
* Reseller, customer and user management.
−
* VM-Manager light (stoney vm): Start, stop, access a VM.
+
* Roles and rights management.
 +
* Billing functionality.
 +
* Searching functionality.
 +
* A consistent look and feel between modules.
 +
* Internationalization.
  
−
GitHub:
+
This functionality is accessible via:
−
* Form follows function: we choose the functionality name (so we are backwards compatible)
+
* A [[:Category:REST_API|REST API]], which serves as the data and business logic abstraction layer and uses JSON as the primary data interchange format.
−
* Web Modules are collected in one repository (create a skeleton module as an example):
+
−
** Data (LDAP): load.ldif
+
−
** API (PHP): api.php
+
−
** Web Interface (HTML/JS/CSS, uses API): gui.js
+
−
** Tests (GUI, Unit-Tests, ...): test.xml
+
−
** CLI (Python, Perl, PHP, Bash with curl ... uses API): api.sh
+
  
−
* Service Modules
+
This [[:Category:REST_API|REST API]] is currenty accessible via:
−
** Provisioning (Perl, ...): prov.pl
+
* A '''c'''ommand '''l'''ine '''i'''nterface.
−
** Helper Scripts, like notifications (quota, backup failed, rsnapshot, ...): helper.pl
+
* An Ajax based web interface.
−
** Tests (CLI, Unit-Tests, ...): test.sh
+
* Any scripting language which supports a REST API and JSON.
  
−
== Module Naming ==
+
=== Self-Service Modules ===
−
The names of the modules are derived from the service functionality. They are different from the marketing names. These marketing names should be configurable per reseller.
+
The [[:Category:Self-Service Modules|Self-Service Modules]] expand the initial stoney cloud functionality. Some examples:
 +
* [[:Category:stoney conductor|stoney conductor]]: A cloud and virtual machine management tool (storage, network, cpu, memory, virtual machines, backups, snapshots, ...).
 +
* [[:Category:stoney vm|stoney vm]]: A simplified sub set of the [[:Category:stoney conductor|stoney conductor]] functionality (start, stop, access a virtual machine).
 +
* [[:Category:stoney backup|stoney backup]]: An on-line backup service for desktops, servers and virtual machines.
 +
* [[:Category:stoney mail|stoney mail]]: A mail service with optional collaboration functionality (based on Open-Xchange).
 +
* [[:Category:stoney monitor|stoney monitor]]: Monitoring (with Zabbix).
 +
* [[:Category:stoney orchestra|stoney orchestra]]: Configuration Management (with Puppet).
 +
* [[:Category:stoney box|stoney box]]: An on-line storage service (Webbrowser access via HTTPS, WebDAV via HTTPS and synchronisation to multiple devices).
 +
* [[:Category:stoney web|stoney web]]: Web & Database hosting service (based on Apache and MariaDB).
  
−
=== Selfcare Module ===
+
=== Self-Service Modules Naming ===
−
* selfcare-backup (on-line backup module, marketing name is stoney safe)
+
The names of the modules are derived from the service functionality. These can be different from the marketing names. These marketing names must configurable per reseller and per customer.
−
* selfcare-storage (on-line storage module, marketing name is stoney box)
+
−
* selfcare-web
+
−
* (selfcare-database) -> may be part of web-web
+
−
* selfcare-dns (currently API code only)
+
  
−
=== Service Modules ===
+
Naming Convention for Provisioning modules belonging to a Self-Service module:
−
* service-web-apache
+
* prov-<SERVICE>-<TYPE>
−
* service-database-mariadb
+
** prov-mail-ox
−
* service-database-mysql
+
** prov-backup-kvm
−
* service-database-postgresql
+
** prov-backup-rsnapshot
−
* service-dns-powerdns
+
** prov-monitoring-zabbix
−
* service-backup-kvm
+
** prov-configuration-management-puppet
−
* service-backup-rsnapshot
+
 
 +
=== Self-Service Modules Installation ===
 +
It would be nice to be able to install new modules without the need to modify the main configuration file. Maybe with single configuration files per module? Or do you have other suggestions?
 +
* Use [http://www.memcached.org/ memcached]?
 +
* Use a file with one include per module?
 +
 
 +
=== Self-Service Modules Splitting ===
 +
The [[:Category:Self-Service Modules|Self-Service Modules]] are split into different parts, based on their functionality:
 +
* '''api''': REST Application Programming Interface.
 +
* '''cli''': A command Line Interface, which uses the REST API to read or write data.
 +
* '''docs''': The self-service module documentation.
 +
* '''helpers''': Helper scripts and programmes, like notifications (quota, backup failed, rsnapshot, ...).
 +
* '''prov-<SERVICE>-<TYPE>''': Scripts and programmes to provision services.
 +
* '''web''': Ajax based web interface.
 +
 
 +
Where nothing else is defined, the following rules apply:
 +
* All source code is hosted on [https://github.com/stoney-cloud GitHub]. See the the [[#GitHub | GitHub]] chapter below for more information.
 +
* Version control via Git, according to [http://nvie.com/posts/a-successful-git-branching-model/ Git flow]. TBD.
 +
* Bug & Enhancement Tracking via GitHub (reproduction of the sprints on GitHub).
 +
* PHP Coding-Standards according to [http://symfony.com/doc/current/contributing/code/standards.html Symfony Coding Standards], otherwise fall-back to [http://www.php-fig.org/psr/psr-2/ PSR-2].
 +
* Unit Tests according [http://phpunit.de/ PHPUnit] (like to [http://symfony.com/doc/current/book/testing.html Symfony]).
 +
* Versions schema according [http://semver.org/ Semantic Versioning]. TBD
 +
* Programming languages:
 +
** '''api''': Symfony.
 +
** '''cli''': Bash or Perl.
 +
** '''helpers''': Bash or Perl. Other languages possible.
 +
** '''prov-<SERVICE>-<TYPE>''': Perl
 +
** '''web''': JavaScript Framework AngularJS (with HTML5).
  
 
== GitHub ==
 
== GitHub ==
−
Proposed directory structure on GitHub.
+
The [[Release Management]] page gives you a nice overview about Release Cycles, Versioning and Development Procedure. The chapter [[Release_Management#Source_Code_Organization |Source Code Organization]] describes nicely, where the code is located on [https://github.com/stoney-cloud/ GitHub].
  
−
=== Directory Structure Selfcare Module ===
+
The following directory structure is proposed:
 
<pre>
 
<pre>
−
/data             #  
+
/api             # REST Application Programming Interface.
−
/data/ldif        # LDAP: load.ldif
+
/cli            # A command Line Interface, which uses the REST API to read or write data.
−
/api              # API (PHP): api.php
+
/docs            # The self-service module documentation.
−
/api/...          # Structure from CWI according to best practice from the Yii-Framework (a separation of configuration and code would be nice).
+
/examples        # Example scripts for proof of concept and similar scripts.
−
/api/...          # It would be nice, if the configuration files could be outside the web root (htdocs), so that a miss-configuration of
+
/helpers        # Helper scripts and programmes, like notifications (quota, backup failed, rsnapshot, ...).
−
/api/...         # the web server does not lead to password leaks.
+
/provisioning    # Scripts and programmes to provision services.
−
/web              # Web Interface (HTML/JS/CSS, uses API): gui.js
+
/schema         # Description, validation specification and documentation of the REST API.
−
/web/...         #  
+
/web            # Ajax based web interface.
−
/cli              # CLI (Python, Perl, PHP, Bash with curl ... uses API): api.sh
+
/LICENSE        # The license the self-service module is released under.
−
/cli/...          #
+
/README.md      # A simple readme file containing an overview and how to install as the absolute minimum.
−
/test            # Tests (GUI, Unit-Tests, ...): test.xml
+
−
/test/...        #
+
 
</pre>
 
</pre>
  
−
It would be nice to be able to install new modules without the need to modify the main configuration file. Maybe with single configuration files per module? Or do you have other suggestions?
+
Every sub folder must contain (where applicable):
 +
* A simple readme file containing an overview and how to install as the absolute minimum.
 +
* Structure according to best practice of the programming language or framework (a separation of configuration and code would be nice).
 +
* A test directory (Unit-Tests, Test-Cases, Test-Data, ...).
 +
* Documentation.
 +
 
 +
=== helpers ===
 +
The following shows how a '''helpers''' directory structure could look like.
 +
<pre>
 +
/helpers                                            # Helper scripts and programmes, like notifications (quota, backup failed, rsnapshot, ...).
 +
/helpers/bin                                        # Binary / executable scripts and programmes.
 +
/helpers/etc                                        # Configuration files and directories.
 +
/helpers/data                                        # Data used for the helpers.
 +
/helpers/data/data.ldif                              # The actual data, in this case a ldif file.
 +
/helpers/test                                        # Tests (CLI, Unit-Tests, ...).
 +
/helpers/README.md                                  # A simple readme file containing an overview and how to install as the absolute minimum.
 +
</pre>
 +
 
 +
=== schema ===
 +
The following shows how a '''schema''' directory structure could look like.
 +
<pre>
 +
/schema                        # Description, validation specification and documentation of the REST API.
 +
/schema/core.schema.json      # Every module has a master file, which includes all json files (the name is derived from the Git Repo module name)
 +
/schema/base.schema.json      # Type definitions.
 +
/schema/reseller.schema.json  # Per resource on file.
 +
/schema/people.schema.json    #
 +
/schema/README.md              # Information how the schema file(s) can be used and how the documentation can be extracted.
 +
</pre>
  
−
=== Directory Structure Service Module ===
+
=== prov-<SERVICE>-<TYPE> ===
−
Example for service-backup-rsnapshot:
+
The following shows how a '''prov-<SERVICE>-<TYPE>''' directory structure could look like. The following example is done with '''prov-backup-rsnapshot'''.
 
<pre>
 
<pre>
−
/bin                                    # Helper Scripts, like notifications (quota, backup failed, rsnapshot, ...): helper.pl
+
/prov-backup-rsnapshot                                        # Scripts and programmes to provision services.
−
/etc                                    # Configuration
+
/prov-backup-rsnapshot/bin                                    # Binary / executable scripts and programmes.
−
/etc/Provsioning                        #
+
/prov-backup-rsnapshot/etc                                    # Configuration files and directories.
−
/etc/Provsioning/Backup                #
+
/prov-backup-rsnapshot/etc/Provsioning                        #
−
/etc/Provsioning/Backup                #
+
/prov-backup-rsnapshot/etc/Provsioning/Backup                #
−
/lib                                    # Provisioning Perl-Modules for a dedicated Service: KVM.pm or rsnapshot.pm
+
/prov-backup-rsnapshot/lib                                    # Provisioning Perl-Modules for a dedicated Service (for example KVM.pm or rsnapshot.pm)
−
/lib/perl                              #
+
/prov-backup-rsnapshot/lib/perl                              #
−
/lib/perl/Provisioning                  #
+
/prov-backup-rsnapshot/lib/perl/Provisioning                  #
−
/lib/perl/Provisioning/Backup          #
+
/prov-backup-rsnapshot/lib/perl/Provisioning/Backup          #
−
/lib/perl/Provisioning/Backup/Rsnapshot #  
+
/prov-backup-rsnapshot/lib/perl/Provisioning/Backup/Rsnapshot #  
−
/libexec                                #
+
/prov-backup-rsnapshot/libexec                                #
−
/test                                  # Tests (CLI, Unit-Tests, ...): test.sh
+
/prov-backup-rsnapshot/test                                  # Tests (CLI, Unit-Tests, ...): test.sh
 +
/prov-backup-rsnapshot/README.md                              # A simple readme file containing an overview and how to install as the absolute minimum.
 
</pre>
 
</pre>
  
−
[[Category:Development]]
+
[[Category:Development]][[Category:Documentation]]

Latest revision as of 16:24, 5 January 2015

Overview

As the stoney cloud actually tries to act as a collection of all aspects of a high availability cloud infrastructure, we have roughly three main sections:

  • Infrastructure: The basis of the whole ecosystem (Build Server, Binary Package Server, Mirror Server, Puppet Server, Operating System).
  • stoney cloud: The actual cloud with an easy to use multi-tenant web based interface.
  • Self-Service Modules: Modules that expand the functionality of the stoney cloud.

To make the base installation of the stoney cloud as small and simple as possible, everything is as modularised as possible.

Requirements

Infrastructure

The article Gentoo Infrastructure describes how use gentoo as an infrastructure backbone for creating a complete and modern IT architecture.

stoney cloud

The main framework called stoney core is responsible for shared functionality.

  • The basic installation and configuration of the initial stoney cloud components (OpenLDAP server, Apache web server, GlusterFS, ...).
  • Reseller, customer and user management.
  • Roles and rights management.
  • Billing functionality.
  • Searching functionality.
  • A consistent look and feel between modules.
  • Internationalization.

This functionality is accessible via:

  • A REST API, which serves as the data and business logic abstraction layer and uses JSON as the primary data interchange format.

This REST API is currenty accessible via:

  • A command line interface.
  • An Ajax based web interface.
  • Any scripting language which supports a REST API and JSON.

Self-Service Modules

The Self-Service Modules expand the initial stoney cloud functionality. Some examples:

  • stoney conductor: A cloud and virtual machine management tool (storage, network, cpu, memory, virtual machines, backups, snapshots, ...).
  • stoney vm: A simplified sub set of the stoney conductor functionality (start, stop, access a virtual machine).
  • stoney backup: An on-line backup service for desktops, servers and virtual machines.
  • stoney mail: A mail service with optional collaboration functionality (based on Open-Xchange).
  • stoney monitor: Monitoring (with Zabbix).
  • stoney orchestra: Configuration Management (with Puppet).
  • stoney box: An on-line storage service (Webbrowser access via HTTPS, WebDAV via HTTPS and synchronisation to multiple devices).
  • stoney web: Web & Database hosting service (based on Apache and MariaDB).

Self-Service Modules Naming

The names of the modules are derived from the service functionality. These can be different from the marketing names. These marketing names must configurable per reseller and per customer.

Naming Convention for Provisioning modules belonging to a Self-Service module:

  • prov-<SERVICE>-<TYPE>
    • prov-mail-ox
    • prov-backup-kvm
    • prov-backup-rsnapshot
    • prov-monitoring-zabbix
    • prov-configuration-management-puppet

Self-Service Modules Installation

It would be nice to be able to install new modules without the need to modify the main configuration file. Maybe with single configuration files per module? Or do you have other suggestions?

  • Use memcached?
  • Use a file with one include per module?

Self-Service Modules Splitting

The Self-Service Modules are split into different parts, based on their functionality:

  • api: REST Application Programming Interface.
  • cli: A command Line Interface, which uses the REST API to read or write data.
  • docs: The self-service module documentation.
  • helpers: Helper scripts and programmes, like notifications (quota, backup failed, rsnapshot, ...).
  • prov-<SERVICE>-<TYPE>: Scripts and programmes to provision services.
  • web: Ajax based web interface.

Where nothing else is defined, the following rules apply:

  • All source code is hosted on GitHub. See the the GitHub chapter below for more information.
  • Version control via Git, according to Git flow. TBD.
  • Bug & Enhancement Tracking via GitHub (reproduction of the sprints on GitHub).
  • PHP Coding-Standards according to Symfony Coding Standards, otherwise fall-back to PSR-2.
  • Unit Tests according PHPUnit (like to Symfony).
  • Versions schema according Semantic Versioning. TBD
  • Programming languages:
    • api: Symfony.
    • cli: Bash or Perl.
    • helpers: Bash or Perl. Other languages possible.
    • prov-<SERVICE>-<TYPE>: Perl
    • web: JavaScript Framework AngularJS (with HTML5).

GitHub

The Release Management page gives you a nice overview about Release Cycles, Versioning and Development Procedure. The chapter Source Code Organization describes nicely, where the code is located on GitHub.

The following directory structure is proposed: 

/api             # REST Application Programming Interface.
/cli             # A command Line Interface, which uses the REST API to read or write data.
/docs            # The self-service module documentation.
/examples        # Example scripts for proof of concept and similar scripts.
/helpers         # Helper scripts and programmes, like notifications (quota, backup failed, rsnapshot, ...).
/provisioning    # Scripts and programmes to provision services.
/schema          # Description, validation specification and documentation of the REST API.
/web             # Ajax based web interface.
/LICENSE         # The license the self-service module is released under.
/README.md       # A simple readme file containing an overview and how to install as the absolute minimum.

Every sub folder must contain (where applicable):

  • A simple readme file containing an overview and how to install as the absolute minimum.
  • Structure according to best practice of the programming language or framework (a separation of configuration and code would be nice).
  • A test directory (Unit-Tests, Test-Cases, Test-Data, ...).
  • Documentation.

helpers

The following shows how a helpers directory structure could look like. 

/helpers                                             # Helper scripts and programmes, like notifications (quota, backup failed, rsnapshot, ...).
/helpers/bin                                         # Binary / executable scripts and programmes.
/helpers/etc                                         # Configuration files and directories.
/helpers/data                                        # Data used for the helpers.
/helpers/data/data.ldif                              # The actual data, in this case a ldif file.
/helpers/test                                        # Tests (CLI, Unit-Tests, ...).
/helpers/README.md                                   # A simple readme file containing an overview and how to install as the absolute minimum.

schema

The following shows how a schema directory structure could look like. 

/schema                        # Description, validation specification and documentation of the REST API.
/schema/core.schema.json       # Every module has a master file, which includes all json files (the name is derived from the Git Repo module name)
/schema/base.schema.json       # Type definitions.
/schema/reseller.schema.json   # Per resource on file.
/schema/people.schema.json     # 
/schema/README.md              # Information how the schema file(s) can be used and how the documentation can be extracted.

prov-<SERVICE>-<TYPE>

The following shows how a prov-<SERVICE>-<TYPE> directory structure could look like. The following example is done with prov-backup-rsnapshot. 

/prov-backup-rsnapshot                                        # Scripts and programmes to provision services.
/prov-backup-rsnapshot/bin                                    # Binary / executable scripts and programmes.
/prov-backup-rsnapshot/etc                                    # Configuration files and directories.
/prov-backup-rsnapshot/etc/Provsioning                        #
/prov-backup-rsnapshot/etc/Provsioning/Backup                 #
/prov-backup-rsnapshot/lib                                    # Provisioning Perl-Modules for a dedicated Service (for example KVM.pm or rsnapshot.pm)
/prov-backup-rsnapshot/lib/perl                               #
/prov-backup-rsnapshot/lib/perl/Provisioning                  #
/prov-backup-rsnapshot/lib/perl/Provisioning/Backup           #
/prov-backup-rsnapshot/lib/perl/Provisioning/Backup/Rsnapshot # 
/prov-backup-rsnapshot/libexec                                #
/prov-backup-rsnapshot/test                                   # Tests (CLI, Unit-Tests, ...): test.sh
/prov-backup-rsnapshot/README.md                              # A simple readme file containing an overview and how to install as the absolute minimum.
Retrieved from "https://wiki.stoney-cloud.org/mediawiki/index.php?title=Modularisation&oldid=4456"