Aegir

two thirds-way through a big refactor. todo; move things from here to other places.

Aegir is a Drupal distribution for managing Drupal site provision and hosting.

Overview

Aegir is a Drupal Distro that uses the Hostmaster front-end to control the Provision module with its Drush integration. Aegir uses Drupal multi-site features to share one Drupal codebase install as a Platform which multiple Sites can be hosted on.

If you want to create a site for a particular Distro, you install the Distro as a Platform then create a Site on that Platform. Sites and their content can be cloned on or migrated between Platforms or pushed to spoke servers. Module dependency is managed via those loaded in the Profile of the Distro with the .make.

Creating a Distro requires a Profile and Make files.

• Wikipedia:Aegir Hosting System
• Aegir file system structure
• https://wiki.koumbit.net/FormationAegir/CheatSheet

• Drupal 7 and Aegir server setup

#aegir on freenode

Deployment

See BOA - 'Barracuda, Octopus, Aegir' bash install script.

• Aegir-up provides a collaborative, distributed development environment, that encourages the use of Drupal best-practices. It deploys the Aegir Hosting System in a local virtual environment for development and testing. It provides all the ease-of-use of Aegir, wrapped in the convenience of Vagrant-based virtual machines. Creating sites and platforms is a matter of a click or two, while rebuilding the entire environment is a matter of minutes.
  • http://drupalcode.org/project/aegir-up.git/blob/refs/heads/7.x-2.x:/README.md

• My Aegir Development Environment - ergonlogic
• Vimeo: Drupal Camp Montreal 2012 - Fearless development with Drush, Vagrant and Aegir

• http://drupal.org/project/puppet-aegir
  • https://github.com/Vertice/puppet-aegir
• http://drupal.org/project/puppet-drush
• https://redmine.koumbit.net/projects/puppet-apt
• https://labs.riseup.net/code/projects/shared-common

Tasks

Creating

Platform

Use a build make file to call the main make (possibly on-top of a base profile).

Base .build file that Git pulls the base development distro .make and .profile (and .info for Profiler lib).

Behind the scenes pseudo;

drush make --working-copy "$PROJECT_MAKE" "$PLATFORM_PATH"
drush provision-save @platform_PlatformName --root="$PLATFORM_PATH" --context_type=platform
drush @hostmaster hosting-import @platform_PlatformName

from aegir;

/usr/local/bin/php /var/aegir/drush/drush.php --php='/usr/local/bin/php'  --context_type='platform' --master_url='http://aegirmaster.example.org/' --root='/var/aegir/platforms/exampleplatform' --server='@server_master' --web_server='@server_master' --makefile='https://raw.github.com/so/me/build.make' --make_working_copy='1' provision-save '@platform_ExamplePlatformDec02' --backend

Site

Things to remember for a site build in hostmaster;

• Don't forget full domain name!
• Create dev.* alias for BOA cache-busting

Behind the scenes;

drush provision-save "@$SITE_DOMAIN" --uri="$SITE_DOMAIN" --context_type='site' --platform="@platform_PlatformName" --profile="$PROJECT_NAME"
drush "@$SITE_DOMAIN" provision-install
drush @hostmaster hosting-task @platform_"$PLATFORM_NAME" verify --force --verbose
drush user-login

In Hostmaster, make sure that all sites have an associated Client, otherwise the site will be listed as available for all Clients/Users you have added.

Importing

• http://community.aegirproject.org/content/importing-complete-drupal-platform
• http://community.aegirproject.org/content/importing-existing-aegir-backups

• remote_import - Drush extension that allows you to fetch remote sites from remote Aegir servers
  • hosting_remote_import - module that provides a UI for remote_import

Cloning

• http://community.aegirproject.org/node/24

Aegir provides an easy method of making entire copies of a site. This includes the actual site files, modules and so on, as well as a copy of the actual database. This feature is called 'Clone' in Aegir, because it is a method of duplicating a site with a new URL or 'site name'.

The feature is very closely linked to the Migrate feature because it is almost the same, except that rather than move the site, it leaves the existing site in place and just copies it to a new name. For this reason, enabling the Clone feature also enables the Migrate feature.

Migrating and updating

• Migrating/upgrading and renaming websites

• Migration Tips
• The best recipes for disaster and how to avoid them
• One last question re: packages

• Migrate sites between installation profiles

In .info;

“old_short_name = new_profile_name”

"Example: your site has been created with that old Commerce Dev platform when it was using ‘commercedev’ as a machine name for its installation profile, and you want to migrate it to newer Commerce 7.x Kickstart platform, but it is using ‘commerce_kickstart’ as a machine name for its installation profile. You only need to add one line in your commerce_kickstart.info file, with “old_short_name = commercedev” (without quotes), just below “name = Commerce Kickstart” line. Then re-verify the newer Commerce Kickstart platform in Aegir and Voilà! – now you can migrate your sites from the old Commerce Dev platform to the new Commerce Kickstart platform"

---

Aegir does not manage profiles..

the following is incomplete!!

Basic core and module upgrade;

• Git commit theme and push to remote
• Use profiler_builder to create new distro files (info/profile/make)
• Edit files (todo/missing info)
• Add new set of profile files to git repo

build a new platform with the upgrades, then migrate sites to the new platform. Clone and migrate to test before.

Be careful about creating additions or changes in sites/*/modules, sites/*/themes, etc. Migrate tasks will fail due to path issues.

"No, really, don't use sites/domain-name/modules for anything and save yourself headaches and frustration."

...

OK, so what is the correct workflow for sites upgrades in Aegir? It highly depends on how you manage your code, but some general rules are always valid and we will list them below:

1. Create or choose new platform.
2. Upload all your contrib modules and themes to the new target platform.
3. Re-verify the target platform in Aegir.
4. Clone your live site with some working subdomain in the old platform.
5. Re-verify old platform and also just cloned site.
6. Migrate cloned site to the new platform.
7. Check if the cloned site works without any issues.
8. If the step 7 above works, you can safely migrate also the live site.

• Migrate sites from one Aegir to another (Apache)

Troubleshooting

If a site install task fails, and the build make/info/profile cannot be fixed and the Install and Verify run again to completion, there is no clean way to remove it.

Instead;

• Delete failed site install task item
• Delete failed site item
• Remove failed site table from DB

• Deleting sites

Provision

• Provision - the Aegir backend
• API reference
• purge option for platforms (--force)
• Manage your Aegir system from the command-line

• Aegir contexts - above and beyond Drush site aliases

Troubleshooting

• How to fix a stuck hostmaster task

/usr/bin/php /var/aegir/drush/drush.php @hostmaster hosting-dispatch
/usr/bin/php /var/aegir/drush/drush.php -y @hostmaster vset hosting_queue_tasks_frequency 1

/usr/bin/php /var/aegir/drush/drush.php @hostmaster hosting-task 1234 --verbose --debug

Run hostmaster tasks with debug

/usr/bin/php /var/aegir/drush/drush.php @hostmaster hosting-cron --verbose --debug

Restore backup;

drush @mysite.com provision-restore /var/aegir/backup/example.com...tgz

Older restore method(?)

drush @hostmaster provision-deploy /var/aegir/backup/example.com...tgz

Pause task cron;

drush @hostmaster hostmaster-pause

See also fimafeng, my older bash provision script for Aegir/Drush.

Remote

To sort with Drush#deployment

Proper master hub remote spoke Aegir requires remote Aegir user, Apache config changes and apache2ctl sudoers for restarting, so shared hosting environments are generally out of the question currently.

Saying that, one can manually create Drush site aliases with the relevant details to manually rsync and sql-sync.

• Remote servers (multiserver) - basics for setting up a remote (slave) Aegir server

• Essential Reading: Aegir Master-Remote Mysteries
• Running Drush Rsync on Windows (Solved!) - dreamhost

• Configuring Aegir Hosting System to use a Remote MySQL server - have hub hostmaster use remote db via drush.rc config

• Aegir Support for Multi-Server Site Deployment and Management for Drupal - Adrian Rossouw, Jul 06 2010

• Further questions on multi-server & development workflow

• Drush synchronization of database and files - Apr 16, 2012. Drush extra.

Export

• http://community.aegirproject.org/discuss/how-export-site-aegir
  • https://gist.github.com/2401607

Aegir contrib

• http://community.aegirproject.org/contrib-modules

• http://community.aegirproject.org/node/75

Access

• https://github.com/computerminds/aegir_http_basic

• http://drupal.org/project/provisionacl - Drush module, granular shell access for users

Managing workflow

• http://drupal.org/project/hosting_queue_runner - tasks are run as soon as possible instead of waiting for a cron run. This makes Aegir appear much more responsive. [1] - now in Aegir core

• https://github.com/computerminds/aegir_make_working_copy - provides git working copy option un hostmaster gui for dev sites. - broken [2]

• http://drupal.org/project/hosting_site_git - hostmaster+provision
• http://drupal.org/project/provision_git - backend only

• https://drupal.org/project/hosting_platform_git
• https://drupal.org/project/provision_platform_git

• http://drupal.org/project/kplatforms
  • http://drupalcode.org/project/kplatforms.git/tree/refs/heads/7.x-2.x

• Hosting Features - provides a Task to Hostmaster that provides integration with a site's Features. It is currently in development but works for the most important task: Update & Commit Features

• aegir-projec - A project-abstraktion on top of platforms and sites

DevShop

• http://drupal.org/project/devshop

• http://drupal.org/project/devshop_provision
• http://drupal.org/project/devshop_hosting

Backups

• http://drupal.org/project/provision_reinstall
  • http://drupal.org/project/hosting_reinstall

• http://drupal.org/project/hosting_backup_gc - allows you to specify how long Aegir should retain backups for

Remote import

Migration between Aegirs

• Remote import - Provision
• Remote import - Hostmaster - Drush extension

Remote Import goes on receiving server, SSH key from server site is coming from

• http://drupalcode.org/project/hosting_remote_import.git/blob/refs/heads/6.x-1.x:/README.txt

Docker

• https://github.com/DominicBoettger/docker-aegir-mariadb

Services

• https://aegir.koumbit.net/
• http://www.koumbit.org/en/services/AegirVPS
  • http://www.koumbit.org/en/articles/aegirvps-new-drupal-hosting-solution

• http://omega8.cc

BOA

Barracuda Octopus Aegir

• https://github.com/omega8cc/boa - meta-installer for an easy Barracuda + Octopus + Aegir install and upgrade on the high performance LEMP stack.

• All libraries & tools required to install and run Nginx based Aegir system.
• Latest release of MariaDB 5.5 or 10.0 database server with Chive manager.
• Latest version of Nginx web server.
• PHP-FPM 5.6, 5.5, 5.4, 5.3 - multi-install mode, configurable per Octopus.
• PHP extensions: Zend OPcache, PHPRedis, UploadProgress, MailParse and ionCube.
• Fast Redis Cache with DB auto-failover for all 6.x and 7.x platforms.
• Fast Redis Lock support with DB auto-failover for all 6.x and 7.x platforms.
• Fast proxy DNS server (pdnsd) with permanent caching.
• Limited Shell, SFTP and FTPS separate accounts per Octopus instance.
• Limited Shell, SFTP and FTPS accounts per Aegir Client with per site access.
• Drush access on command line in all shell accounts.
• Drush Make access on command line for main shell account only.
• Support for New Relic monitoring with per Octopus instance license key.
• Solr 4 cores can be added/updated/deleted via site level INI settings.
• HTTPS access with self-signed certificate for all hosted sites.
• Magic Speed Booster cache, working like a Boost + AuthCache, but per user.
• Entry level XSS built-in protection on the Nginx level.
• Firewall csf/lfd integrated with Nginx abuse guard.
• PHP errors debugging, including WSOD, enabled on the fly on dev. aliases.
• Boost, AdvAgg, Domain Access and Drupal for Facebook built-in support.
• Built-in collection of useful modules available in all platforms.
• Autonomous Maintenance & Auto-Healing scripts in /var/xdrago.
• Every 10 seconds uptime/self-healing local monitoring.
• Automated, rotated daily backups for all databases in /data/disk/arch/sql.

Optional:

• Compass Tools.
• SPDY Nginx support.
• PFS (Perfect Forward Secrecy) support in Nginx.
• HHVM support - see docs/HHVM.txt for details.
• MultiCore Apache Solr 1.4.1 with Jetty 7 - see docs/SOLR.txt for details.
• MultiCore Apache Solr 3.6.2 with Jetty 8 - see docs/SOLR.txt for details.
• MultiCore Apache Solr 4.2.0 with Jetty 8 or Jetty 9 on Precise and Wheezy.
• New Relic Apps Monitor with per Octopus license and per Site reporting.
• Image Optimize toolkit binaries.
• FFmpeg support.
• Bind9 DNS server.
• Webmin Control Panel.
• SQL Buddy database manager.
• Collectd server monitor.
• LDAP Nginx support via third-party module (experimental).
• MongoDB driver for PHP 5.3 (experimental).
• GEOS extension for PHP 5.3 (experimental).

Documentation =

• http://bit.ly/boa-docs

• README
• INSTALL

Aegir master install goes in /var/aegir. Not as tuned and featured as the Octopus Aegir install. Best practice is to use Octopus for production.

• Updates
• CHANGELOG
• UPGRADE

• REMOTE
• SSL
• SOLR

• BOND.sh - tuning script

• How to disable all caching and aggregation? - Using "dev.*" subdomain in site alias disabled hardcoded performance. Also for PHP errors and logging.

• Are there any specific good habits to learn?
• Your Drupal Site Upgrade -- Safe Workflow

• The best recipes for disaster
• What are best practices for BOA file cleanup?

• https://github.com/omega8cc/boa/blob/1c04aced13507cf7364166b690dad09afe2da919/CHANGELOG.txt#L2907 - Drush restrictions

• Are bug fixes/patches applied to the current BOA stable release?

Configuration

.barracuda.conf xtra shortcodes [3]

PDS --- fast DNS cache server (pdnsd) (default)
BND --- Bind9 DNS Server
SLR --- MultiCore Apache Solr Tomcat
CHV --- Chive DB Manager (default)
BDD --- SQL Buddy DB Manager
CGP --- Collectd Graph Panel
WMN --- Webmin Control Panel
CSF --- csf/lfd Firewall (default)
FTP --- ??? (default)

Nginx configuration layout; /etc/nginx/nginx.conf

user www-data;
worker_processes  4;  
pid /var/run/nginx.pid;

events {
  multi_accept on; 
}

http {
  default_type application/octet-stream;
  gzip on; 
  gzip_disable "msie6";
  keepalive_timeout 65; 
  sendfile on; 
  tcp_nodelay on; 
  tcp_nopush on; 
  types_hash_max_size 8192;
  include /etc/nginx/mime.types;
  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

• /etc/nginx/nginx.conf
  • /etc/nginx/conf.d/aegir.conf -> /var/aegir/config/nginx.conf -> /var/aegir/config/server_master/nginx.conf barracuda specific
    • /var/aegir/config/server_master/nginx/pre.d/*;
      • extra_ip.conf
      • extra_ip_ssl.conf
      • nginx_speed_purge.conf
      • nginx_wild_ssl.conf
    • /var/aegir/config/server_master/nginx/platform.d/*;
      • /data/disk/o1/config/o1.nginx.conf - octopus specific
        • /data/disk/o1/config/server_master/nginx/vhost.d/*; - o1 hosted Drupal sites
    • /var/aegir/config/server_master/nginx/vhost.d/*; - master and admin tool sites
      • chive.master.purple.thingsandstuff.org
      • master.purple.thingsandstuff.org
    • /var/aegir/config/server_master/nginx/post.d/*; - empty

Install and upgrade

xdrago cron runs tool helper scripts

• usage.sh

See also HINTS.txt

Do not use apt to upgrade. Upgrading got easier with BOA 2. See UPGRADE.txt for full details.

# Download and run BOA Meta Installer.
wget -q -U iCab http://files.aegir.cc/BOA.sh.txt
bash BOA.sh.txt

Will honour settings in;
/root/.barracuda.cnf
/root/.o1.octopus.cnf
/root/.o2.octopus.cnf

# To upgrade system and Aegir Master Instance to Stable
barracuda up-stable

To upgrade system and Aegir Master Instance to HEAD
barracuda up-head

#To upgrade selected Aegir Satellite Instance to Stable
octopus up-stable o1

# To upgrade *all* Aegir Satellite Instances to Stable
octopus up-stable all

# To upgrade selected Aegir Satellite Instance to HEAD
octopus up-head o1

# To upgrade *all* Aegir Satellite Instances to HEAD
octopus up-head all

See also Security#csf.2Flfd

• https://github.com/omega8cc/boa/issues/500 - Lets Encrypt (free SSL)

Caching

• Redis is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

New system uses only Redis cache and the same configuration for all Drupal 6 and Drupal 7 platforms. This new system doesn't require any extra module to be enabled in any site. Complete integration is already enabled by default for every platform/site installed by default and for every custom platform as before - the next day after first site on the custom platform has been created. You can disable this caching layer using the same modules/cache/NO.txt control file as before. While there is just one cache engine (Redis) used, there is also an automatic, instant failover to standard database caching, just in case Redis is not available for some reason. You can also disable Redis cache on the fly for debugging by adding ?noredis=1 to any URL.