Stack

From Things and Stuff Wiki
Revision as of 17:40, 25 August 2014 by Milk (talk | contribs) (→‎Extra)
Jump to navigation Jump to search


See also Server, Box, Distros, E-mail#Server, Platforms, BOA

to sort out big time

General

Hardware

Open Compute

Clustering

Virtualisation

Hardware level

QEMU

KVM

  • KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko. KVM also requires a modified QEMU although work is underway to get the required changes upstream. Using KVM, one can run multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualized hardware: a network card, disk, graphics adapter, etc. 
grep -E "(vmx|svm)" --color=always /proc/cpuinfo

If nothing is displayed after running that command, then your processor does not support hardware virtualization, and you will not be able to use KVM.

Xen

  • Xen Hypervisor is an open source virtualization platform that powers the world's largest clouds in production and is the foundation of many commercial products. Xen powers public clouds such as Amazon Web Services, Rackspace Public Cloud and many others. Examples of Xen based server products include Huawei UVP, Oracle VM and XenServer. Examples of client products and appliances include QubesOS, XenClient and Netscaler. Xen is 9 years old, mature and its stability and versatility is second to none.

VirtualBox

VirtualBox config;

  • Enable host Linux modules: vboxnetflt and vboxnetadp
  • Enable host-only adapter 'virtualbox0' in VB Preferences > Network if not previously enabled, then select in guest preferences
  • Keep VM names lowercase and no spaces for easier command-line stuff

Install guest additions for mouse, etc.

Provisioning VirtualBoxes for local dev work;

Vargrant
  • Vagrant uses Oracle’s VirtualBox to build configurable, lightweight, and portable virtual machines dynamically. The first couple pages serve to introduce you to Vagrant and what it has to offer while the rest of the guide is a technical walkthrough for building a fully functional web development environment. The getting started guide concludes by explaining how to package the newly created vagrant environment so other developers can get up and running in just a couple commands.
vagrant package --vagrantfile Vagrantfile.pkg --include README.txt
  stored in ~/.vagrant.d/boxes
vagrant box add lucid32 http://files.vagrantup.com/lucid32.box
vagrant box remove lucid32
vagrant box list
  • Veewee is a tool for easily (and repeatedly) building custom Vagrant base boxes, KVMs, and virtual machine images.
  • PuPHPet - A simple GUI to set up virtual machines for PHP Web development.

Other

  • bhyve - the "BSD hypervisor" is a legacy-free hypervisor/virtual machine manager developed on FreeBSD and relies on modern CPU features such as Extended Page Tables (EPT) and VirtIO network and storage drivers. bhyve was officially released on January 20th, 2014 as part of FreeBSD 10.0.

Operating system level

  • Operating system-level virtualization is a server virtualization method where the kernel of an operating system allows for multiple isolated user-space instances, instead of just one. Such instances (often called containers, VEs, VPSs or jails) may look and feel like a real server, from the point of view of its owner. On Unix systems, this technology can be thought of as an advanced implementation of the standard chroot mechanism. In addition to isolation mechanisms, the kernel often provides resource management features to limit the impact of one container's activities on the other containers.
  • http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines

FreeBSD jail

LinuX Containers

  • lmctfy is the open source version of Google’s container stack, which provides Linux application containers.
  • warden - Manages isolated, ephemeral, and resource controlled environments.
  • wsh - execute command in a Linux Container through unix socket
  • Pipework lets you connect together containers in arbitrarily complex scenarios. Pipework works with "plain" LXC containers (created with lxc-start), and therefore, it also works with the awesome Docker.

Docker

Setup
sysctl net.ipv4.ip_forward=1
  enable network forwarding

net.ipv4.ip_forward=1
  persistent network forwarding, goes in /etc/sysctl.d/docker.conf

sudo <path to>/docker -d &
   start docker in daemon mode

sudo systemctl enable docker
  start docker as service

sudo systemctl start docker
  start on system boot

ls -lah /var/run/docker.sock
  exists when docker is running
Notes

Add user to docker group to avoid sudo.

Docker isn't screen/tmux friendly due to tty handling. Easy fix is to ssh in to the instance.

sshd probably doesn't start by default. tis in root path. 

/usr/sbin/sshd

If you are behind a proxy you can launch docker with this command 

sudo HTTP_PROXY=http://youruserproxy:yourpassword@theaddressproxy:theportproxy/ docker -d &
Commands
# Download an ubuntu image
docker pull [name]

docker run [OPTIONS] IMAGE[:TAG] [COMMAND] [ARG...]
docker run ubuntu uname -a

# Run an interactive shell in the ubuntu image,
# allocate a tty, attach stdin and stdout
# To detach the tty without exiting the shell,
# use the escape sequence Ctrl-p + Ctrl-q
docker run -i -t ubuntu /bin/bash

# Bind TCP port 8080 of the container to TCP port 80 on 127.0.0.1 of the host machine.
docker run -p 127.0.0.1:80:8080 <image> <cmd>

# Bind TCP port 8080 of the container to a dynamically allocated TCP port on 127.0.0.1 of the host machine.
docker run -p 127.0.0.1::8080 <image> <cmd>

# Bind TCP port 8080 of the container to TCP port 80 on all available interfaces of the host machine.
docker run -p 80:8080 <image> <cmd>

# Bind TCP port 8080 of the container to a dynamically allocated TCP port on all available interfaces of the host machine.
docker run -p 8080 <image> <cmd>

# List your containers
docker images

# Listing all running containers
docker ps

Running docker command returns container ID 

# run a process which echoes 'hello world' in every second
CONTAINER_ID=$(sudo docker run -d ubuntu /bin/sh -c "while true; do echo hello world; sleep 1; done")

# attach the console to the container
docker attach $CONTAINER_ID

# stop an active container
docker stop $CONTAINER_ID

# kill an active container
docker kill $CONTAINER_ID

# Commit your container to a new named image
docker commit <container_id> <some_name>
Guides
Extra
  • dockerlite lets you run Linux apps in lightweight, isolated environments, using LXC (Linux Containers). Using BTRFS snapshots, dockerlite can save the state of a given environment in a frozen "image", and later, create more environments ("containers") out of that image.
  • DockerUI is a web interface to interact with the Remote API. The goal is to provide a pure client side implementation so it is effortless to connect and manage docker.
  • boot2docker is a lightweight Linux distribution based on Tiny Core Linux made specifically to run Docker containers. It runs completely from RAM, weights ~38mb and boots in ~5-6s (YMMV).
  • Docker Desktop - Dockerfile that creates a docker image which, once executed, creates a container that runs X11 and SSH services. The ssh is used to forward X11 and provide you encrypted data communication between the docker container and your local machine. Xpra + Xephyr allows to display the applications running inside of the container such as Firefox, LibreOffice, xterm, etc. with recovery connection capabilities.
  • Autodock is a docker automation tool. It automatically sorts servers in your Docker cluster by lowest load. It then distributes the containers you want to create among them. After bootstrapping the containers with Paramiko and Salt it saves this information to the ETCD cluster. Autodock was designed to use saltstack and etcd for configuration management and replication. [8]
  • Fig - Fast, isolated development environments using Docker. [9]
  • Drone is a Continuous Integration platform built on Docker [10]

Linux-VServer

  • Linux-VServer provides virtualization for GNU/Linux systems. This is accomplished by kernel level isolation. It allows to run multiple virtual units at once. Those units are sufficiently isolated to guarantee the required security, but utilize available resources efficiently, as they run on the same kernel.

OpenVZ

  • OpenVZ is container-based virtualization for Linux. OpenVZ creates multiple secure, isolated Linux containers (otherwise known as VEs or VPSs) on a single physical server enabling better server utilization and ensuring that applications do not conflict. Each container performs and executes exactly like a stand-alone server; a container can be rebooted independently and have root access, users, IP addresses, memory, processes, files, applications, system libraries and configuration files.

CoreOS

  • CoreOS is Linux for the container world. Linux kernel + systemd. That's about it. CoreOS has just enough bits to run containers, but does not ship a package manager itself. In fact, the root partition is completely read-only, to guarantee consistency and make updates reliable. Use docker as a package manager to build and push your app. The primitive of an application is a container, not a one-off package. Build containers using docker, by hand, or however you see fit! [11]

Lguest

Solaris Containers / Zones

Desktop level

Virtual machines

  • Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.

Disk images

Bitnami

ZeroVM

Orchestration and management

See also Hosting#PaaS

OpenStack

  • Nova is the project name for OpenStack Compute, a cloud computing fabric controller, the main part of an IaaS system. Individuals and organizations can use Nova to host and manage their own cloud computing systems. Nova originated as a project out of NASA Ames Research Laboratory.
  • Swift is a highly available, distributed, eventually consistent object/blob store. Organizations can use Swift to store lots of data efficiently, safely, and cheaply.
  • Glance provides services for discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image.
  • Cinder provides an infrastructure for managing volumes in OpenStack. It was originally a Nova component called nova-volume, but has become an independent project since the Folsom release.
  • Neutron (was Quantum) provides "networking as a service" between interface devices (e.g., vNICs) managed by other Openstack services (e.g., nova).
  • Keystone is the identity service used by OpenStack for authentication (authN) and high-level authorization (authZ). It currently supports token-based authN and user-service authorization. It has recently been rearchitected to allow for expansion to support proxying external services and AuthN/AuthZ mechanisms such as oAuth, SAML and openID in future versions.

OpenNebula

  • OpenNebula.org is an open-source project developing the industry standard solution for building and managing virtualized enterprise data centers and enterprise private clouds.
    • OpenNebula is an open-source cloud computing toolkit for managing heterogeneous distributed data center infrastructures.

OpenShift

Cloud Foundry

Juju

  • Juju enables you to use Charms to deploy your application architectures to EC2, OpenStack, Azure your data center and even your own Ubuntu based laptop. Moving between environments is simple giving you the flexibility to switch hosts whenever you want — for free.

Landscape

Eucalyptus

Cocaine

  • Cocaine (Configurable Omnipotent Custom Applications Integrated Network Engine) is an open-source PaaS (platform as a service) system for creating custom cloud hosting apps that are similar to Google App Engine or Heroku. Any library or service can be implemented as a service in Cocaine using a special API. Several indispensable services have already been implemented this way, including a service for detecting a user's region or language, a service for accessing MongoDB storage, and a URL fetcher.

MaestroNG

  • MaestroNG - command-line utility that allows for automatically managing the orchestrated deployment and bring up of a set of service instance containers that compose an environment on a set of target host machines. Each host machine is expected to run a Docker daemon. Maestro will then contact the Docker daemon of each host in the environment to figure out the status of the environment and what actions to take based on the requested command.

Dokku

  • Dokku - Docker powered mini-Heroku. The smallest PaaS implementation you've ever seen.

Project Atomic

  • Project Atomic integrates the tools and patterns of container-based application and service deployment with trusted operating system platforms to deliver an end-to-end hosting architecture that's modern, reliable and secure. [14]

Deis

Phusion Passenger

  • Phusion Passenger - Web server and application server. Designed to be fast, robust and lightweight. Makes web app deployments a lot simpler and less complex, by managing your apps' processes and resources for you.

Consul

  • Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable.

GNU parallel

  • GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.

Skydock

  • Skydock monitors docker events when containers start, stop, die, kill, etc and inserts records into a dynamic DNS server skydns. This allows standard DNS queries for services running inside docker containers. Because lets face it, if you have to modify your application code to work with other service discovery solutions you might as well just give up. DNS just works and it works well.

Flynn

  • Flynn simplifies deploying and maintaining applications. Instead of using complex configuration management systems, Flynn allows self-serve management of containerized deployments, making life easier for ops and developers. [17] [18]

Gaudi

Terraform

Nixos

Service discovery

See also Network#Distributed

to merge into Configuration and management?

Zookeeper

  • ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

Doozer

  • Doozer is a highly-available, completely consistent store for small amounts of extremely important data. When the data changes, it can notify connected clients immediately (no polling), making it ideal for infrequently-updated data for which clients want real-time updates. Doozer is good for name service, database master elections, and configuration data shared between several machines. See When Should I Use It?, below, for details.

etcd

  • etcd - A highly-available key value store for shared configuration and service discovery. etcd is inspired by Apache ZooKeeper and doozer, with a focus on being: Simple: curl'able user facing API (HTTP+JSON), Secure: optional SSL client cert authentication, Fast: benchmarked 1000s of writes/s per instance, Reliable: properly distributed using Raft

SkyDNS

  • SkyDNS is a distributed service for announcement and discovery of services. It leverages Raft for high-availability and consensus, and utilizes DNS queries to discover available services. This is done by leveraging SRV records in DNS, with special meaning given to subdomains, priorities and weights.

Serf

  • Serf is a decentralized solution for service discovery and orchestration that is lightweight, highly available, and fault tolerant.

Used by Flynn.

Discoverd

  • Discoverd - A simple but powerful service discovery system written in Go. It's currently backed by etcd, but can be extended to use ZooKeeper or other distributed consistent stores.

Provision and configuration

See also *nix#Management

Puppet

apt-get install puppet puppetmaster facter rubygems puppet-module

Foreman

Boxen

Chef

Salt

  • Salt is a configuration management system, capable of maintaining remote nodes in defined states (for example, ensuring that specific packages are installed and specific services are running), a distributed remote execution system used to execute commands and query data on remote nodes, either individually or by arbitrary selection criteria

Foreman

  • The Foreman is a complete lifecycle management tool for physical and virtual servers. Through deep integration with configuration management, DHCP, DNS, TFTP, and PXE-based unattended installations, Foreman manages every stage of the lifecycle of your physical or virtual servers. The Foreman provides comprehensive, auditable interaction facilities including a web frontend and robust, RESTful API.

Fabric

Capistrano

  • Capistrano is a utility and framework for executing commands in parallel on multiple remote machines, via SSH. It uses a simple DSL (borrowed in part from Rake) that allows you to define tasks, which may be applied to machines in certain roles. It also supports tunneling connections via some gateway machine to allow operations to be performed behind VPN's and firewalls. Capistrano was originally designed to simplify and automate deployment of web applications to distributed environments, and originally came bundled with a set of tasks designed for deploying Rails applications.

CFEngine

Synapse

  • Synapse enables you to remotely manage a large number of hosts. It brings together features of Configuration Management and Orchestration in a lightweight framework. Written in Python and using AMQP for messaging between the nodes.

Archipel

Ubuntu Orchestra

Ansible

  • Sovereign – Ansible playbooks to build and maintain your own personal cloud [22]

Other

  • Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files.

Continuous process

"Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly."

Jenkins

StriderCD

  • Strider is an Open Source Continuous Deployment / Continuous Integration platform. It is written in Node.JS / JavaScript and uses MongoDB as a backing store. BSD license. A focus on Continuous Deployment rather than just Continuous Integration: Designed to be easy to install & setup. Deployable & usable on Heroku free plan. Intended for deployment on private infrastructure. An emphasis on extensibility. Plugins are powerful, easy to write and simple to install. Out-of-the-box support for projects written in Node.JS, Python (generic and Django/Pyramid) and Selenium/Sauce Labs tests. Commercial support, consulting & hosting available

Git based

Services


  • http://bytemark.co.uk/hosting/symbiosis Symbiosis comes as standard with all new Bytemark servers, so if you're a recent customer or have re-imaged your system recently, it's ready to go. You can easily host PHP, Perl/CGI, htaccess files and MySQL, including any custom modules and settings that you might need. You have root access, so you don't need to ask permission to host new domains, different PHP settings, or create a database. You can even host any other Linux technology such as Java, Rails or Erlang. you can host any number of mailboxes under a domain, including forwarders, IMAP/POP3 boxes and autoresponders. There is a simple local anti-spam service based on Spamassassin, but if you need managed filtering you can quickly elect to pass your email through our anti-spam service.

Testing

to sort

  • Centmin Mod is for installation on CentOS only and written by George Liu (eva2000) with the addition of a shell menu based installer (shown on the right). Centmin Mod shell based menu allows you to do basic Nginx & PHP related management including upgrading or downgrading Nginx & PHP or setting up Nginx vhosts.


  • Ajenti includes dozens of pre-made plugins that let you configure both your OS and server software. List of supported software includes Apache, BIND9, Cron, CTDB, DHCPD, NFSD, Iptables, Munin, MySQL, Netatalk, NGINX, PostgreSQL, Samba, lm-sensors, Squid 3, Supervisor.
  • Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark, and other applications on a dynamically shared pool of nodes.
  • https://github.com/mesosphere/marathon Mesos framework for long running services

Tasksel

other tasks

  • firewall ports
  • services and daemons..
Retrieved from "https://wiki.thingsandstuff.org/index.php?title=Stack&oldid=25896"