Stack

From Things and Stuff Wiki
Revision as of 15:42, 29 March 2018 by Milk (talk | contribs) (→‎To sort)
Jump to navigation Jump to search


See also Server, Box, Distros, E-mail#Server, Platforms, BOA

to sort out big time

General


  • YouTube: Container Orchestration Wars (2017) - Karl Isenberg explores the features of several container orchestrators—breaking down the feature sets and characteristics into categories, and scoring multiple solutions against each other while comparing them to other cloud platform layers like infrastructure (IaaS), applications platforms (PaaS), serverless architecture (FaaS), and distributed operating systems—to explain what functionality to look for in a container orchestrator, which products are good at which feature sets, and how you can apply this methodology in your research of other container orchestrators.

Hardware

Open Compute

Clustering






Hadoop

  • Apache Hadoop - project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.


Orchestration and management

See also Hosting#PaaS

OpenStack

  • Nova is the project name for OpenStack Compute, a cloud computing fabric controller, the main part of an IaaS system. Individuals and organizations can use Nova to host and manage their own cloud computing systems. Nova originated as a project out of NASA Ames Research Laboratory.
  • Swift is a highly available, distributed, eventually consistent object/blob store. Organizations can use Swift to store lots of data efficiently, safely, and cheaply.
  • Glance provides services for discovering, registering, and retrieving virtual machine images. Glance has a RESTful API that allows querying of VM image metadata as well as retrieval of the actual image.
  • Cinder provides an infrastructure for managing volumes in OpenStack. It was originally a Nova component called nova-volume, but has become an independent project since the Folsom release.
  • Neutron (was Quantum) provides "networking as a service" between interface devices (e.g., vNICs) managed by other Openstack services (e.g., nova).
  • Keystone is the identity service used by OpenStack for authentication (authN) and high-level authorization (authZ). It currently supports token-based authN and user-service authorization. It has recently been rearchitected to allow for expansion to support proxying external services and AuthN/AuthZ mechanisms such as oAuth, SAML and openID in future versions.

OpenNebula

  • OpenNebula.org is an open-source project developing the industry standard solution for building and managing virtualized enterprise data centers and enterprise private clouds.
    • OpenNebula is an open-source cloud computing toolkit for managing heterogeneous distributed data center infrastructures.

OpenShift

Cloud Foundry

Juju

  • Juju enables you to use Charms to deploy your application architectures to EC2, OpenStack, Azure your data center and even your own Ubuntu based laptop. Moving between environments is simple giving you the flexibility to switch hosts whenever you want — for free.

Landscape

Eucalyptus

Cocaine

  • Cocaine (Configurable Omnipotent Custom Applications Integrated Network Engine) is an open-source PaaS (platform as a service) system for creating custom cloud hosting apps that are similar to Google App Engine or Heroku. Any library or service can be implemented as a service in Cocaine using a special API. Several indispensable services have already been implemented this way, including a service for detecting a user's region or language, a service for accessing MongoDB storage, and a URL fetcher.

MaestroNG

  • MaestroNG - command-line utility that allows for automatically managing the orchestrated deployment and bring up of a set of service instance containers that compose an environment on a set of target host machines. Each host machine is expected to run a Docker daemon. Maestro will then contact the Docker daemon of each host in the environment to figure out the status of the environment and what actions to take based on the requested command.

Dokku

  • Dokku - Docker powered mini-Heroku. The smallest PaaS implementation you've ever seen.

Project Atomic

  • Project Atomic integrates the tools and patterns of container-based application and service deployment with trusted operating system platforms to deliver an end-to-end hosting architecture that's modern, reliable and secure. [5]

Deis

Phusion Passenger

  • Phusion Passenger - Web server and application server. Designed to be fast, robust and lightweight. Makes web app deployments a lot simpler and less complex, by managing your apps' processes and resources for you.

Consul

  • Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable.

GNU parallel

  • GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.

Skydock

  • Skydock monitors docker events when containers start, stop, die, kill, etc and inserts records into a dynamic DNS server skydns. This allows standard DNS queries for services running inside docker containers. Because lets face it, if you have to modify your application code to work with other service discovery solutions you might as well just give up. DNS just works and it works well.

Flynn

  • Flynn simplifies deploying and maintaining applications. Instead of using complex configuration management systems, Flynn allows self-serve management of containerized deployments, making life easier for ops and developers. [7] [8]

Gaudi

Terraform

Nixos

VirtKick

Dokku

DC/OS

Cockpit


Kubernetes

  • Kubernetes | Production-Grade Container Orchestration - an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.


  • Deis - powerful, open source tools that make it easy for teams to create and manage applications on Kubernetes. [11] [12]

Service discovery

See also Network#Distributed

to merge into Configuration and management?

  • https://en.wikipedia.org/wiki/Service_discovery - the automatic detection of devices and services offered by these devices on a computer network. A service discovery protocol (SDP) is a network protocol that helps accomplish service discovery. Service discovery aims to reduce the configuration efforts from users. Service discovery requires a common language to allow software agents to make use of one another's services without the need for continuous user intervention.




Zookeeper

  • ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

Doozer

  • Doozer is a highly-available, completely consistent store for small amounts of extremely important data. When the data changes, it can notify connected clients immediately (no polling), making it ideal for infrequently-updated data for which clients want real-time updates. Doozer is good for name service, database master elections, and configuration data shared between several machines. See When Should I Use It?, below, for details.

etcd

  • etcd - A highly-available key value store for shared configuration and service discovery. etcd is inspired by Apache ZooKeeper and doozer, with a focus on being: Simple: curl'able user facing API (HTTP+JSON), Secure: optional SSL client cert authentication, Fast: benchmarked 1000s of writes/s per instance, Reliable: properly distributed using Raft

SkyDNS

  • SkyDNS is a distributed service for announcement and discovery of services. It leverages Raft for high-availability and consensus, and utilizes DNS queries to discover available services. This is done by leveraging SRV records in DNS, with special meaning given to subdomains, priorities and weights.

Serf

  • Serf is a decentralized solution for service discovery and orchestration that is lightweight, highly available, and fault tolerant.

Used by Flynn.

Discoverd

  • Discoverd - A simple but powerful service discovery system written in Go. It's currently backed by etcd, but can be extended to use ZooKeeper or other distributed consistent stores.

Provision and configuration

See also *nix#Management

Puppet

apt-get install puppet puppetmaster facter rubygems puppet-module

Foreman

Boxen

Chef


Salt

  • Salt is a configuration management system, capable of maintaining remote nodes in defined states (for example, ensuring that specific packages are installed and specific services are running), a distributed remote execution system used to execute commands and query data on remote nodes, either individually or by arbitrary selection criteria

Ansible

  • Sovereign – Ansible playbooks to build and maintain your own personal cloud [15]

Foreman

  • The Foreman is a complete lifecycle management tool for physical and virtual servers. Through deep integration with configuration management, DHCP, DNS, TFTP, and PXE-based unattended installations, Foreman manages every stage of the lifecycle of your physical or virtual servers. The Foreman provides comprehensive, auditable interaction facilities including a web frontend and robust, RESTful API.

Fabric

Capistrano

  • Capistrano is a utility and framework for executing commands in parallel on multiple remote machines, via SSH. It uses a simple DSL (borrowed in part from Rake) that allows you to define tasks, which may be applied to machines in certain roles. It also supports tunneling connections via some gateway machine to allow operations to be performed behind VPN's and firewalls. Capistrano was originally designed to simplify and automate deployment of web applications to distributed environments, and originally came bundled with a set of tasks designed for deploying Rails applications.

CFEngine

Rudder

Synapse

  • Synapse enables you to remotely manage a large number of hosts. It brings together features of Configuration Management and Orchestration in a lightweight framework. Written in Python and using AMQP for messaging between the nodes.

Archipel

Ubuntu Orchestra

SM Framework

  • SM framework is a framework for developing, maintaining and sharing scripts, while exposing them for easy automation via a clean, consistent DSL-centric command-line interface. SM Framework Core (sm) is installed onto a system and is then used to manage that system through extensions. The primary requirement for sm is zsh (>= 4.2), secondarily are standard unix tools such as {bash (>= 3.2), GNU awk, GNU sed, grep, which, ls, cp, tar, curl, gunzip, bunzip2}. While not a direct dependency, nearly all open source extensions currently install via git (> 1.7).

Bcfg2

  • Bcfg2 - helps system administrators produce a consistent, reproducible, and verifiable description of their environment, and offers visualization and reporting tools to aid in day-to-day administrative tasks. It is the fifth generation of configuration management tools developed in the Mathematics and Computer Science Division of Argonne National Laboratory. It is based on an operational model in which the specification can be used to validate and optionally change the state of clients, but in a feature unique to Bcfg2 the client's response to the specification can also be used to assess the completeness of the specification. Using this feature, Bcfg2 provides an objective measure of how good a job an administrator has done in specifying the configuration of client systems. Bcfg2 is therefore built to help administrators construct an accurate, comprehensive specification. Bcfg2 has been designed from the ground up to support gentle reconciliation between the specification and current client states. It is designed to gracefully cope with manual system modifications. Finally, due to the rapid pace of updates on modern networks, client systems are constantly changing; if required in your environment, Bcfg2 can enable the construction of complex change management and deployment strategies.

Fossil

  • Fossil - a simple, high-reliability, distributed software configuration management system with these advanced features: In addition to doing distributed version control like Git and Mercurial, Fossil also supports bug tracking, wiki, and technotes. Fossil has a built-in and intuitive web interface with a rich variety of information pages (examples) promoting situational awareness. Fossil is a single self-contained stand-alone executable. To install, simply download a precompiled binary for Linux, Mac, OpenBSD, or Windows and put it on your $PATH. Easy-to-compile source code is also available. No custom protocols or TCP ports. Fossil uses ordinary HTTP (or HTTPS or SSH) for network communications, so it works fine from behind restrictive firewalls, including proxies. The protocol is bandwidth efficient to the point that Fossil can be used comfortably over dial-up or over the exceedingly slow Wifi on airliners. No server is required, but if you want to set one up, Fossil supports four easy server configurations. Fossil supports "autosync" mode which helps to keep projects moving forward by reducing the amount of needless forking and merging often associated with distributed projects. Fossil stores content using an enduring file format in an SQLite database so that transactions are atomic even if interrupted by a power loss or system crash. Automatic self-checks verify that all aspects of the repository are consistent prior to each commit. Fossil works with repository files (a database with the project's complete history) and with checked-out local trees (the working directory you use to do your work). (See definitions for more background.) The workflow looks like this: Create or clone a repository file. (fossil init or fossil clone) Check out a local tree. (fossil open) Perform operations on the repository (including repository configuration).


cdist

  • cdist - a usable configuration management system. It adheres to the KISS principle and is being used in small up to enterprise grade environments. cdist is an alternative to other configuration management systems like bcfg2, chef, cfengine and puppet.

Fleet


git-deliver

deliver

cocaine

Zookeeper

  • ZooKeeper - a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them ,which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.


init-scripts

Slaughter

Augeas

  • Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files.

Itamae

  • Itamae - Configuration management tool inspired by Chef, but simpler and lightweight. Formerly known as Lightchef.

Holo

  • Holo - users ship configuration in packages, usually called "holograms". These can be built with the regular package building tools (debuild, rpmbuild, makepkg, etc.) or with Holo's own holo-build tool that offers a much more pleasant syntax and process. Let's go through an example hologram that installs and starts an OpenSSH server and tweaks some of its configuration.

FAI

  • FAI is a non-interactive system to install, customize and manage Linux systems and software configurations on computers as well as virtual machines and chroot environments, from small networks to large-scale infrastructures like clusters and cloud environments. It's a tool for unattended mass deployment of Linux. You can take one or more virgin PC's, turn on the power, and after a few minutes, the systems are installed, and completely configured to your exact needs, without any interaction necessary.

Propellor

  • Propellor - a configuration management system using Haskell and Git. Each system has a list of properties, which Propellor ensures are satisfied. Linux and FreeBSD are supported. Propellor is configured via a git repository, which typically lives in ~/.propellor/ on your development machine. Propellor clones the repository to each host it manages, in a secure way. See components for details. Properties are defined using Haskell in the file ~/.propellor/config.hs. There is fairly complete API documentation, which includes many built-in Properties for dealing with Apt and Apache , Cron and Commands , Dns and Docker, etc. There is no special language as used in puppet, chef, ansible, etc.. just the full power of Haskell. Hopefully that power can be put to good use in making declarative properties that are powerful, nicely idempotent, and easy to adapt to a system's special needs!

doit

  • doit - a task management & automation tool. doit comes from the idea of bringing the power of build-tools to execute any kind of task. doit is a modern open-source build-tool written in python designed to be simple to use and flexible to deal with complex work-flows. It is specially suitable for building and managing custom work-flows where there is no out-of-the-box solution available. doit has been successfully used on: systems test/integration automation, scientific computational pipelines, content generation, configuration management, etc.

Other






  • FOG Project - Deploy and manage any desktop operating system, anywhere. FOG Project can capture, deploy, and manage Windows, Mac OSX, and various Linux distributions. Computers can be securely managed with FOG Project remotely, from anywhere in the world

Exodus

  • https://github.com/intoli/exodus - a tool that makes it easy to successfully relocate Linux ELF binaries from one system to another. This is useful in situations where you don't have root access on a machine or where a package simply isn't available for a given Linux distribution. For example, CentOS 6.X and Amazon Linux don't have packages for Google Chrome or aria2. Server-oriented distributions tend to have more limited and outdated packages than desktop distributions, so it's fairly common that one might have a piece of software installed on their laptop that they can't easily install on a remote machine.


Habitat

  • Habitat - centers application configuration, management, and behavior around the application itself, not the infrastructure that the app runs on. Habitat provides automation that can programmatically and declaratively build, deploy, and manage your application and services, both stateful and stateless. Habitat can be deployed and run on various infrastructure environments including bare metal, VM, containers, and PaaS.

To sort


Mesos

  • Apache Mesos - a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark, and other applications on a dynamically shared pool of nodes.

Graph Engine

  • Graph Engine - a distributed in-memory data processing engine, underpinned by a strongly-typed RAM store and a general distributed computation engine. The distributed RAM store provides a globally addressable high-performance key-value store over a cluster of machines. Through the RAM store, GE enables the fast random data access power over a large distributed data set. The capability of fast data exploration and distributed parallel computing makes GE a natural large graph processing platform. GE supports both low-latency online query processing and high-throughput offline analytics on billion-node large graphs.

Flynn

Dokku

ShipBuilder


Symbiosis

  • Symbiosis - comes as standard with all new Bytemark servers, so if you're a recent customer or have re-imaged your system recently, it's ready to go. You can easily host PHP, Perl/CGI, htaccess files and MySQL, including any custom modules and settings that you might need. You have root access, so you don't need to ask permission to host new domains, different PHP settings, or create a database. You can even host any other Linux technology such as Java, Rails or Erlang. you can host any number of mailboxes under a domain, including forwarders, IMAP/POP3 boxes and autoresponders. There is a simple local anti-spam service based on Spamassassin, but if you need managed filtering you can quickly elect to pass your email through our anti-spam service.

TurnKey Linux

Retrieved from "https://wiki.thingsandstuff.org/index.php?title=Stack&oldid=45665"