*nix

From Things and Stuff Wiki
Revision as of 15:49, 13 October 2021 by Milk (talk | contribs) (→‎Dune)
Jump to navigation Jump to search


General

See also #Commands, Storage/Files, Distros, Debugging, #Tmux, GUI, etc.

Mainly Linux.


  • https://en.wikipedia.org/wiki/Multics - Multiplexed Information and Computing Service, is an influential early time-sharing operating system, based around the concept of a single-level memory. Virtually all modern operating systems were heavily influenced by Multics, often through Unix, which had been created by the people who had worked on Multics—either directly (Linux, macOS) or indirectly (Microsoft Windows NT).
  • Multics - the story of the Multics operating system for people interested in the system's history, especially Multicians. The site's goals are to preserve the technical ideas and advances of Multics so others don't need to reinvent them. record the history of Multics, its builders, and its users before we all forget. give credit where it's due for important innovations. remember some good times and good people.


  • List of UNIX alternatives with desirable capabilities - "People often ask if there's examples of OS's better than UNIX. This comes up when critics like me bash its design choices or especially if someone posts UNIX Hater's Handbook. Here's a few, esp from past, that give you an idea that did things differently in a way worth copying in UNIX or another OS project."[1]


Community

See also Chat#IRC - Freenode



Troubleshooting

  • LinuxCommand.org - Now what? You have Linux installed and running. The GUI is working fine, but you are getting tired of changing your desktop themes. You keep seeing this "terminal" thing.

man pages

  • https://en.wikipedia.org/wiki/Man_page - short for manual page) is a form of software documentation usually found on a Unix or Unix-like operating system. Topics covered include computer programs (including library and system calls), formal standards and conventions, and even abstract concepts. A user may invoke a man page by issuing the man command. By default, man typically uses a terminal pager program such as more or less to display its output.
man man
 # rtfm.


  • Linux man pages online - links from this page display HTML renderings of the man pages from the Linux man-pages project as well as a curated collection of pages from various other free software projects. List of all man pages: by section,  alphabetically, by project, by Individual sections (page names only)
  • Linux man-pages project - documents the Linux kernel and C library interfaces that are employed by user-space programs. With respect to the C library, the primary focus is the GNU C library (glibc), although, where known, documentation of variations on other C libraries available for Linux is also included. The project provides manual pages in the following sections:
  1. User commands; man-pages includes a very few Section 1 pages that document programs supplied by the GNU C library.
  2. System calls documents the system calls provided by the Linux kernel.
  3. Library functions documents the functions provided by the standard C library.
  4. Devices documents details of various devices, most of which reside in /dev.
  5. Files describes various file formats, and includes proc(5), which documents the /proc file system.
  6. Overviews, conventions, and miscellaneous.
  7. Superuser and system administration commands; man-pages includes a very few Section 8 pages that document programs supplied by the GNU C library.



man --html="google-chrome-stable file%c//%s" --html cd
 # load the manual into chrome browser
  • man whoami | man2html # a Perl program that reads formatted nroff text from standard input (stdin) and writes a HTML document to standard output (stdout). A CGI program is provided to allow hypertext navigation of a system's manpages via a web browser.


  • https://github.com/gapan/gtkman - a simple GTK+2 manual page viewer, written in Python. Manual pages are viewed by specifying their name and optionally the section they are in, just as with the original man command. The manual pages are displayed in simple text form using the default system monospace font.


whatis ls
  # list the various manual pages for the ls command



apropos [string]
  # search the whatis database for strings

apropos ls lists all manual pages that contain ls in their title.



  • mangle.sh - Mangle man pages to show just the parts you need (suitable for aliasing to "man") [3]


  • bro pages are a highly readable supplement to man pages with concise, common-case examples for Unix commands


  • mandoc - a suite of tools compiling mdoc, the roff macro language of choice for BSD manual pages, and man, the predominant historical language for UNIX manuals. It is small, ISO C, ISC-licensed, and quite fast. The main component of the toolset is the mandoc utility program, based on the libmandoc validating compiler, to format output for UTF-8 and ASCII UNIX terminals, HTML 5, PostScript, and PDF. mandoc has predominantly been developed on OpenBSD and is both an OpenBSD and a BSD.lv project. We strive to support all interested free operating systems, in particular FreeBSD, NetBSD, DragonFly, illumos, Minix 3, and both GNU- and musl-based Linux distributions, as well as all systems running the pkgsrc portable package build system. To support mandoc development, consider donating to the OpenBSD foundation.





UNIX

  • https://en.wikipedia.org/wiki/Unix - a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, development starting in the 1970s at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and others.

Initially intended for use inside the Bell System, AT&T licensed Unix to outside parties in the late 1970s, leading to a variety of both academic and commercial Unix variants from vendors like the University of California, Berkeley (BSD), Microsoft (Xenix), IBM (AIX), and Sun Microsystems (Solaris). In the early 1990s, AT&T sold its rights in Unix to Novell, which then sold its Unix business to the Santa Cruz Operation (SCO) in 1995.[4] The UNIX trademark passed to The Open Group, a neutral industry consortium, which allows the use of the mark for certified operating systems that comply with the Single UNIX Specification (SUS). As of 2014, the Unix version with the largest installed base is Apple's macOS.

Unix systems are characterized by a modular design that is sometimes called the "Unix philosophy". This concept entails that the operating system provides a set of simple tools that each perform a limited, well-defined function, with a unified filesystem as the main means of communication, and a shell scripting and command language to combine the tools to perform complex workflows. Unix distinguishes itself from its predecessors as the first portable operating system: almost the entire operating system is written in the C programming language, thus allowing Unix to reach numerous platforms.


  • https://en.wikipedia.org/wiki/*nix - or Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification. A Unix-like application is one that behaves like the corresponding Unix command or shell. There is no standard for defining the term, and some difference of opinion is possible as to the degree to which a given operating system or application is "Unix-like".

Guides





History

See also Distros#BSD




Research UNIX

  • https://en.wikipedia.org/wiki/Research_Unix - a term used to refer to versions of the Unix operating system for DEC PDP-7, PDP-11, VAX and Interdata 7/32 and 8/32 computers, developed in the Bell Labs Computing Science Research Center (frequently referred to as Department 1127).

The term Research Unix first appeared in the Bell System Technical Journal (Vol. 57, No. 6, Pt. 2 Jul/Aug 1978) to distinguish it from other versions internal to Bell Labs (such as PWB/UNIX and MERT) whose code-base had diverged from the primary CSRC version. However, that term was little-used until Version 8 Unix, but has been retroactively applied to earlier versions as well. Prior to V8, the operating system was most commonly called simply UNIX (in caps) or the UNIX Time-Sharing System.

AT&T licensed Version 5 to educational institutions, and Version 6 also to commercial sites. Schools paid $200 and others $20,000, discouraging most commercial use, but Version 6 was the most widely used version into the 1980s. Research Unix versions are often referred to by the edition of the manual that describes them, because early versions and the last few were never officially released outside of Bell Labs, and grew organically. So, the first Research Unix would be the First Edition, and the last the Tenth Edition. Another common way of referring to them is Version x (or Vx) Unix, where x is the manual edition. All modern editions of Unix—excepting Unix-like implementations such as Coherent, Minix, and Linux—derive from the 7th Edition.

Starting with the 8th Edition, versions of Research Unix had a close relationship to BSD. This began by using 4.1cBSD as the basis for the 8th Edition. In a Usenet post from 2000, Dennis Ritchie described these later versions of Research Unix as being closer to BSD than they were to UNIX System V, which also included some BSD code.


  • https://en.wikipedia.org/wiki/Multi-Environment_Real-Time - was a hybrid time-sharing/real-time operating system developed in the 1970s at Bell Labs for use in embedded minicomputers (in particular PDP-11s). It was later renamed UNIX Real-Time (UNIX-RT). A version called Duplex Multi Environment Real Time (DMERT) was the operating system for the AT&T 3B20D telephone switching minicomputer, designed for high availability; DMERT was later renamed Unix RTR (Real-Time Reliable).



  • https://en.wikipedia.org/wiki/Version_6_Unix - also called Version 6 Unix or just V6, was the first version of the Unix operating system to see wide release outside Bell Labs. It was released in May 1975 and, like its direct predecessor, targeted the DEC PDP-11 family of minicomputers. It was superseded by Version 7 Unix in 1978/1979, although V6 systems remained in regular operation until at least 1985.


  • https://en.wikipedia.org/wiki/xv6 - xv6 is a modern reimplementation of Sixth Edition Unix in ANSI C for multiprocessor x86 systems. It is used for pedagogical purposes in MIT's Operating Systems Engineering (6.828) course.


  • https://en.wikipedia.org/wiki/PWB/UNIX - an early, now discontinued, version of the Unix operating system created in the Bell Labs Computer Science Research Group of AT&T. Its stated goal was to provide a time-sharing working environment for large groups of programmers, writing software for larger batch processing computers. Two major releases of Programmer's Workbench were produced. PWB/UNIX 1.0, released July 1, 1977 was based on Version 6 Unix; PWB 2.0 was based on Version 7 Unix. The operating system was advertised by Bell System Software as late as 1981 and edition 1.0 was still on an AT&T price list for educational institutions in 1984. Most of PWB/UNIX was later incorporated in the commercial UNIX System III and UNIX System V releases.


  • https://en.wikipedia.org/wiki/Version_7_Unix - an important early release of the Unix operating system. V7, released in 1979, was the last Bell Laboratories release to see widespread distribution before the commercialization of Unix by AT&T Corporation in the early 1980s. V7 was originally developed for Digital Equipment Corporation's PDP-11 minicomputers and was later ported to other platforms.
  • https://en.wikipedia.org/wiki/UNIX/32V - an early version of the Unix operating system from Bell Laboratories, released in June 1979. 32V was a direct port of the Seventh Edition Unix to the DEC VAX architecture.


  • https://en.wikipedia.org/wiki/Ancient_UNIX - a term coined by Santa Cruz Operation,[citation needed] to describe early releases of the Unix code base released prior to Unix System III, particularly the Research Unix releases prior to and including Version 7 (the base for UNIX/32V as well as later developments of AT&T Unix).

System III / V

AT&T announced System III in late 1981, and it was first released outside of Bell Labs in 1982. UNIX System III was a mix of various AT&T Unixes: PWB/UNIX 2.0, CB UNIX 3.0, UNIX/TS 3.0.1 and UNIX/32V. System III supported the DEC PDP-11 and VAX computers.

The system was apparently called System III because it was considered the outside release of UNIX/TS 3.0.1 and CB UNIX 3 which were internally supported Bell Labs Unices; its manual refers to it as UNIX Release 3.0 and there were no Unix versions called System I or System II. There was no official release of UNIX/TS 4.0 (which would have been System IV) either, so System III was succeeded by System V, based on UNIX/TS 5.0.


  • https://en.wikipedia.org/wiki/UNIX_System_V - pronounced: "System Five", is one of the first commercial versions of the Unix operating system. It was originally developed by AT&T and first released in 1983. Four major versions of System V were released, numbered 1, 2, 3, and 4. System V Release 4, or SVR4, was commercially the most successful version, being the result of an effort, marketed as "Unix System Unification", which solicited the collaboration of the major Unix vendors. It was the source of several common commercial Unix features. System V is sometimes abbreviated to SysV.


BSD

  • https://en.wikipedia.org/wiki/Berkeley_Software_Distribution - a Unix operating system derivative developed and distributed by the Computer Systems Research Group (CSRG) of the University of California, Berkeley, from 1977 to 1995. Today the term "BSD" is often used non-specifically to refer to any of the BSD descendants which together form a branch of the family of Unix-like operating systems. Operating systems derived from the original BSD code remain actively developed and widely used.

In the 1980s, BSD was widely adopted by vendors of workstation-class systems in the form of proprietary Unix variants such as DEC ULTRIX and Sun Microsystems SunOS. Although these proprietary BSD derivatives were largely superseded by the UNIX System V Release 4 and OSF/1 systems in the 1990s (both of which incorporated BSD code and are the basis of other modern Unix systems), later BSD releases provided a basis for several open source development projects that are ongoing, e.g., FreeBSD, OpenBSD, NetBSD, Darwin, and TrueOS. These, in turn, have been incorporated in whole or in part in modern proprietary operating systems, e.g., the TCP/IP networking code in Windows NT 3.1 and most of the foundation of Apple's macOS and iOS.

to sort


  • unix-jun72 - Restoration of 1st Edition UNIX kernel sources from pdf document.




  • Ulix - Literate Unix, is a Unix-like operating system that was developed at University of Erlangen-Nuremberg. We used Donald E. Knuth's concept of Literate Programming for the implementation and documentation. The intention was to create a fully working system which can be used in operating system courses to show students how OS concepts (such as paging and scheduling) can be implemented. Literate programs are very accessible because they can be read like a book; the order of presentation is not enforced by program logic or compiler restrictions, but instead is guided by the implementer's creative process. Ulix was written in C and Assembler for the Intel x86 architecture; for literate programming we used Norman Ramsey's noweb tool.


POSIX / SUS

  • POSIX, an acronym for "Portable Operating System Interface", is a family of standards specified by the IEEE for maintaining compatibility between operating systems. POSIX defines the application programming interface (API), along with command line shells and utility interfaces, for software compatibility with variants of Unix and other operating systems.


  • UniForum - In 1981, the Unix platform was just emerging and there were no centralized resources in place to support this growth environment. UniForum was founded by a group of engineers and entrepreneurs who sought to provide a forum for individuals and companies interested in Unix and open systems. UniForum was born in 1980 as /usr/group, with the distinct mission to make Unix a standardized operating system that industry would adopt and use, within an open competitive framework. Later, /usr/group evolved into UniForum as a reflection of what we had become: a thriving community of open systems professionals. In March 1997, UniForum merged its operations and staff with The Open Group.


  • http://en.wikipedia.org/wiki/Single_UNIX_Specification - the collective name of a family of standards for computer operating systems, compliance with which is required to qualify for to use the "UNIX" trademark. The core specifications of the SUS are developed and maintained by the Austin Group, which is a joint working group of IEEE, ISO JTC 1 SC22 and The Open Group.




  • POSIX.1-2008 (IEEE Std 1003.1™-2008 / The Open Group Technical Standard Base Specifications, Issue 7). The 2013 edition incorporates Technical Corrigendum 1 addressing problems discovered since the approval of the 2008 edition.

POSIX.1-2008 defines a standard operating system interface and environment, including a command interpreter (or “shell”), and common utility programs to support applications portability at the source code level. POSIX.1-2008 is intended to be used by both application developers and system implementors and comprises four major components (each in an associated volume):

  • General terms, concepts, and interfaces common to all volumes of this standard, including utility conventions and C-language header definitions, are included in the Base Definitions volume.
  • Definitions for system service functions and subroutines, language-specific system services for the C programming language, function issues, including portability, error handling, and error recovery, are included in the System Interfaces volume.
  • Definitions for a standard source code-level interface to command interpretation services (a “shell”) and common utility programs for application programs are included in the Shell and Utilities volume.
  • Extended rationale that did not fit well into the rest of the document structure, which contains historical information concerning the contents of POSIX.1-2008 and why features were included or discarded by the standard developers, is included in the Rationale (Informative) volume.

The following areas are outside the scope of POSIX.1-2008:

  • Graphics interfaces
  • Database management system interfaces
  • Record I/O considerations
  • Object or binary code portability
  • System configuration and resource availability


  • Open POSIX Test Suite - a test suite for POSIX 2001 APIs, not tied to specific implementations. It provides conformance, functional, and stress testing. Initial focus is on Threads, Clocks & Timers, Signals, Message Queues, and Semaphores.




Apple

See also Apple


  • https://en.wikipedia.org/wiki/A/UX - Apple Computer's implementation of the Unix operating system for Macintosh computers, integrated with System 7's graphical interface and application compatibility. Launched in 1988 and discontinued in 1995 with version 3.1.1, it is Apple's first official Unix-based operating system. A/UX requires select models of 68k-based Macintosh with an FPU and a paged memory management unit (PMMU), including the Macintosh II, SE/30, Quadra, and Centris series. It was never the predecessor to macOS, a variant of UNIX currently bundled with Apple's desktop computers. Described by InfoWorld as "an open systems solution with the Macintosh at its heart", the operating system is based on UNIX System V Release 2.2. It includes some additional features from System V Releases 3 and 4 and BSD versions 4.2 and 4.3. It is POSIX- and System V Interface Definition (SVID)-compliant and includes TCP/IP networking from version 2 onward. Having a Unix-compatible, POSIX-compliant operating system made it possible for Apple to bid for large contracts to supply computers to U.S. federal government institutes

MINIX

  • https://en.wikipedia.org/wiki/MINIX - a POSIX-compliant (since version 2.0), Unix-like computer operating system based on a microkernel architecture. MINIX was first released in 1987, with its complete source code made available to universities for study in courses and research. It has been free and open-source software since it was re-licensed under the BSD license in April 2000.[

Linux

  • Linux/README - These are the release notes for Linux version 4. Read them carefully, as they tell you what this is all about, explain how to install the kernel, and what to do if something goes wrong.

Linux is a clone of the operating system Unix, written from scratch by Linus Torvalds with assistance from a loosely-knit team of hackers across the Net. It aims towards POSIX and Single UNIX Specification compliance. It has all the features you would expect in a modern fully-fledged Unix, including true multitasking, virtual memory, shared libraries, demand loading, shared copy-on-write executables, proper memory management, and multistack networking including IPv4 and IPv6. It is distributed under the GNU General Public License - see the accompanying COPYING file for more details.




News

  • LWN.net - a reader-supported news site dedicated to producing the best coverage from within the Linux and free software development communities.

Guides



  • Read Just Enough Linux - aims at getting you comfortable with some of the commands used in Linux and to hopefully help you 'get' Linux.




  • eLinux wiki - to preserve and present information about the development and use of Linux in embedded systems as well as open source projects and tools for general embedded development

History

Kernel

  • The Linux Kernel Archives - Linux Kernel Organization is a California Public Benefit Corporation established in 2002 to distribute the Linux kernel and other Open Source software to the public without charge. The Linux Kernel Organization is managed by The Linux Foundation, which provides full technical, financial and staffing support for running and maintaining the kernel.org infrastructure.




  • linux-insides - A series of posts about the linux kernel and its insides.


  • Gittup - This is an entire(-ish) linux distribution in git. Everything is built with tup. That's why it's called gittup. You get the whole development history (where applicable) as part of the standard distribution.





  • The Cathedral and the Bazaar - Eric S. Raymond anatomizes a successful open-source project, fetchmail, that was run as a deliberate test of the surprising theories about software engineering suggested by the history of Linux. I discuss these theories in terms of two fundamentally different development styles, the ``cathedral model of most of the commercial world versus the ``bazaar model of the Linux world. I show that these models derive from opposing assumptions about the nature of the software-debugging task. I then make a sustained argument from the Linux experience for the proposition that ``Given enough eyeballs, all bugs are shallow, suggest productive analogies with other self-correcting systems of selfish agents, and conclude with some exploration of the implications of this insight for the future of software. [32]




  • https://en.wikipedia.org/wiki/System.map - a symbol table used by the kernel. A symbol table is a look-up between symbol names and their addresses in memory. A symbol name may be the name of a variable or the name of a function. The System.map is required when the address of a symbol name, or the symbol name of an address, is needed. It is especially useful for debugging kernel panics and kernel oopses. The kernel does the address-to-name translation itself when CONFIG_KALLSYMS is enabled so that tools like ksymoops are not required.



Building




  • https://en.wikipedia.org/wiki/menuconfig - one of five similar tools that can configure Linux source, a necessary early step needed to compile the source code. make menuconfig, with a menu-driven user interface, allows the user to choose the features of Linux (and other options) that will be compiled. It is normally invoked using the command make menuconfig, menuconfig is a target in Linux Makefile. See also config, nconfig, gconfig, xconfig.


  • https://wiki.archlinux.org/index.php/Modprobed-db - keeps a running list of ALL modules ever probed on a system and allow for easy recall. This is very useful for users wishing to build a minimal kernel via a make localmodconfig which simply takes every module currently probed and switches everything BUT them off in the .config for a kernel resulting in smaller kernel packages and reduced compilation times.
  • Linux Kernel Seeds - pre-made kernel seeds to help you get your Linux system up and running.


  • https://github.com/distcc/distcc - a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network to speed up building. It should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile. Further, one can use it together with native Arch build tools such as makepkg.

Modules

  • https://en.wikipedia.org/wiki/Loadable_kernel_module - or LKM, an object file that contains code to extend the running kernel, or so-called base kernel, of an operating system. LKMs are typically used to add support for new hardware (as device drivers) and/or filesystems, or for adding system calls. When the functionality provided by a LKM is no longer required, it can be unloaded in order to free memory and other resources.
  • Kernel modules are pieces of code that can be loaded and unloaded into the kernel upon demand. They extend the functionality of the kernel without the need to reboot the system.


  • modprobe is a Linux program originally written by Rusty Russell and used to add a loadable kernel module (LKM) to the Linux kernel or to remove an LKM from the kernel. It is commonly used indirectly: udev relies upon modprobe to load drivers for automatically detected hardware.
  • http://linux.die.net/man/8/insmod - trivial program to insert a module into the kernel: if the filename is a hyphen, the module is taken from standard input. Most users will want to use modprobe(8) instead, which is more clever.


  • depmod - handle dependency descriptions for loadable kernel modules
  • Dynamic Kernel Module Support (DKMS) is a framework used to generate Linux kernel modules whose sources do not generally reside in the Linux kernel source tree. DKMS enables kernel device drivers to be automatically rebuilt when a new kernel is installed. An essential feature of DKMS is that it automatically recompiles all DKMS modules if a new kernel version is installed. This allows drivers and devices outside of the mainline kernel to continue working after a Linux kernel upgrade. Another benefit of DKMS is that it allows the installation of a new driver on an existing system, running an arbitrary kernel version, without any need for manual compilation or precompiled packages provided by the vendor.


  • Granary - a framework for creating flexible and efficient tools that analyse or debug arbitrary, binary Linux kernel modules. Granary uses dynamic binary translation (DBT) to dynamically rewrite and comprehensively instrument kernel modules. Granary makes it easy to create flexible tools by supporting context-aware runtime code specialisation, and by integrating high-level static analysis information with low-level instruction manipulation. Module analysis tools built on Granary are efficient because of Granary’s adoption of a relaxed transparency model, and because of Granary’s ability to instrument module code without imposing overhead on non-module kernel code.
  • https://wiki.archlinux.org/index.php/modprobed-db - keeps a running list of ALL modules ever probed on a system and allow for easy recall. This is very useful for users wishing to build a minimal kernel via a make localmodconfig which simply takes every module currently probed and switches everything BUT them off in the .config for a kernel resulting in smaller kernel packages and reduced compilation times.

Drivers

  • LKDDb is an attempt to build a comprensive database of hardware and protocols know by Linux kernels. The driver database includes numeric identifiers of hardware, the kernel configuration menu needed to build the driver and the driver filename. The database is build automagically from kernel sources, so it is very easy to have always the database updated.


Firmware for PCs falls into four categories:

  • Updates to the CPU to work around errata, usually referred to as microcode.
  • Firmware for video controllers. On x86 machines this seems to only apply to ATI devices : Radeons require firmware to be able to use KMS (kernel modesetting - the preferred option) as well as for Xorg. For earlier radeon chips (before the R600), the firmware is still in the kernel.
  • Firmware updates for wired network ports. Mostly they work even without the updates, but one must assume that they will work better with the updated firmware.
  • Firmware for other devices, such as wifi. These devices are not required for the PC to boot, but need the firmware before these devices can be used.




Parameters

There are three ways to pass options to the kernel and thus control its behaviour:

  • When building the kernel.
  • When starting the kernel (usually, when invoked from a boot loader).
  • At runtime (through the files in /proc and /sys). See sysctl for details.
  • bootparam - introduction to boot time parameters of the Linux kernel


  • https://en.wikipedia.org/wiki/Sysctl - an interface for examining and dynamically changing parameters in Unix-like operating systems. The BSD implementation of sysctl uses system calls or system call wrappers, while the Linux implementation primarily uses files contained in a virtual file system.
  • Deprecation of /etc/sysctl.conf From version 207 on, systemd will not apply the settings from /etc/sysctl.conf anymore: it will only apply those from /etc/sysctl.d/*. Since the settings of our /etc/sysctl.conf shipped by procps-ng have become kernel defaults anyway, we have decided to deprecate this file.
cat /proc/sys/net/ipv4/ip_forward

Messages


Timers

  • kernel.org: hrtimers - subsystem for high-resolution kernel timers, the Linux Kernel documentation.

Sources

Kernel patchsets

linux-zen

linux-rt

linux-ck

linux-pf

Nvidia 340 drivers not available.

linux-xanmod

  • XanMod Kernel - a general-purpose Linux kernel distribution with custom settings and new features.Built to provide a more rock-solid, responsive and smooth desktop experience.Supports all recent 64-bit versions of Debian and Ubuntu-based systems.

Hardware


Standards

Linux Standard Base

  • Linux Standard Base (LSB) is a joint project by several Linux distributions under the organizational structure of the Linux Foundation to standardize the software system structure, including the filesystem hierarchy, used with Linux operating system. The LSB is based on the POSIX specification, the Single UNIX Specification, and several other open standards, but extends them in certain areas.


freedesktop.org

  • freedesktop.org is open source / open discussion software projects working on interoperability and shared technology for X Window System desktops. The most famous X desktops are GNOME and KDE, but developers working on any Linux/UNIX GUI technology are welcome to participate. freedesktop.org is building a base platform for desktop software on Linux and UNIX. The elements of this platform have become the backend for higher-level application-visible APIs such as Qt, GTK+, XUL, VCL, WINE, GNOME, and KDE. The base platform is both software and specifications. 
$XDG_DATA_HOME defines the base directory relative to which user specific data files should be stored. If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used.

$XDG_CONFIG_HOME defines the base directory relative to which user specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.

$XDG_DATA_DIRS defines the preference-ordered set of base directories to search for data files in addition to the $XDG_DATA_HOME base directory. The directories in $XDG_DATA_DIRS should be seperated with a colon ':'.

If $XDG_DATA_DIRS is either not set or empty, a value equal to /usr/local/share/:/usr/share/ should be used.

$XDG_CONFIG_DIRS defines the preference-ordered set of base directories to search for configuration files in addition to the $XDG_CONFIG_HOME base directory. The directories in $XDG_CONFIG_DIRS should be seperated with a colon ':'.

If $XDG_CONFIG_DIRS is either not set or empty, a value equal to /etc/xdg should be used.

The order of base directories denotes their importance; the first directory listed is the most important. When the same information is defined in multiple places the information defined relative to the more important base directory takes precedent. The base directory defined by $XDG_DATA_HOME is considered more important than any of the base directories defined by $XDG_DATA_DIRS. The base directory defined by $XDG_CONFIG_HOME is considered more important than any of the base directories defined by $XDG_CONFIG_DIRS.





  • /etc/machine-id - contains the unique machine ID of the local system that is set during installation or boot. The machine ID is a single newline-terminated, hexadecimal, 32-character, lowercase ID. When decoded from hexadecimal, this corresponds to a 16-byte/128-bit value. This ID may not be all zeros.The machine ID is usually generated from a random source during system installation or first boot and stays constant for all subsequent boots. Optionally, for stateless systems, it is generated during runtime during early boot if necessary.The machine ID may be set, for example when network booting, with the systemd.machine_id= kernel command line parameter or by passing the option --machine-id= to systemd. An ID is specified in this manner has higher priority and will be used instead of the ID stored in /etc/machine-id.

Execution


Processes

  • https://en.wikipedia.org/wiki/Process_(computing) - an instance of a computer program that is being executed. It contains the program code and its current activity. Depending on the operating system (OS), a process may be made up of multiple threads of execution that execute instructions concurrently.
  • https://en.wikipedia.org/wiki/Process_management_(computing) - an integral part of any modern-day operating system (OS). The OS must allocate resources to processes, enable processes to share and exchange information, protect the resources of each process from other processes and enable synchronisation among processes. To meet these requirements, the OS must maintain a data structure for each process, which describes the state and resource ownership of that process, and which enables the OS to exert control over each process.




  • UPX: the Ultimate Packer for eXecutables - Homepage - a free, portable, extendable, high-performance executable packer for several executable formats.
    • https://en.wikipedia.org/wiki/UPX - UPX uses a data compression algorithm called UCL, which is an open source implementation of portions of the proprietary NRV (Not Really Vanished) algorithm. UCL has been designed to be simple enough that a decompressor can be implemented in just a few hundred bytes of code. UCL requires no additional memory to be allocated for decompression, a considerable advantage that means that a UPX packed executable usually requires no additional memory. UPX (since 2.90 beta) can use LZMA on most platforms; however, this is disabled by default for 16-bit due to slow decompression speed on older computers (use --lzma to force it on). Starting with version 3.91, UPX also supports 64-Bit (x64) executable files on the Windows platform.[7] This feature is currently declared as experimental.


  • https://en.wikipedia.org/wiki/Process_control_block - also called Task Controlling Block, process table, Task Struct, or Switchframe, is a data structure in the operating system kernel containing the information needed to manage a particular process. The PCB is "the manifestation of a process in an operating system".


  • https://en.wikipedia.org/wiki/Process_identifier - (normally referred to as the process ID or just PID) is a number used by most operating system kernels — such as that of UNIX, Mac OS X or Microsoft Windows — to uniquely identify an active process. This number may be used as a parameter in various function calls, allowing processes to be manipulated, such as adjusting the process's priority or killing it altogether.
  • https://en.wikipedia.org/wiki/Process_group - In POSIX-conformant operating systems, a process group denotes a collection of one or more processes. Process groups are used to control the distribution of signals. A signal directed to a process group is delivered individually to all of the processes that are members of the group.

Process groups are themselves grouped into sessions. Process groups are not permitted to migrate from one session to another, and a process may only create new process groups belonging to the same session as it itself belongs to. Processes are not permitted to join process groups that are not in the same session as they themselves are.

New process images created by a call to a function of the exec family inherit the process group membership and the session membership of the old process image.






BPF




Memory

System calls

See also Debugging#Tracing

  • https://en.wikipedia.org/wiki/System_call - how a program requests a service from an operating system's kernel. This may include hardware-related services (for example, accessing a hard disk drive), creation and execution of new processes, and communication with integral kernel services such as process scheduling. System calls provide an essential interface between a process and the operating system.






/usr/include/sys/syscall.h
/usr/include/bits/syscall.h




System calls are generally not invoked directly, but rather via wrapper functions in glibc (or perhaps some other library). For details of direct invocation of a system call, see intro(2). Often, but not always, the name of the wrapper function is the same as the name of the system call that it invokes.


  • https://en.wikipedia.org/wiki/exec_(system_call) - a functionality of an operating system that runs an executable file in the context of an already existing process, replacing the previous executable. This act is also referred to as an overlay. It is especially important in Unix-like systems, although exists elsewhere. As a new process is not created, the process identifier (PID) does not change, but the machine code, data, heap, and stack of the process are replaced by those of the new program. The exec is available for many programming languages including compilable languages and some scripting languages. In OS command interpreters, the exec built-in command replaces the shell process with the specified program.



  • kexec(8) - a system call that enables you to load and boot into another kernel from the currently running kernel. kexec performs the function of the boot loader from within the kernel. The primary difference between a standard system boot and a kexec boot is that the hardware initialization normally performed by the BIOS or firmware (depending on architecture) is not performed during a kexec boot. This has the effect of reducing the time required for a reboot.





  • https://en.wikipedia.org/wiki/wait_(system_call) - suspends the execution of the parent process while the child executes. When the child process terminates, it returns an exit status to the operating system, which is then returned to the waiting parent process. The parent process then resumes execution.




  • https://en.wikipedia.org/wiki/epoll - a Linux kernel system call, a scalable I/O event notification mechanism, first introduced in Linux kernel 2.5.44. It is meant to replace the older POSIX select(2) and poll(2) system calls, to achieve better performance in more demanding applications, where the number of watched file descriptors is large (unlike the older system calls, which operate in O(n) time, epoll operates in O(1) time). epoll is similar to FreeBSD's kqueue, in that it operates on a configurable kernel object, exposed to user space as a file descriptor of its own.


  • https://en.wikipedia.org/wiki/sync_(Unix) - commits to disk all data in the kernel filesystem buffers, i.e., data which has been scheduled for writing via low-level I/O system calls. Note that higher-level I/O layers such as stdio may maintain separate buffers of their own.


  • https://github.com/stewartsmith/libeatmydata - An LD_PRELOAD library that disables all forms of writing data safely to disk. fsync() becomes a NO-OP, O_SYNC is removed etc. The idea is to use in testing to get faster test runs where real durability is not required. DO NOT use libeatmydata on software where you care about what it stores. It's called lib***EAT-MY-DATA*** for a reason.


  • https://en.wikipedia.org/wiki/select_(Unix) - for examining the status of file descriptors of open input/output channels. The select system call is similar to the poll facility introduced in UNIX System V and later operating systems.
  • https://en.wikipedia.org/wiki/splice_(system_call) - a Linux-specific system call that moves data between a file descriptor and a pipe without a round trip to user space. The related system call vmsplice() moves or copies data between a pipe and user space. Ideally, splice and vmsplice work by remapping pages and do not actually copy any data, which may improve I/O performance. As linear addresses do not necessarily correspond to contiguous physical addresses, this may not be possible in all cases and on all hardware combinations.





  • https://github.com/majek/fluxcapacitor - a tool for making your program run without blocking on timeouts, on functions like poll and select, by spoofing POSIX time functions. It is somewhat similar to:FreezeGun for PythonTimeCop or DeLorean for Ruby. While these tools patch time libraries in Ruby and Python, fluxcapacitor works on a lower layer by "patching" low-level syscalls. That way, it can lie about time to any program in any programming language, as long as it runs on Linux.

Kernel interfaces


  • https://en.wikipedia.org/wiki/Application_binary_interface - the interface between two program modules, one of which is often a library or operating system, at the level of machine code. An ABI determines such details as how functions are called and in which binary format information should be passed from one program component to the next, or to the operating system in the case of a system call.

Adhering to ABIs (which may or may not be officially standardized) is usually the job of the compiler, OS or library writer, but application programmers may have to deal with ABIs directly when writing programs in a mix of programming languages, using foreign function call interfaces between them. ABIs differ from application programming interfaces (APIs), which similarly define interfaces between program components, but at the source code level.




  • http://linux-abi.sourceforge.net/ - The Linux abi is a patch to the linux kernel that allows a linux system to run foreign binaries. This was developed and written by Christoph Hellwig and Joerg Ahrens as a follow on to the iBCS/iBCS2 project written for the older 2.2.x kernel by Mike Jagdis.




Loader

  • https://en.wikipedia.org/wiki/Loader_(computing) - the part of an operating system that is responsible for loading programs and libraries. It is one of the essential stages in the process of starting a program, as it places programs into memory and prepares them for execution. Loading a program involves reading the contents of the executable file containing the program instructions into memory, and then carrying out other required preparatory tasks to prepare the executable for running. Once loading is complete, the operating system starts the program by passing control to the loaded program code.


  • https://en.wikipedia.org/wiki/Binfmt_misc - a capability of the Linux kernel which allows arbitrary executable file formats to be recognized and passed to certain user space applications, such as emulators and virtual machines. It is one of a number of binary format handlers in the kernel that are involved in preparing a user-space program to run.The executable formats are registered through the special purpose file system binfmt_misc file-system interface (usually mounted under part of /proc). This is either done directly by sending special sequences to the register procfs file or using a wrapper like Debian-based distributions binfmt-support package or systemd's systemd-binfmt.service

Formats

ELF

  • https://en.wikipedia.org/wiki/Executable_and_Linkable_Format - ELF, formerly called Extensible Linking Format, is a common standard file format for executables, object code, shared libraries, and core dumps. First published in the System V Release 4 (SVR4) Application Binary Interface (ABI) specification, and later in the Tool Interface Standard, it was quickly accepted among different vendors of Unix systems. In 1999 it was chosen as the standard binary file format for Unix and Unix-like systems on x86 by the 86open project. ELF is flexible and extensible by design, and it is not bound to any particular processor or architecture. This has allowed it to be adopted by many different operating systems on many different platforms.





Shared FLAT

bFLT

  • Binary Flat Format - uClinux uses a Binary Flat format commonly known as BFLT. It is a relatively simple and lightweight executable format based on the original a.out format. It has seen two major revisions, version 2 and more recently version 4. Version 2 is used by the m68k-coff compilers and is still in use with the ARM elf2flt converter while version 4 is used with the m68k elf2flt converter. Like many open source projects worked on by many individuals, little features have creped into versions without the revision field changing. As a consequence what we detail in each version may not strictly be the case in all circumstances. Both the 2.0.x and 2.4.x kernels’ bFLT loaders in the CVS support both formats. Earlier kernels may need to have binfmt_flat.c and flat.c patched to support version 4, should version 4 binaries report the following message when loading

FatELF

  • FatELF - a file format that embeds multiple ELF binaries for different architectures into one file. This is the Linux equivalent of what Mac OS X calls "Universal Binaries."The format is very simple: it adds some accounting info at the start of the file, and then appends all the ELF binaries after it, adding padding for alignment. The end of the file isn't touched, so you can still do things like self-extracting .zip files for multiple architectures with FatELF.FatELF lets you pack binaries into one file, seperated by OS ABI, OS ABI version, byte order and word size, and most importantly, CPU architecture.Work is focused on GNU/Linux, but this could be applied to most modern Unix systems: the BSDs, Solaris, etc.
αcτµαlly pδrταblε εxεcµταblε

Libraries

  • Program Library HOWTO - David A. Wheeler - This HOWTO for programmers discusses how to create and use program libraries on Linux. This includes static libraries, shared libraries, and dynamically loaded libraries.
/lib
  # main library directory

/usr/lib
  # /lib can be a symlink to /usr/lib


Static

  • https://en.wikipedia.org/wiki/Static_library - or statically-linked library is a set of routines, external functions and variables which are resolved in a caller at compile-time and copied into a target application by a compiler, linker, or binder, producing an object file and a stand-alone executable. This executable and the process of compiling it are both known as a static build of the program. Historically, libraries could only be static. Static libraries are either merged with other static libraries and object files during building/linking to form a single executable, or they may be loaded at run-time into the address space of the loaded executable at a static memory offset determined at compile-time/link-time.

Dynamic

  • https://en.wikipedia.org/wiki/Dynamic_linker - the part of an operating system that loads and links the shared libraries needed by an executable when it is executed (at "run time"), by copying the content of libraries from persistent storage to RAM, and filling jump tables and relocating pointers. The specific operating system and executable format determine how the dynamic linker functions and how it is implemented.

Linking is often referred to as a process that is performed when the executable is compiled, while a dynamic linker is a special part of an operating system that loads external shared libraries into a running process and then binds those shared libraries dynamically to the running process. This approach is also called dynamic linking or late linking.


  • https://en.wikipedia.org/wiki/Dynamic_loading - a mechanism by which a computer program can, at run time, load a library (or other binary) into memory, retrieve the addresses of functions and variables contained in the library, execute those functions or access those variables, and unload the library from memory. Unlike static linking and dynamic linking, dynamic loading allows a computer program to start up in the absence of these libraries, to discover available libraries, and to potentially gain additional functionality.



  • http://cateee.net/lkddb/web-lkddb/USELIB.html - This option enables the uselib syscall, a system call used in the dynamic linker from libc5 and earlier. glibc does not use this system call. If you intend to run programs built on libc5 or earlier, you may need to enable this syscall. Current systems running glibc can safely disable this.
  • http://linux.die.net/man/2/uselib - system call uselib() serves to load a shared library to be used by the calling process. It is given a pathname. The address where to load is found in the library itself. The library can have any recognized binary format.


The programs ld.so and ld-linux.so* find and load the shared libraries needed by a program, prepare the program to run, and then run it.

Linux binaries require dynamic linking (linking at run time) unless the -static option was given to ld(1) during compilation.

The program ld.so handles a.out binaries, a format used long ago; ld-linux.so* handles ELF (/lib/ld-linux.so.1 for libc5, /lib/ld-linux.so.2 for glibc2), which everybody has been using for years now. Otherwise both have the same behavior, and use the same support files and programs ldd(1), ldconfig(8) and /etc/ld.so.conf.




  • ldconfig - configure dynamic linker run-time bindings
ldd /usr/bin/ls

linux-vdso.so.1 (0x00007ffd9afde000) libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f35e72f9000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f35e6f58000) /lib64/ld-linux-x86-64.so.2 (0x00007f35e74fd000)


"Linux puts you in full control. This is not always seen from everyone’s perspective, but a power user loves to be in control. ... what happens when we set up the environmental variable LD_PRELOAD? This variable forces some libraries to be loaded for a program. In our case, it loads unrandom.so for random_num, even though the program itself does not ask for it. The following command may be interesting:" 

$ LD_PRELOAD=$PWD/unrandom.so ldd random_nums
linux-vdso.so.1 =>  (0x00007fff369dc000)
/some/path/to/unrandom.so (0x00007f262b439000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f262b044000)
/lib64/ld-linux-x86-64.so.2 (0x00007f262b63d000)



  • https://github.com/wolfcw/libfaketime - libfaketime intercepts various system calls that programs use to retrieve the current date and time. It then reports modified (faked) dates and times (as specified by you, the user) to these programs. This means you can modify the system time a program sees without having to change the time system-wide.



  • https://github.com/riboseinc/retrace - a versatile security vulnerability / bug discovery tool through monitoring and modifying the behavior of compiled binaries on Linux, OpenBSD/FreeBSD/NetBSD (shared object) and macOS (dynamic library).


  • https://github.com/SimonKagstrom/kcov - a Linux/OSX code coverage tester for compiled languages, Python and Bash. Kcov was originally a fork of Bcov, but has since evolved to support a large feature set in addition to that of Bcov. Kcov, like Bcov, uses DWARF debugging information for compiled programs to make it possible to collect coverage information without special compiler switches.


  • https://github.com/sickill/stderred - stderr in red. stderred hooks on write() and a family of stream functions (fwrite, fprintf, error...) from libc in order to colorize all stderr output that goes to terminal thus making it distinguishable from stdout. Basically it wraps text that goes to file with descriptor "2" with proper ANSI escape codes making text red.


  • https://github.com/zardus/preeny - helps you pwn noobs by making it easier to interact with services locally. It disables fork(), rand(), and alarm() and, if you want, can convert a server application to a console one using clever/hackish tricks, and can even patch binaries!


  • https://elinux.org/Electric_Fence - Electric Fence helps you detect two common programming bugs: software that overruns the boundaries of a malloc()memory allocation, software that touches a memory allocation that has been released by free(). Unlike other malloc() debuggers, Electric Fence will detect read accesses as well as writes, and it will pinpoint the exact instruction that causes an error. Electric Fence uses the virtual memory hardware of your computer to place an inaccessible memory page immediately after (or before, at the user's option) each memory allocation.


  • https://github.com/dex4er/fakechroot - fakechroot runs a command in an environment were is additional possibility to use chroot(8) command without root privileges. This is useful for allowing users to create own chrooted environment with possibility to install another packages without need for root privileges. fakechroot replaces some libc library functions (chroot(2), open(2), etc.) by ones that simulate the effect of being called with root privileges. These wrapper functions are in a shared library libfakechroot.so which is loaded through the LD_PRELOAD mechanism of the dynamic loader. (See ld.so(8))
  • https://blitiri.com.ar/p/libfiu - libfiu is a C library for fault injection. It aims to simplify automated testing of software behaviour at failure scenarios, for software which is supposed to be fault-resistant or at least gracefully cope with failures. It provides functions to mark "points of failure" inside your code (the core API), and functions to enable/disable the failure of those points (the control API).
  • https://github.com/paultag/tmperamental - a library that intercepts filesystem writes, and causes loud failures when writes are attempted on /tmp/*. The idea here is by setting TMPDIR and friends, you are able to use libtmperamental to catch programs in the act of not respecting such env-vars.

vDSO

  • https://en.wikipedia.org/wiki/vDSO - virtual dynamically linked shared object) is a Linux kernel mechanism for exporting a carefully selected set of kernel space routines to user space applications so that applications can call these kernel space routines in-process, without incurring the performance penalty of a context switch that is inherent when calling these same kernel space routines by means of the system call interface. vDSO has been developed to offer the vsyscall features and to overcome its limitations: small amount of allocated memory allowing only 4 system calls and same addresses ABI in each process compromising security. This security issue has been by emulating a virtual system call introducing a penalty latency.

a.out

  • https://en.wikipedia.org/wiki/a.out - a file format used in older versions of Unix-like computer operating systems for executables, object code, and, in later systems, shared libraries. The name stands for "assembler output", and was coined by Ken Thompson as the fixed name for output of his PDP-7 assembler in 1968. The term was subsequently applied to the format of the resulting file, to contrast with other formats for object code.

"a.out" remains the default output file name for executables created by certain compilers and linkers when no output name is specified, even though the created files actually are not in the a.out format.

https://blackfin.uclinux.org/doku.php?id=toolchain:executable_file_formats


Memory

See also Computing#Memory



  • ps_mem - A utility to accurately report the core memory usage for a program


Paging



  • https://en.wikipedia.org/wiki/Non-uniform_memory_access - a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. Under NUMA, a processor can access its own local memory faster than non-local memory (memory local to another processor or memory shared between processors). The benefits of NUMA are limited to particular workloads, notably on servers where the data is often associated strongly with certain tasks or users.


Co-ordination

  • https://en.wikipedia.org/wiki/Synchronization_(computer_science) - refers to one of two distinct but related concepts: synchronization of processes, and synchronization of data. Process synchronization refers to the idea that multiple processes are to join up or handshake at a certain point, in order to reach an agreement or commit to a certain sequence of action. Data synchronization refers to the idea of keeping multiple copies of a dataset in coherence with one another, or to maintain data integrity. Process synchronization primitives are commonly used to implement data synchronization.


RCU

  • LWN.net: What is RCU, Fundamentally? - Read-copy update (RCU) is a synchronization mechanism that was added to the Linux kernel in October of 2002. RCU achieves scalability improvements by allowing reads to occur concurrently with updates. In contrast with conventional locking primitives that ensure mutual exclusion among concurrent threads regardless of whether they be readers or updaters, or with reader-writer locks that allow concurrent reads but not in the presence of updates, RCU supports concurrency between a single updater and multiple readers. RCU ensures that reads are coherent by maintaining multiple versions of objects and ensuring that they are not freed up until all pre-existing read-side critical sections complete. RCU defines and uses efficient and scalable mechanisms for publishing and reading new versions of an object, and also for deferring the collection of old versions. These mechanisms distribute the work among read and update paths in such a way as to make read paths extremely fast. In some cases (non-preemptable kernels), RCU's read-side primitives have zero overhead.



  • https://en.wikipedia.org/wiki/Read-copy-update - a synchronization mechanism based on mutual exclusion.[note 1] It is used when performance of reads is crucial and is an example of space–time tradeoff, enabling fast operations at the cost of more space.Read-copy-update allows multiple threads to efficiently read from shared memory by deferring updates after pre-existing reads to a later time while simultaneously marking the data, ensuring new readers will read the updated data. This makes all readers proceed as if there were no synchronization involved, hence they will be fast, but also making updates more difficult.

FPU

Inter-process communication

  • https://en.wikipedia.org/wiki/Inter-process_communication - the activity of sharing data across multiple and commonly specialized processes using communication protocols. Typically, applications using IPC are categorized as clients and servers, where the client requests data and the server responds to client requests. Many applications are both clients and servers, as commonly seen in distributed computing. Methods for achieving IPC are divided into categories which vary based on software requirements, such as performance and modularity requirements, and system circumstances, such as network bandwidth and latency.


Interrupts

The Linux kernel provides the following IPC mechanisms: Signals, Anonymous Pipes, Named Pipes or FIFOs, SysV Message Queues, POSIX Message Queues, SysV Shared memory, POSIX Shared memory, SysV semaphores, POSIX semaphores, FUTEX locks, File-backed and anonymous shared memory using mmap, UNIX Domain Sockets, Netlink Sockets, Network Sockets, Inotify mechanisms, FUSE subsystem, D-Bus subsystem.

Interrupts can be viewed as a mean of communication between the CPU and the OS kernel. Signals can be viewed as a mean of communication between the OS kernel and OS processes.

Interrupts may be initiated by the CPU (exceptions - e.g.: divide by zero, page fault), devices (hardware interrupts - e.g: input available), or by a CPU instruction (traps - e.g: syscalls, breakpoints). They are eventually managed by the CPU, which "interrupts" the current task, and invokes an OS-kernel provided ISR/interrupt handler.

Signals may be initiated by the OS kernel (e.g: SIGFPE, SIGSEGV, SIGIO), or by a process(kill()). They are eventually managed by the OS kernel, which delivers them to the target thread/process, invoking either a generic action (ignore, terminate, terminate and dump core) or a process-provided signal handler. [50]



Shared memory

SysV
POSIX

Unix signals

  • https://en.wikipedia.org/wiki/Unix_signal - a limited form of inter-process communication used in Unix, Unix-like, and other POSIX-compliant operating systems. A signal is an asynchronous notification sent to a process or to a specific thread within the same process in order to notify it of an event that occurred. Signals originated in 1970s Bell Labs Unix and have been more recently specified in the POSIX standard.

When a signal is sent, the operating system interrupts the target process' normal flow of execution to deliver the signal. Execution can be interrupted during any non-atomic instruction. If the process has previously registered a signal handler, that routine is executed. Otherwise, the default signal handler is executed.


A signal is a software interrupt delivered to a process. The operating system uses signals to report exceptional situations to an executing program. Some signals report errors such as references to invalid memory addresses; others report asynchronous events, such as disconnection of a phone line.



Unix domain sockets



Pipes

If |& is used, the standard error of command is connected to command2's standard input through the pipe; it is shorthand for 2>&1 |.






  • Socketpipe - directly connects the input/output of a remote process with local programs over a TCP/IP socket. Although the initial communication setup is performed through client-server intermediaries such as SSH or RSH, the communication channel that socketpipe establishes is a direct socket connection between the local and the remote processes, similar to a local pipe. The use of socketpipe removes the inefficiency of multiple data copies and context switches through RSH/SSH, and can in many cases offer dramatic throughput improvements.


  • Spiped - pronounced "ess-pipe-dee", is a utility for creating symmetrically encrypted and authenticated pipes between socket addresses, so that one may connect to one address (e.g., a UNIX socket on localhost) and transparently have a connection established to another address (e.g., a UNIX socket on a different system). This is similar to 'ssh -L' functionality, but does not use SSH and requires a pre-shared symmetric key.


  • https://github.com/elkowar/pipr - a commandline pipe-building tool, written in Rust!Pipr can automatically evaluate the pipeline you're editing in the background, showing you the results as you go. This makes writing complex sed and awk chains a lot easier, as you'll immediately see what they do.Because this could be dangerous, (imagine typing rm ./*.txt to delete all text files, but it already being executed at rm ./*, deleting all files in your working directory), Pipr uses bubblewrap to execute your command in an isolated, read-only environment, making it safe to use. I wont give any guarantees, but you should be good :D.To allow for even more efficiency, Pipr features a command history and a bookmark system, allowing you to quickly go back to previously worked on pipelines or look at how you did something before.
pv
  • pv - Pipe Viewer, a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.
pv source > target
  # cp with progress bar

tar czf images.tar.gz image1 image2 image3 | pv > target
  # tar

pv source | gzip > target.gz
  # or
gzip < source | pv > target.gz
  # gzip

-L 25k
  # rate limit

-d PID[:FD], --watchfd PID[:FD]
  # watch existing process



  • Pipeglade - Graphical User Interfaces, The UNIX Way, a helper program that displays graphical user interfaces for other programs. It renders the GUI definition found in a GtkBuilder file (created using the Glade Interface Designer), and communicates with the main program solely via pipes or fifos. [58]



Named pipes


D-Bus

  • D-Bus - a message bus system, a simple way for applications to talk to one another. In addition to interprocess communication, D-Bus helps coordinate process lifecycle; it makes it simple and reliable to code a "single instance" application or daemon, and to launch applications and daemons on demand when their services are needed.

D-Bus supplies both a system daemon (for events such as "new hardware device added" or "printer queue changed") and a per-user-login-session daemon (for general IPC needs among user applications). Also, the message bus is built on top of a general one-to-one message passing framework, which can be used by any two apps to communicate directly (without going through the message bus daemon). Currently the communicating applications are on one computer, or through unencrypted TCP/IP suitable for use behind a firewall with shared NFS home directories. (Help wanted with better remote transports - the transport mechanism is well-abstracted and extensible.)



kdbus

  • http://lwn.net/Articles/580194/ - "kdbus, an in-kernel implementation of D-Bus. This implementation is able to carry large amounts of data; it can be reasonably used for gigabyte-sized message streams. It can perform zero-copy message passing, but even in the worst case, a message and its response are passed with no more than two copy operations, two validations, and two context switches. Full credential information (user ID, process ID, SELinux label, control group information, capabilities, and much more) is passed with each message, and all messages carry timestamps. Kdbus is always available to the system (no need to wait for the D-Bus daemon to be started), Linux security modules can hook into it directly, various race conditions have been fixed, and the API has simplified."

VARLINK

  • VARLINK - an interface description format and protocol that aims to make services accessible to both humans and machines in the simplest feasible way.A varlink interface combines the classic UNIX command line options, STDIN/OUT/ERROR text formats, man pages, service metadata and provides the equivalent over a single file descriptor, a.k.a. “FD3”.Varlink is plain-text, type-safe, discoverable, self-documenting, remotable, testable, easy to debug. Varlink is accessible from any programming environment. [59]

Utils

at bg chroot fg time



kill

In Unix and Unix-like operating systems, kill is a command used to send a signal to a process. By default, the message sent is the termination signal, which requests that the process exit. But kill is something of a misnomer; the signal sent may have nothing to do with process killing. The kill command is a wrapper around the kill() system call, which sends signals to processes or process groups on the system, referenced by their numeric process IDs (PIDs) or process group IDs (PGIDs). kill is always provided as a standalone utility as defined by the POSIX standard. However, most shells have built-in kill commands that may slightly differ from it.

A process can be sent a SIGTERM signal in four ways (the process ID is '1234' in this case): 

kill 1234
kill -s TERM 1234
kill -TERM 1234
kill -15 1234

The process can be sent a SIGKILL signal in three ways: 

kill -s KILL 1234
kill -KILL 1234
kill -9 1234

killall -u username
  terminate all username's applications to log them out

nohup

  • nohup - Run a command immune to hangups, runs the given command with hangup signals ignored, so that the command can continue running in the background after you log out.

reptyr

  • https://github.com/nelhage/reptyr - a utility for taking an existing running program and attaching it to a new terminal. Started a long-running process over ssh, but have to leave and don't want to interrupt it? Just start a screen, use reptyr to grab it, and then kill the ssh session and head on home.

Debugging

See also Development#Debugging


Scheduling


  • https://en.wikipedia.org/wiki/O(1)_scheduler - a kernel scheduling design that can schedule processes within a constant amount of time, regardless of how many processes are running on the operating system. One of the major goals of operating system designers is to minimize overhead and jitter of OS services, so that application programmers who use them endure less of a performance impact. O(1) scheduler providing "constant time" scheduling services has helped in this regard.



nice

nice -n 19 [command]
  run a command at lowest priority

nice -n -20 [command]
  run a command at highest priority
VeryNice

ionice

  • https://linux.die.net/man/1/ionice - sets or gets the io scheduling class and priority for a program. If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process.

Ananicy

Threads

  • https://en.wikipedia.org/wiki/Thread_(computer_science) - a thread of execution is the smallest sequence of programmed instructions that can be managed independently by a scheduler, which is typically a part of the operating system. The implementation of threads and processes differs between operating systems, but in most cases a thread is a component of a process. Multiple threads can exist within one process, executing concurrently (one starting before others finish) and share resources such as memory, while different processes do not share these resources. In particular, the threads of a process share its instructions (executable code) and its context (the values of its variables at any given time).
  • https://en.wikipedia.org/wiki/Green_threads - threads that are scheduled by a runtime library or virtual machine (VM) instead of natively by the underlying operating system. Green threads emulate multithreaded environments without relying on any native OS capabilities, and they are managed in user space instead of kernel space, enabling them to work in environments that do not have native thread support.



"If you want to run code on many platforms, go for Posix Threads. They are available almost everywhere and are quite mature. OTOH if you only use Linux/gcc std::thread is perfectly fine - it has a higher abstraction level, a really good interface and plays nicely with other C++11 classes.

"The C++11 std::thread class unfortunately doesn't work reliably (yet) on every platform, even if C++11 seems available. For instance in native Android std::thread or Win64 it just does not work or has severe performance bottlenecks (as of 2012).

"A good replacement is boost::thread - it is very similar to std::thread (actually it is from the same author) and works reliably, but, of course, it introduces another dependency from a third party library."

"The std::thread library is implemented on top of pthreads in an environment supporting pthreads (for example: libstdc++).

"I think the big difference between the two is abstraction. std::thread is a C++ class library. The std::thread library includes many abstract features, for example: scoped locks, recursive mutexes, future/promise design pattern implementations, and more."


Checkpointing

  • https://github.com/maaziz/cryopid - allows you to capture the state of a running process in Linux and save it to a file. This file can then be used to resume the process later on, either

after a reboot or even on another machine. The advantages of CryoPID over other checkpointing systems available for Linux is that is does not require any prior thought in order to use it on a process. Binaries do not need modification or special loading procedures. The checkpointed binary need not be killed either.


CPU scaling

  • https://en.wikipedia.org/wiki/Dynamic_frequency_scaling - also known as CPU throttling) is a technique in computer architecture whereby the frequency of a microprocessor can be automatically adjusted "on the fly" depending on the actual needs, to conserve power and reduce the amount of heat generated by the chip. Dynamic frequency scaling helps preserve battery on mobile devices and decrease cooling cost and noise on quiet computing settings, or can be useful as a security measure for overheated systems (e.g. after poor overclocking). Dynamic frequency scaling is used in all ranges of computing systems, ranging from mobile systems to data centers to reduce the power at the times of low workload.


cat /sys/devices/system/cpu/cpufreq/policy*/scaling_available_governors

Watchdog


Security


cgroups

  • Control Groups - provide a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour.


Capabilities

  • https://en.wikipedia.org/wiki/Capability-based_security - a concept in the design of secure computing systems, one of the existing security models. A capability (known in some systems as a key) is a communicable, unforgeable token of authority. It refers to a value that references an object along with an associated set of access rights. A user program on a capability-based operating system must use a capability to access an object. Capability-based security refers to the principle of designing user programs such that they directly share capabilities with each other according to the principle of least privilege, and to the operating system infrastructure necessary to make such transactions efficient and secure. Capability-based security is to be contrasted with an approach that uses hierarchical protection domains.
  • Linux manual page: capabilities(7) - For the purpose of performing permission checks, traditional UNIX implementations distinguish two categories of processes: privileged processes (whose effective user ID is 0, referred to as superuser or root), and unprivileged processes (whose effective UID is nonzero). Privileged processes bypass all kernel permission checks, while unprivileged processes are subject to full permission checking based on the process's credentials (usually: effective UID, effective GID, and supplementary group list). Starting with kernel 2.2, Linux divides the privileges traditionally associated with superuser into distinct units, known as capabilities, which can be independently enabled and disabled. Capabilities are a per-thread attribute.

Process Accounting

  • Process Accounting - a security method in which an administrator may keep track of system resources used and their allocation among users, provide for system monitoring, and minimally track a user's commands.Process accounting has both positive and negative points. One of the positives is that an intrusion may be narrowed down to the point of entry. A negative is the amount of logs generated by process accounting, and the disk space they may require. This section walks an administrator through the basics of process accounting.

Boot


Booting a Linux installation involves multiple stages and software components, including firmware initialization, execution of a boot loader, loading and startup of a Linux kernel image, and execution of various startup scripts and daemons. For each of these stages and components there are different variations and approaches; for example, GRUB, LILO, SYSLINUX or Loadlin can be used as boot loaders, while the startup scripts can be either traditional init-style, or the system configuration can be performed through more modern alternatives such as systemd or Upstart.





  • e4rat - reduce boot time (into X) by some 50% for ext4

BIOS / UEFI

  • https://en.wikipedia.org/wiki/BIOS - an acronym for Basic Input/Output System and also known as the System BIOS, ROM BIOS or PC BIOS) is a type of firmware used to perform hardware initialization during the booting process (power-on startup) on IBM PC compatible computers, and to provide runtime services for operating systems and programs. The BIOS firmware is built into personal computers (PCs), and it is the first software they run when powered on. The name itself originates from the Basic Input/Output System used in the CP/M operating system in 1975. Originally proprietary to the IBM PC, the BIOS has been reverse engineered by companies looking to create compatible systems and the interface of that original system serves as a de facto standard.

The fundamental purposes of the BIOS in modern PCs are to initialize and test the system hardware components, and to load a boot loader or an operating system from a mass memory device. The BIOS additionally provides an abstraction layer for the hardware, i.e., a consistent way for application programs and operating systems to interact with the keyboard, display, and other input/output (I/O) devices. Variations in the system hardware are hidden by the BIOS from programs that use BIOS services instead of directly accessing the hardware. MS-DOS (PC DOS), which was the dominant PC operating system from the early 1980s until the mid-1990s, relied on BIOS services for disk, keyboard, and text display functions. MS Windows NT, Linux, and other protected mode operating systems in general ignore the abstraction layer provided by the BIOS and do not use it after loading, instead accessing the hardware components directly.

Most BIOS implementations are specifically designed to work with a particular computer or motherboard model, by interfacing with various devices that make up the complementary system chipset. Originally, BIOS firmware was stored in a ROM chip on the PC motherboard; in modern computer systems, the BIOS contents are stored on flash memory so it can be rewritten without removing the chip from the motherboard. This allows easy updates to the BIOS firmware so new features can be added or bugs can be fixed, but it also creates a possibility for the computer to become infected with BIOS rootkits.



  • https://en.wikipedia.org/wiki/Unified_EFI_Forum or UEFI Forum is an alliance between several leading technology companies to modernize the booting process. The board of directors includes representatives from eleven "Promoter" companies: AMD, American Megatrends, Apple, Dell, HP, IBM, Insyde Software, Intel, Lenovo, Microsoft, and Phoenix Technologies.


ACPI

  • https://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface - ACPI - specification provides an open standard that the operating systems can use for computer hardware discovery, configuration, power management, and monitoring. Internally, ACPI exports the available functionalities by providing certain instruction lists as part of the system firmware, which the operating system kernel interprets and executes to perform desired operations, using a form of embedded virtual machine.

"Overall design decision was not without criticism. In November 2003, Linus Torvalds—author of the Linux kernel—described ACPI as "a complete design disaster in every way". In 2001, other senior Linux software developers like Alan Cox expressed concerns about the requirements that bytecode from an external source must be run by the kernel with full privileges, as well as the overall complexity of the ACPI specification. In 2014, Mark Shuttleworth, founder of the Ubuntu Linux distribution, compared ACPI with Trojan horses."


  • iasl - ACPI Source Language compiler/decompiler



Coreboot

  • https://en.wikipedia.org/wiki/Coreboot - formerly known as LinuxBIOS, is a free software project aimed at replacing proprietary firmware (BIOS or UEFI) found in most computers with a lightweight firmware designed to perform only the minimum number of tasks necessary to load and run a modern 32-bit or 64-bit operating system.

Since coreboot initializes the bare hardware, it must be ported to every chipset and motherboard that it supports. As a result, coreboot is available only for a limited number of hardware platforms and motherboard models.

Network boot


  • https://en.wikipedia.org/wiki/Preboot_Execution_Environment - PXE, sometimes pronounced as pixie, a specification that describes a standardized client-server environment that boots a software assembly, retrieved from a network, on PXE-enabled clients. On the client side it requires only a PXE-capable network interface controller (NIC), and uses a small set of industry-standard network protocols such as DHCP and TFTP.

The concept behind the PXE originated in the early days of protocols like BOOTP/DHCP/TFTP, and as of 2015 it forms part of the Unified Extensible Firmware Interface (UEFI) standard. Given fast and reliable local area networks (LANs), PXE is the most frequent choice for operating system booting, installation and deployment.


  • netboot.xyz - a way to select various operating system installers or utilities from one place within the BIOS without the need of having to go retrieve the media to run the tool. iPXE is used to provide a user friendly menu from within the BIOS that lets you easily choose the operating system you want along with any specific types of versions or bootable flags. You can remote attach the ISO to servers, set it up as a rescue option in Grub, or even set up your home network to boot to it by default so that it's always available. [72]


  • PXELINUX - a Syslinux derivative, for booting from a network server using a network ROM conforming to the Intel PXE (Pre-Execution Environment) specification. PXELINUX is not a program intended to be flashed or burned into a PROM on the network card. For such possibility, check out iPXE (http://ipxe.org/). If you want to create PXE-compliant boot PROM for your network card (to use with PXELINUX, for example), check out NetBoot (http://netboot.sourceforge.net/).


  • Netboot - tools for network boot, allows to remote boot a computer over an IP network without access to a hard disk or a diskette. It is therefore ideally suited for diskless or thin clients using an x86 processor. Netboot is open source software licensed by the GNU GPL, and contains all necessary tools including a PXE compliant bootrom. Operating systems supported on the diskless client are Linux, FreeBSD and many DOS variants.


  • iPXE - the leading open source network boot firmware. It provides a full PXE implementation enhanced with additional features such as: boot from a web server via HTTP, boot from an iSCSI SAN, boot from a Fibre Channel SAN via FCoE, boot from an AoE SAN, boot from a wireless network, boot from a wide-area network, boot from an Infiniband network, control the boot process with a script, You can use iPXE to replace the existing PXE ROM on your network card, or you can chainload into iPXE to obtain the features of iPXE without the hassle of reflashing. iPXE is free, open-source software licensed under the GNU GPL (with some portions under GPL-compatible licences), and is included in products from several network card manufacturers and OEMs.
    • https://en.wikipedia.org/wiki/iPXE - an open-source implementation of the Preboot eXecution Environment (PXE) client firmware and bootloader, created in 2010 as a fork of gPXE. It can be used to enable computers without built-in PXE support to boot from the network, or to extend an existing PXE client implementation so it supports additional protocols. While standardized PXE clients use TFTP to transfer data, non-standardized iPXE client firmware adds the ability to retrieve data through other protocols, including HTTP, iSCSI, ATA over Ethernet (AoE), and Fibre Channel over Ethernet (FCoE). Also, on supported hardware iPXE firmware can use a Wi-Fi link rather than requiring a wired connection. iPXE firmware cannot be considered as a "drop-in" replacement for PXE firmware.



  • ERPXE - a complete PXE solution featuring a broad range of recovery tools and various OS installations in one box.

ERPXE is a free and open source project, which uses the GNU General Public License version 3.



  • https://en.wikipedia.org/wiki/Remote_Initial_Program_Load - a protocol for starting a computer and loading its operating system from server via a network. Such a server runs a network operating system such as LAN Manager, LAN Server, Windows NT Server, Novell NetWare, Solaris or Linux. RIPL is similar to Preboot Execution Environment (PXE), but it uses the Novell NetWare based boot method. It was originally developed by IBM.



Boot sector

Boot loader

"In order to boot Arch Linux, a Linux-capable boot loader such as GRUB(2), Syslinux, LILO or GRUB Legacy must be installed to the Master Boot Record or the GUID Partition Table. The boot loader is responsible for loading the kernel and initial ramdisk before initiating the boot process."

loadlin

  • https://en.wikipedia.org/wiki/loadlin - a Linux boot loader that runs under DOS or Microsoft Windows (95, 98 or Me only). It allows the Linux system to load and replace the running DOS/Windows without altering existing DOS/Windows system files. Due to its structure, loadlin only works on DOS-based operating systems, and will not work on NT-based versions of Windows.

LILO

  • https://en.wikipedia.org/wiki/LILO_(boot_loader) - a boot loader for Linux and was the default boot loader for most Linux distributions in the years after the popularity of loadlin. Today, most distributions use GRUB as the default boot loader. LILO has been discontinued in December 2015, with a request for potential developers.

GRUB

grub-mkconfig -o /boot/grub/grub.cfg
  # regenerate config after changing kernels installed


Syslinux


rEFInd

Plop Boot Manager(s)

Smart BootManager

systemd-boot


SeaBIOS

  • SeaBIOS - an open source implementation of a 16bit X86 BIOS. SeaBIOS can run in an emulator or it can run natively on X86 hardware with the use of coreboot.

U-Boot

Petitboot

  • Petitboot - a platform independent bootloader based on the Linux kexec warm reboot mechanism. Petitboot supports loading kernel, initrd and device tree files from any Linux mountable filesystem, plus can load files from the network using the FTP, SFTP, TFTP, NFS, HTTP and HTTPS protocols. Petitboot can boot any operating system that includes kexec boot support.Petitboot includes an ncurses based user interface program. This user interface program can be used to boot the system remotely via telnet or ssh sessions. Multiple user interface programs can run simultaneously.Petitboot is licensed under the GPLv2.

Petitboot has a client-server design. The petitboot server, pb-discover, uses several event mechanisms to dynamically update its knowledge of available boot configurations as they become available or unavailable at run time. The petitboot client programs implement the petitboot user interfaces and display the curent boot configurations known by pb-discover.

pb-discover exposes a generic event mechanism that allows utility programs to provide boot configuration information. Petitboot includes utility programs to process boot configuration info from DHCP servers, and if configured to do so, will manage any boot configuration info received.





zectl

Initial ramdisk

initrd

  • initrd (initial ramdisk) is a scheme for loading a temporary root file system into memory in the boot process of the Linux kernel. initrd and initramfs refer to two different methods of achieving this. Both are commonly used to make preparations before the real root file system can be mounted.

initramfs

  • https://en.wikipedia.org/wiki/initramfs - abbreviated from "initial RAM file system", is the successor of initrd ("initial ramdisk"). It is a cpio ("copy in and out") archive of the initial file system that gets loaded into memory during the Linux startup process. The Linux kernel copies the contents of the archive into the built-in rootfs file-system-like infrastructure (which is either based on ramfs, or the more regulated tmpfs), and then attempts to run the resulting /init program, before falling back on older methods of initialization; the init program is meant to complete certain tasks before the "real" or final root file system is mounted over rootfs. Thus, initramfs needs to contain all of the device drivers and tools needed to mount the final root file system.






Mkinitcpio
ps -p 1 -o comm=
Dracut

distri

u-root

  • u-root - an embeddable root file system intended to be placed in a flash device as part of the firmware image, along with a Linux kernel. Unlike most embedded root file systems, which consist of large binaries, u-root only has five: an init program and four Go compiler binaries.

Kernal loading

  • https://en.wikipedia.org/wiki/vmlinux - a statically linked executable file that contains the Linux kernel in one of the object file formats supported by Linux, which includes ELF, COFF and a.out. The vmlinux file might be required for kernel debugging, symbol table generation or other operations, but must be made bootable before being used as an operating system kernel by adding a multiboot header, bootsector and setup routines.
  • https://wiki.archlinux.org/index.php/kernel_mode_setting - a method for setting display resolution and depth in the kernel space rather than user space. The Linux kernel's implementation of KMS enables native resolution in the framebuffer and allows for instant console (tty) switching. KMS also enables newer technologies (such as DRI2) which will help reduce artifacts and increase 3D performance, even kernel space power-saving.


  • https://en.wikipedia.org/wiki/Device_tree - also written devicetree, is a data structure describing the hardware components of a particular computer so that the operating system's kernel can use and manage those components, including the CPU or CPUs, the memory, the buses and the peripherals.The device tree was derived from SPARC-based workstations and servers via the Open Firmware project. The current Devicetree specification is targeted at smaller systems, but is still used with some server-class systems (for instance, those described by the Power Architecture Platform Reference). Personal computers with the x86 architecture generally do not use device trees, relying instead on various auto configuration protocols to discover hardware. Systems which use device trees usually pass a static device tree (perhaps stored in ROM) to the operating system, but can also generate a device tree in the early stages of booting. As an example, Das U-Boot and kexec can pass a device tree when launching a new operating system. On systems with a boot loader that does not support device trees, a static device tree may be installed along with the operating system; the Linux kernel supports this approach. The Devicetree specification is currently managed by a community named devicetree.org, which is associated with, among others, Linaro and Arm.

init

  • init (short for initialization) is a daemon process that is the direct or indirect ancestor of all other processes. It automatically adopts all orphaned processes. Init is the first process started during booting, and is typically assigned PID number 1. It is started by the kernel using a hard-coded filename, and if the kernel is unable to start it, a kernel panic will result. Init continues running until the system is shut down. The design of init has diverged in Unix systems such as System III and System V, from the functionality provided by the init in Research Unix and its BSD derivatives. The usage on most Linux distributions is compatible with System V, but some distributions, such as Slackware, use a BSD-style and others, such as Gentoo, have their own customized version. Several replacement init implementations have been written which attempt to address design limitations in the standard versions. These include systemd and Upstart, the latter being used by Ubuntu and some other Linux distributions.






  • https://en.wikipedia.org/wiki/Run_commands - In the context of Unix-like systems, the term rc stands for the phrase "run commands". It is used for any file that contains startup information for a command. It is believed to have originated somewhere in 1965 from a runcom facility from the MIT Compatible Time-Sharing System (CTSS). From Brian Kernighan and Dennis Ritchie: There was a facility that would execute a bunch of commands stored in a file; it was called runcom for "run commands", and the file began to be called "a runcom". rc in Unix is a fossil from that usage. Tom Van Vleck, a Multics engineer, has also reminisced about the extension rc: "The idea of having the command processing shell be an ordinary slave program came from the Multics design, and a predecessor program on CTSS by Louis Pouzin called RUNCOM, the source of the '.rc' suffix on some Unix configuration files." This is also the origin of the name of the Plan 9 from Bell Labs shell by Tom Duff, the rc shell. It is called "rc" because the main job of a shell is to "run commands".While not historically precise, rc may also be expanded as "run control", because an rc file controls how a program runs. For instance, the editor Vim looks for and reads the contents of the .vimrc file to determine its initial configuration. In The Art of Unix Programming, Eric S. Raymond consistently refers to rc files as "run-control" files. [74]



Research UNIX / BSD


SysVinit

  • http://linux.die.net/man/8/chkconfig - provides a simple command-line tool for maintaining the /etc/rc[0-6].d directory hierarchy by relieving system administrators of the task of directly manipulating the numerous symbolic links in those directories.


runlevels


daemontools


runit

  • runsv - starts and monitors a service and optionally an appendant log service
  • runsvdir - starts and monitors a collection of runsv(8) processes
  • chpst - runs a program with a changed process state
  • svlogd - runit’s service logging daemon

Supervisor

  • Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. Instead it is meant to be used to control processes related to a project or a customer, and is meant to start like any other program at boot time. 2004.

Upstart

  • http://upstart.ubuntu.com/ - 2006
    • http://en.wikipedia.org/wiki/Upstart - an event-based replacement for the traditional init daemon – the method by which several Unix-like computer operating systems perform tasks when the computer is started. It was written by Scott James Remnant, a former employee of Canonical Ltd. Upstart operates asynchronously; it handles starting of the tasks and services during boot and stopping them during shutdown, and also supervises the tasks and services while the system is running. Easy transition and perfect backwards compatibility with sysvinit were the explicit design goals. Upstart allows for extensions to its event model through the use of initctl to input custom, single events, or event bridges to integrate many or more-complicated events. By default, Upstart includes bridges for socket, dbus, udev, file, and dconf events; additionally, more bridges (for example, a Mach ports bridge, or a devd (found on FreeBSD systems) bridge) are possible.

OpenRC

  • OpenRC is a dependency based init system that works with the system provided init program, normally /sbin/init. It is not a replacement for /sbin/init. OpenRC is 100% compatible with Gentoo init scripts, which means you can probably find one for the daemons you want to start in the Gentoo Portage Tree. OpenRC, however, is not exclusively used by Gentoo Linux, so our goal is to be platform-agnostic. 2007.

systemd

  • systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. It can work as a drop-in replacement for sysvinit. 2010.


  • systemd is a system and service manager for Linux operating systems. When run as first process on boot (as PID 1), it acts as init system that brings up and maintains userspace services.




  • systemctl - used to introspect and control the state of the "systemd" system and service manager. Please refer to systemd(1) for an introduction into the basic concepts and functionality this tool manages.
systemctl [OPTIONS...] COMMAND [NAME...]

alias sy=systemctl



Unit files



  • https://www.freedesktop.org/software/systemd/man/systemd.target.html - A unit configuration file whose name ends in ".target" encodes information about a target unit of systemd, which is used for grouping units and as well-known synchronization points during start-up. Target units do not offer any additional functionality on top of the generic functionality provided by units. They exist merely to group units via dependencies (useful as boot targets), and to establish standardized names for synchronization points used in dependencies between units. Among other things, target units are a more flexible replacement for SysV runlevels in the classic SysV init system. (And for compatibility reasons special target units such as runlevel3.target exist which are used by the SysV runlevel compatibility code in systemd. See systemd.special(7) for details).



  • https://www.freedesktop.org/software/systemd/man/systemd.special.html - basic.target, bluetooth.target, ctrl-alt-del.target, cryptsetup.target, cryptsetup-pre.target, dbus.service, dbus.socket, default.target, display-manager.service, emergency.target, exit.target, final.target, getty.target, graphical.target, halt.target, hibernate.target, hybrid-sleep.target, initrd-fs.target, kbrequest.target, kexec.target, local-fs.target, local-fs-pre.target, multi-user.target, network.target, network-online.target, network-pre.target, nss-lookup.target, nss-user-lookup.target, paths.target, poweroff.target, printer.target, reboot.target, remote-fs.target, remote-fs-pre.target, rescue.target, initrd-root-fs.target, rpcbind.target, runlevel2.target, runlevel3.target, runlevel4.target, runlevel5.target, shutdown.target, sigpwr.target, sleep.target, slices.target, smartcard.target, sockets.target, sound.target, suspend.target, swap.target, sysinit.target, syslog.socket, system-update.target, time-sync.target, timers.target, umount.target, -.slice, system.slice, user.slice, machine.slice
Boot


systemd-journald

systemd-journald is a system service that collects and stores logging data. It creates and maintains structured, indexed journals based on logging information that is received from a variety of sources:

  • Kernel log messages, via kmsg
  • Simple system log messages, via the libc syslog(3) call
  • Structured system log messages via the native Journal API, see sd_journal_print(4)
  • Standard output and standard error of system services
  • Audit records, via the audit subsystem


journalctl
  # see all syslog messages

journalctl -b
  # show syslog messages since boot

journalctl -b -1
  # show syslog msgs for previous boot

journalctl --since today
journalctl --since yesterday
journalctl --since 2014-01-23
  # filter by time


journalctl -e
  # jump to end of the last 1000 messages

journalctl -f
  # follow on new messages

journalctl -r
  # reverse messages, recent at top



journalctl -u sshd
  # show messages related to ssh daemon

journalctl -pb err
  # error messages since boot

journalctl -k
  # kernel messages



journalctl _PID=1
  # messages for PID 1 (systemd)

journalctl _COMM=cron
  # messages for cron

journalctl -o verbose -
  # output verbose journal record meta information

output formats; 

cat              json             json-sse         short-full       short-monotonic  short-unix                                                    
export           json-pretty      short            short-iso        short-precise    verbose



journalctl --disk-usage
systemd-networkd


systemd-analyze
systemd-analyze
  # list basic boot time

systemd-analyze blame
  # list specific boot timings

systemd-analyze critical-chain
  # list blocking boot timings

systemd-analyze dot --to-pattern='*.target' --from-pattern='*.target' | dot -Tpng > targets.png
  # generate DAG image of service targets
Other
Articles



launchd

sinit

Other





Reboot and shutdown




Real-Time

  • linuxrealtime - the Linux Real Time Manual. The goal of this manual/wiki is to serve as platform for real-time experts to share their knowledge; and for system designers and developers to use as a guide to implement real-time applications using Linux.





  • Xenomai - brings POSIX and traditional RTOS APIs for porting time-critical applications to Linux-based platforms. When the native Linux kernel cannot meet the response time requirements of the application, Xenomai supplements it with Cobalt, a small real-time infrastructure which schedules time-critical activities independently from the main kernel logic.


  • rts - a collection of systemd services and scripts that allow to set pre-defined real-time related scheduling settings, using tuna.

Devices

BIO

Driver

In Linux environments, programmers can build device drivers as parts of the kernel, separately as loadable modules, or as user-mode drivers (for certain types of devices where kernel interfaces exist, such as for USB devices). Makedev includes a list of the devices in Linux: ttyS (terminal), lp (parallel port), hd (disk), loop, sound (these include mixer, sequencer, dsp, and audio)...

The Microsoft Windows .sys files and Linux .ko modules contain loadable device drivers. The advantage of loadable device drivers is that they can be loaded only when necessary and then unloaded, thus saving kernel memory.


Because of the diversity of modern hardware and operating systems, drivers operate in many different environments. Drivers may interface with:

  • Printers
  • Video adapters
  • Network cards
  • Sound cards
  • Local buses of various sorts—in particular, for bus mastering on modern systems
  • Low-bandwidth I/O buses of various sorts (for pointing devices such as mice, keyboards, USB, etc.)
  • Computer storage devices such as hard disk, CD-ROM, and floppy disk buses (ATA, SATA, SCSI)
  • Implementing support for different file systems
  • Image scanners
  • Digital cameras

Common levels of abstraction for device drivers include:

For hardware:

  • Interfacing directly
  • Writing to or reading from a device control register
  • Using some higher-level interface (e.g. Video BIOS)
  • Using another lower-level device driver (e.g. file system drivers using disk drivers)
  • Simulating work with hardware, while doing something entirely different[citation needed]

For software:

  • Allowing the operating system direct access to hardware resources
  • Implementing only primitives
  • Implementing an interface for non-driver software (e.g., TWAIN)
  • Implementing a language, sometimes quite high-level (e.g., PostScript)

So choosing and installing the correct device drivers for given hardware is often a key component of computer system configuration.

Virtual device drivers

Virtual device drivers represent a particular variant of device drivers. They are used to emulate a hardware device, particularly in virtualization environments, for example when a DOS program is run on a Microsoft Windows computer or when a guest operating system is run on, for example, a Xen host. Instead of enabling the guest operating system to dialog with hardware, virtual device drivers take the opposite role and emulate a piece of hardware, so that the guest operating system and its drivers running inside a virtual machine can have the illusion of accessing real hardware. Attempts by the guest operating system to access the hardware are routed to the virtual device driver in the host operating system as e.g., function calls. The virtual device driver can also send simulated processor-level events like interrupts into the virtual machine. Virtual devices may also operate in a non-virtualized environment. For example, a virtual network adapter is used with a virtual private network, while a virtual disk device is used with iSCSI. A good example for virtual device drivers can be Daemon Tools.

There are several variants of virtual device drivers, such as VxDs, VLMs, VDDs.




  • https://en.wikipedia.org/wiki/Raw_device - a special kind of logical device associated with a character device file that allows a storage device such as a hard disk drive to be accessed directly, bypassing the operating system's caches and buffers (although the hardware caches might still be used). Applications like a database management system can use raw devices directly, enabling them to manage how data is cached, rather than deferring this task to the operating system.
  • http://linux.die.net/man/8/raw - used to bind a Linux raw character device to a block device. Any block device may be used: at the time of binding, the device driver does not even have to be accessible (it may be loaded on demand as a kernel module later).


CUSE enables character devices to be implemented from userland much like FUSE allows filesystems. On initialization /dev/cuse is created. By opening the file and replying to the CUSE_INIT request userland CUSE server can create a character device. After that the operation is very similar to FUSE.


Device file

Device nodes correspond to resources that an operating system's kernel has already allocated. Unix identifies those resources by a major number and a minor number, both stored as part of the structure of a node. The assignment of these numbers occurs uniquely in different operating systems and on different computer platforms. Generally, the major number identifies the device driver and the minor number identifies a particular device (possibly out of many) that the driver controls: in this case, the system may pass the minor number to a driver. However, in the presence of dynamic number allocation, this may not be the case (e.g. on FreeBSD 5 and up).

As with other special file types, the computer system accesses device nodes using standard system calls and treats them like regular computer files. Two standard types of device files exist; unfortunately their names are, for historical reasons, rather counter-intuitive, and explanations of the difference between the two are often incorrect as a result.

Nodes are created by the mknod system call. The command-line program for creating nodes is also called mknod. Nodes can be moved or deleted by the usual filesystem system calls (rename, unlink) and commands (mv, rm). When passed the option -R or -a while copying a device node, the cp -l command creates a new device node with the same attributes of the original.





  • https://en.wikipedia.org/wiki/Ioctl - an abbreviation of input/output control, is a system call for device-specific input/output operations and other operations which cannot be expressed by regular system calls. It takes a parameter specifying a request code; the effect of a call depends completely on the request code. Request codes are often device-specific. For instance, a CD-ROM device driver which can instruct a physical device to eject a disc would provide an ioctl request code to do that. Device-independent request codes are sometimes used to give userspace access to kernel functions which are only used by core system software or still under development.

The ioctl system call first appeared in Version 7 of Unix under that name. It is supported by most Unix and Unix-like systems, including Linux and Mac OS X, though the available request codes differ from system to system. Microsoft Windows provides a similar function, named "DeviceIoControl", in its Win32 AP


  • https://en.wikipedia.org/wiki/Netlink - Netlink is designed and used for transferring miscellaneous networking information between the kernel space and userspace processes. Networking utilities, such as the iproute2 family and the utilities used for configuring mac80211-based wireless drivers, use Netlink to communicate with the Linux kernel from userspace. Netlink provides a standard socket-based interface for userspace processes, and a kernel-side API for internal use by kernel modules. Originally, Netlink used the AF_NETLINK socket family. Netlink is designed to be a more flexible successor to ioctl; RFC 3549 describes the protocol in detail.



/dev/null

/dev/zero

/dev/full

  • https://en.wikipedia.org/wiki//dev/full - a special file that always returns the error code ENOSPC (meaning "No space left on device") on writing, and provides an infinite number of null characters to any process that reads from it (similar to /dev/zero). This device is usually used when testing the behaviour of a program when it encounters a "disk full" error.

Device manager

makedev

devfsd

  • https://en.wikipedia.org/wiki/devfsd - a device manager for the Linux kernel. Primarily, it creates device nodes in the /dev directory when kernel drivers make the underlying hardware accessible. The nodes exist in a virtual device file system named devfs. In systems that support many different types of hardware, each of which has its own device nodes, this is more convenient than creating all possible device nodes beforehand and in a real filesystem.

While devfs was a step forward, it had several disadvantages of its own. Since version 2.5 of the Linux kernel, devfs has been succeeded by udev and devtmpfs.

udev

  • udev is a generic kernel device manager that subsumes the use of hotplug and hwdetect. It runs as a daemon on a Linux system and listens (via netlink socket) to uevents the kernel sends out if a new device is initialized or a device is removed from the system. Loads kernel modules by utilizing coding parallelism to provide a potential performance advantage versus loading these modules serially. The modules are therefore loaded asynchronously.


Manages /dev. Rules in /etc/udev/rules.d/ and /usr/lib/udev/rules.d/


  • udevadm - expects a command and command specific options. It controls the runtime behavior of systemd-udevd, requests kernel events, manages the event queue, and provides simple debugging mechanisms.
udevadm info -a -n [device path]


udisksd (udisks2) and udisks-daemon (udisks) are started on-demand by D-Bus, and should not be enabled explicitly (see man udisksd and man udisks-daemon). They can be controlled through the command-line with udisksctl and udisks, respectively. See man udisksctl and man udisks for more information.



storaged

  • storaged - next evolution step of udisks2. storaged started as a "complementary daemon" for udisks2 providing API for LVM (at that time heavily required by the Cockpit web management project ASAP). Later the LVM API part was converted into a udisks2 module and an ability to load modules was added to udisks2, but with no luck merging the changes to the upstream udisks2 codebase. The only way forward was to fork udisks2 and continue with the development in the forked project - named storaged. You can read more details about this in an explanatory email from Tomas Smetana 1.

Since then storaged has been getting a lot of attention and contributions including extra features like support for bcache, Btrfs and iSCSI (the client side) as well as new tests and memory and performance analysis. We believe storaged will become the storage configuration and management userspace API similarly to how NetworkManager became an API for networking. Not only for personal/desktop use cases, but also for enterprise solutions.

However, we realize that there's an existing API, the udisks2 API, covering a great portion of the existing features and thus after a short period of providing a new incompatible API storaged was changed to provide an extended version of the udisks2 API. So it can be used as a drop-in replacement of udisks2 with no changes required in the existing tools/libraries/daemons/... using the udisks2 API. New features have been being added to the original API in a backwards-compatible way. In fact, starting with Fedora 25 storaged is being used/installed in place of udisks2

ldm

vdev

Console

vcs

/dev/vcs0 
  a character device with major number 7 and minor number 0, usually of mode 0644 and owner root.tty. It refers to the memory of the currently displayed virtual console terminal.

/dev/vcs[1-63]
   character devices for virtual console terminals, they have major number 7 and minor number 1 to 63, usually mode 0644 and owner root.tty.

/dev/vcsa[0-63] 

   the same, but using unsigned shorts (in host byte order) that include attributes, and prefixed with four bytes giving the screen dimensions and cursor position: lines, columns, x, y. (x = y = 0 at the top left corner of the screen.)


pts

/dev/ptmx


netconsole

  • https://wiki.archlinux.org/index.php/Netconsole - a kernel module that sends all kernel log messages (i.e. dmesg) over the network to another computer, without involving user space (e.g. syslogd). Name "netconsole" is a misnomer because it's not really a "console", more like a remote logging service. It can be used either built-in or as a module. Built-in netconsole initializes immediately after NIC cards and will bring up the specified interface as soon as possible. The module is mainly used for capturing kernel panic output from a headless machine, or in other situations where the user space is no more functional.

Storage

See Storage

Graphics

See Graphics, Computer#Video card, GUI

Clock

/dev/rtc*

Most computers have one or more hardware clocks which record the current "wall clock" time. These are called "Real Time Clocks" (RTCs). One of these usually has battery backup power so that it tracks the time even while the computer is turned off. RTCs often provide alarms and other interrupts.



Input

/dev/uinput


  • https://en.wikipedia.org/wiki/evdev - short for 'event device', is a generic input event interface in the Linux kernel. It generalizes raw input events from device drivers and makes them available through character devices in the /dev/input/ directory. The user-space library for the Linux kernel component evdev is called libevdev. Libevdev abstracts the evdev ioctls through type-safe interfaces and provides functions to change the appearance of the device. Libevdev shares similarities with the read system call. It sits below the process that handles input events, in between the kernel and that process. Linux kernel → libevdev → xf86-input-evdev → X server → X client



  • http://hetgrotebos.org/wiki/uinput-mapper - a program that can clone and remap (in clones) input devices to other (newly created) input devices (locally or over the network). It also offers a nice API to both the /dev/input/* and /dev/uinput devices in Python.



  • https://github.com/gvalkov/python-evdev
    • http://python-evdev.readthedocs.io/en/latest - provides bindings to the generic input event interface in Linux. The evdev interface serves the purpose of passing events generated in the kernel directly to userspace through character devices that are typically located in /dev/input/. This package also comes with bindings to uinput, the userspace input subsystem. Uinput allows userspace programs to create and handle input devices that can inject events directly into the input subsystem. In other words, python-evdev allows you to read and write input events on Linux. An event can be a key or button press, a mouse movement or a tap on a touchscreen.




  • https://github.com/Blub/netevent - a tool which can be used to share linux event devices with other machines (either via /dev/uinput or by implementing a client for the same protocol with other means).

File systems

See Storage

Files and directories

See Storage

Home Directories

  • Home Directories - manages home directories of regular (“human”) users. Each directory it manages encapsulates both the data store and the user record of the user so that it comprehensively describes the user account, and is thus naturally portable between systems without any further, external metadata. This document describes the format used by these home directories, in context of the storage mechanism used.

Command-line

See also Typography#Terminal, Terminals

Console

  • https://en.wikipedia.org/wiki/System_console - or computer console, root console, operator's console, or simply console is the text entry and display device for system administration messages, particularly those from the BIOS or boot loader, the kernel, from the init system and from the system logger. It is a physical device consisting of a keyboard and a screen, and traditionally is a text terminal, but may also be a graphical terminal. System consoles are generalized to computer terminals, which are abstracted respectively by virtual consoles and terminal emulators. Today communication with system consoles is generally done abstractly, via the standard streams (stdin, stdout, and stderr), but there may be system-specific interfaces, for example those used by the system kernel.

User space

  • User space - Wikipedia - A modern computer operating system usually segregates virtual memory into kernel space and user space. Primarily, this separation serves to provide memory protection and hardware protection from malicious or errant software behaviour. Kernel space is strictly reserved for running a privileged operating system kernel, kernel extensions, and most device drivers. In contrast, user space is the memory area where application software and some drivers execute.


Busybox

  • Busybox - combines tiny versions of many common UNIX utilities into a single small executable. It provides replacements for most of the utilities you usually find in GNU fileutils, shellutils, etc. The utilities in BusyBox generally have fewer options than their full-featured GNU cousins; however, the options that are included provide the expected functionality and behave very much like their GNU counterparts. BusyBox provides a fairly complete environment for any small or embedded system.

toybox

  • toybox - combines common Linux command line utilities together into a single BSD-licensed executable that's simple, small, fast, reasonably standards-compliant, and powerful enough to turn Android into a development environment. See the links on the left for details.

gokrazy

  • gokrazy - a pure-Go userland for your Raspberry Pi 3 appliances. For a long time, we were unhappy with having to care about security issues and Linux distribution maintenance on our various Raspberry Pis. Then, we had a crazy idea: what if we got rid of memory-unsafe languages and all software we don’t strictly need? Turns out this is feasible. gokrazy is the result. [86] [87]

Login

systemd-logind

  • systemd-logind.service - a system service that manages user logins. It is responsible for: Keeping track of users and sessions, their processes and their idle state. This is implemented by allocating a systemd slice unit for each user below user.slice, and a scope unit below it for each concurrent session of a user. Also, a per-user service manager is started as system service instance of user@.service for each logged in user; Generating and managing session IDs. If auditing is available and an audit session ID is already set for a session, then this ID is reused as the session ID. Otherwise, an independent session counter is used; Providing PolicyKit-based access for users for operations such as system shutdown or sleep; Implementing a shutdown/sleep inhibition logic for applications; Handling of power/sleep hardware keys; Multi-seat management; Session switch management; Device access management for users; Automatic spawning of text logins (gettys) on virtual console activation and user runtime directory management

elogind

  • https://github.com/elogind/elogind - the systemd project's "logind", extracted out to be a standalone daemon. It integrates with PAM to know the set of users that are logged in to a system and whether they are logged in graphically, on the console, or remotely. Elogind exposes this information via the standard org.freedesktop.login1 D-Bus interface, as well as through the file system using systemd's standard /run/systemd layout. Elogind also provides "libelogind", which is a subset of the facilities offered by "libsystemd". There is a "libelogind.pc" pkg-config file as well.

Session

sessiond

  • sessiond - a standalone X session manager that reports the idle status of a session to systemd-logind.service(8) and handles its lock, unlock, sleep, and shutdown signals. sessiond also provides hooks triggered by inactivity or a signal, automatic backlight dimming on idle, and optional management of DPMS settings.

Shell


/etc/shells
  # list of shells installed on system






General

to find;

  • way of making previous command screen output be pushed to a buffer that can be flipped through/forked?
  • paste pwd to readline/zle
  • make alt-delete not break things
Alt-left/right
  move forwards/back one word
  • chsh - change your login shell
    • https://en.wikipedia.org/wiki/Chsh - is a command on Unix-like operating systems that is used to change a login shell. Users can either supply the pathname of the shell that they wish to change to on the command line, or supply no arguments, in which case chsh allows the user to change the shell interactively. chsh is a setuid program that modifies the /etc/passwd file, and only allows ordinary users to modify their own login shells. The superuser can modify the shells of other users, by supplying the name of the user whose shell is to be modified as a command-line argument.


Environment

  • direnv - unclutter your .profile, an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.



Standard streams

  • https://en.wikipedia.org/wiki/Standard_streams - preconnected input and output communication channels[1] between a computer program and its environment when it begins execution. The three I/O connections are called standard input (stdin), standard output (stdout) and standard error (stderr). Originally I/O happened via a physically connected system console (input via keyboard, output via monitor), but standard streams abstract this. When a command is executed via an interactive shell, the streams are typically connected to the text terminal on which the shell is running, but can be changed with redirection, e.g. via a pipeline. More generally, a child process will inherit the standard streams of its parent process.

n the C programming language, the standard input, output, and error streams are attached to the existing Unix file descriptors 0, 1 and 2 respectively. In a POSIX environment the <unistd.h> definitions STDIN_FILENO, STDOUT_FILENO or STDERR_FILENO should be used instead rather than magic numbers. File pointers stdin, stdout, and stderr are also provided.

  • stdin - 0
  • stdout - 1
  • stderr - 2

Echo



Here document

tr a-z A-Z << END_TEXT
  one two three
  four five six
  END_TEXT

This yields the output: 

ONE TWO THREE
FOUR FIVE SIX

Redirection

The > and >> are redirection operators for FD's (File Descriptors)

In bash you have tree standard FD's that are the standard input (strin), the standard output (strout) and the standard error (strerr). These can also be called by FD 0, FD 1 and FD 2 respectively.

Normally you would have you would have all FD's pointing to the terminal, but this can be changed by using redirection.

For example, if you call: 

command > log.txt

You will redirect the output to the file log.txt This is similar as calling: 

command 1> log.txt

As this only redirects strout you will still be able to see the error in the terminal. In order to redirect strerr to you log.txt file you will have to run: 

command 2> log.txt

Again, this only redirects strerr. If you wish to redirect both stdout and stderr you need to duplicate your stderr output to stdout by using the >& command. 

command 1> log.txt 2>&1

To understand this command you need to read it form right to left, first a copy of stderr is made to stdout, then strout is redirected to the log.txt file.

When you use redirection in this way, bash will not look if the file exists or not and simply create one regardless if that implies erasing the existing one. If you want to avoid loosing what has already been written in your log file you can then use the >> command in the same ways explained above, but in this case all the outputs will be appended to existing files.

Readline

Used by bash and other programs. 


bind -p
  # list bindings readline is capable of




  • rlwrap(1) - Linux man page - runs the specified command, intercepting user input in order to provide readline's line editing, persistent history and completion.rlwrap tries to be completely transparent - you (or your shell) shouldn't notice any difference between command and rlwrap command - except the added readline functionality, of course. This should even hold true when you are re-directing, piping and sending signals from and to command, or when command manipulates its terminal settings.


IFS

Wildcards

Patterns

  • http://mywiki.wooledge.org/BashGuide/Patterns - BASH offers three different kinds of pattern matching. Pattern matching serves two roles in the shell: selecting filenames within a directory, or determining whether a string conforms to a desired format. On the command line you will mostly use globs. These are a fairly straight-forward form of patterns that can easily be used to match a range of files, or to check variables against simple rules. The second type of pattern matching involves extended globs, which allow more complicated expressions than regular globs. Since version 3.0, Bash also supports regular expression patterns. These will be useful mainly in scripts to test user input or parse data. (You can't use a regular expression to select filenames; only globs and extended globs can do that.)

Substitution

print \T\h\i\s\ \i\s\ \*\p\o\i\n\t\l\e\s\s\*\ \ 
     \-\ \b\u\t\ \v\a\l\i\d\!

% read string
  This is a *string* with various `special' characters
% print -r -- ${(q)string}
  This\ is\ a\ \*string\*\ with\ various\ \`special\'\ characters


Prompt

  • https://github.com/dolmen/angel-PS1 - your guardian angel for the Unix world. The implementation is a daemon, a background process, that will service your shell every time it needs to display the prompt.


  • https://github.com/starship/starship - minimal, blazing fast, and extremely customizable prompt for any shell!The prompt shows information you need while you're working, while staying sleek and out of the way. Rust.


History


  • https://github.com/cantino/mcfly - replaces your default ctrl-r Bash history search with an intelligent search engine that takes into account your working directory and the context of recently executed commands. McFly's suggestions are prioritized in real time with a small neural network. [96]

Other

  • https://github.com/neurobin/shc - A generic shell script compiler. Shc takes a script, which is specified on the command line and produces C source code. The generated source code is then compiled and linked to produce a stripped binary executable.The compiled binary will still be dependent on the shell specified in the first line of the shell code (i.e shebang) (i.e. #!/bin/sh), thus shc does not create completely independent binaries.




  • https://github.com/leahneukirchen/snooze -tool for waiting until a particular time and then running a command. Together with a service supervision system such as runit, this can be used to replace cron(8).

Thumpson shell / sh

  • https://en.wikipedia.org/wiki/Thompson_shell - the first Unix shell, introduced in the first version of Unix in 1971, and was written by Ken Thompson.[1] It was a simple command interpreter, not designed for scripting, but nonetheless introduced several innovative features to the command-line interface and led to the development of the later Unix shells.

Bourne shel / sh

The Bourne shell, sh, was written by Stephen Bourne at AT&T as the original Unix command line interpreter; it introduced the basic features common to all the Unix shells, including piping, here documents, command substitution, variables, control structures for condition-testing and looping and filename wildcarding. The language, including the use of a reversed keyword to mark the end of a block, was influenced by ALGOL 68.



bash









man: echo



Scripting



#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

set -e
  # exit if any command finishes with non 0 exit code
set -u
  # exit if undefined variable is used
set -x
  # causes bash to print each command before executing it
set -o pipefail
  # prevents errors in a pipeline from being masked
IFS=$'\n\t'
  # word splitting will happen only on newlines and tab characters

Maths

a=$(( 5 + 3 ))

(( a = 23 ))

(( ++a ))

(( a++ ))


Options


Completion


Prompt


Aliases

\curl
  # bypass any "curl" alias [103]


Variables


  • declare or typeset - builtins, they are exact synonyms, permit modifying the properties of variables. This is a very weak form of the typing available in certain programming languages. The declare command is specific to version 2 or later of Bash. The typeset command also works in ksh scripts.


VARIABLE=test
VARIABLE="test one two three"


export VARIABLE="make global to current shell env"


TZ=:/etc/localtime
  # read in from file



for x in {1..9}; do echo mv myfile$x other_dir/$x.txt; done

Special Parameters


$_
  last word of previous line


Etc.






Trap


Other

  • https://github.com/niieani/bash-oo-framework - a standard library and a boilerplate framework for writing tools using bash. It's modular and lightweight, while managing to implement some concepts from C#, Java or JavaScript into bash. The Infinity Framework is also plug & play: include it at the beginning of your existing script to import any of the individual features such as error handling, and start using other features gradually. The aim of Bash Infinity is to maximize readability of bash scripts, minimize the amount of code repeat and create a central repository for well-written, and a well-tested standard library for bash. [108]


  • https://github.com/dylanaraps/bush - an experiment to see how many standard tools and functions we can re-implement in pure bash. No external processes are used, only shell builtins. Bush is not meant to be used for anything serious and there's probably edge cases and bugs throughout.


  • https://github.com/dylanaraps/pure-bash-bible - goal of this book is to document known and unknown methods of doing various tasks using only built-in bash features. Using the snippets from this bible can help remove unneeded dependencies from scripts and in most cases make them faster.


Testing


zsh



Configuration

  • zbkrc README file - This file describes startup files for Z-shell (zsh), bash and ksh

Commands are then read from $ZDOTDIR/.zshenv. If the shell is a login shell, commands are read from /etc/zprofile and then $ZDOTDIR/.zprofile. Then, if the shell is interactive, commands are read from /etc/zshrc and then $ZDOTDIR/.zshrc. Finally, if the shell is a login shell, /etc/zlogin and $ZDOTDIR/.zlogin are read.



Variables

ZLE

The ZSH equivalent of readline for bash. 


bindkey "\e[A" history-beginning-search-backward
bindkey "\e[B" history-beginning-search-forward

Expansion

$_
  last entered word

$?
  returned exit code of last exec

Prompt





reset-prompt-and-accept-line() {
    zle reset-prompt
    zle accept-line
}

zle -N reset-prompt-and-accept-line

bindkey '^m' reset-prompt-and-accept-line







Completion

setopt menu_complete
  # auto select first completion option

compdef g=git
  # basic alias completion [110]

compdef _files my-local-python-script
  # simple _files completion [111]

compdef '_files -g "*.(eps|ps|pdf)"' okular
  # restrict to some file extensions

compdef -e 'words[1]=(git pull origin); service=git; (( CURRENT+=2 )); _git' ggl
  # make alias use existing completion [112]


From http://news.ycombinator.com/item?id=4867369 :

Tab completion and aliases need not be mutually exclusive. For instance, in zsh you can use: 

alias gs='git status'
compdef _git gs=git-status

FYI: I also mapped g without parameters to `git status -sb` but g with parameters will simply execute everything as normal 

g () {
  if [ $# -eq 0 ]
  then
    git status -sb
  else
    git $*
  fi
}
compdef g=git


Regex

autoload -U regexp-replace

regexp-replace VARNAME REGEXP REPLACE

PATH="$PATH:/foo/bar"
echo $PATH
regexp-replace PATH ':/foo/bar'


Plugins



  • Antigen - A plugin manager for zsh, inspired by oh-my-zsh and vundle. Antigen is a small set of functions that help you easily manage your shell (zsh) plugins, called bundles. The concept is pretty much the same as bundles in a typical vim+pathogen setup. Antigen is to zsh, what Vundle is to vim.





  • https://github.com/marlonrichert/zsh-snap - the light-weight plugin manager & Git repo manager for Zsh that's easy to grok. While tailored for Zsh plugins specifically, Znap also functions as a general-pupose utility for managing Git repos.



  • https://github.com/oknowton/zsh-dwim - attempts to predict what you will want to do next. It provides a key binding (control-u) that will replace the current (or previous) command line with the command you will want to run next.


  • zsh-history-substring-search - This is a clean-room implementation of the Fish shell's history search feature, where you can type in any part of any previously entered command and press the UP and DOWN arrow keys to cycle through the matching commands. You can also use K and J in VI mode or ^P and ^N in EMACS mode for the same.


  • https://github.com/zsh-users/zsh-completions - Additional completion definitions for Zsh. This projects aims at gathering/developing new completion scripts that are not available in Zsh yet. The scripts are meant to be contributed to the Zsh project when stable enough.



  • https://github.com/zsh-users/fizsh - the Friendly Interactive ZSHell. It is a front end to ZSH. It provides the user of ZSH with interactive syntax-highting and Matlab-like history search. It also has a both short and informative prompt.




  • https://github.com/rimraf/k - a zsh script / plugin to make directory listings more readable, adding a bit of color and some git status information on files and directories.





Themes

Frameworks


  • Prezto is the configuration framework for Zsh; it enriches the command line interface environment with sane defaults, aliases, functions, auto completion, and prompt themes.


  • https://github.com/romkatv/zsh4humans - A turnkey configuration for Z shell that aims to work really well out of the box. It combines the best Zsh plugins into a coherent whole that feels like a finished product rather than a DIY starter kit.

Dotfiles

ash / dash

  • https://en.wikipedia.org/wiki/Almquist_shell - The Almquist shell (also known as A Shell, ash and sh) is a lightweight Unix shell originally written by Kenneth Almquist in the late 1980s. Initially a clone of the System V.4 variant of the Bourne shell, it replaced the original Bourne shell in the BSD versions of Unix released in the early 1990s. Derivative versions of ash are still installed as the default shell (/bin/sh) on FreeBSD, NetBSD, DragonFly BSD, MINIX, and Android, and in some Linux distributions.

Debian and derived Linux distributions such as Ubuntu ship a version of ash, known as dash (Debian Almquist shell), as the default /bin/sh, although Bash is the default login shell for interactive use. The reason for using dash is faster shell script execution, especially during startup of the operating system, compared to previous versions of Debian and Ubuntu that used Bash for this purpose.

Ash is also fairly popular in embedded Linux systems; its code was incorporated into the BusyBox catch-all executable often employed in this area.

csh

  • https://en.wikipedia.org/wiki/C_shell - a Unix shell created by Bill Joy while he was a graduate student at University of California, Berkeley in the late 1970s. It has been widely distributed, beginning with the 2BSD release of the Berkeley Software Distribution (BSD) which Joy first distributed in 1978. Other early contributors to the ideas or the code were Michael Ubell, Eric Allman, Mike O'Brien and Jim Kulp. The C shell is a command processor typically run in a text window, allowing the user to type commands. The C shell can also read commands from a file, called a script. Like all Unix shells, it supports filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration. What differentiated the C shell from others, especially in the 1980s, were its interactive features and overall style. Its new features made it easier and faster to use. The overall style of the language looked more like C and was seen as more readable.On many systems, such as macOS and Red Hat Linux, csh is actually tcsh, an improved version of csh. Often one of the two files is either a hard link or a symbolic link to the other, so that either name refers to the same improved version of the C shell.


  • An Introduction to the C shell - William Joy, revised for 4.3BSD by Mark Seiden - a new command language interpreter for UNIXsystems. It incorporates good features of other shells and a history mechanism similar to the redo of INTERLISP. While incorporating many features of other shells which make writing shell programs (shell scripts) easier, most of the features unique to csh are designed more for the interactive UNIX user. UNIX users who have read a general introduction to the system will find a valuable basic explanation of the shell here. Simple terminal interaction with csh is possible after reading just the first section of this document. The second section describes the shell's capabilities which you can explore after you have begun to become acquainted with the shell. Later sections introduce features which are useful, but not necessary for all users of the shell. Additional information includes an appendix listing special characters of the shell and a glossary of terms and commands introduced in this manual.




tcsh

  • https://en.wikipedia.org/wiki/tcsh - a Unix shell based on and compatible with the C shell (csh). It is essentially the C shell with programmable command-line completion, command-line editing, and a few other features. Unlike the other common shells, functions cannot be defined in a tcsh script and the user must use aliases instead (as in csh). It is the native root shell for BSD-based systems such as FreeBSD.tcsh added filename and command completion and command line editing concepts borrowed from the Tenex system, which is the source of the “t”. Because it only added functionality and did not change what was there, tcsh remained backward compatible with the original C shell. Though it started as a side branch from the original csh source tree that Bill Joy had created, tcsh is now the main branch for ongoing development. tcsh is very stable but new releases continue to appear roughly once a year, consisting mostly of minor bug fixes.

Korn

rc

lshell

  • lshell is a shell coded in Python, that lets you restrict a user's environment to limited sets of commands, choose to enable/disable any command over SSH (e.g. SCP, SFTP, rsync, etc.), log user's commands, implement timing restriction, and more.

git-shell

  • git-shell - Restricted login shell for Git-only SSH access

Inferno

  • Inferno is a distributed operating system, originally developed at Bell Labs, but now developed and maintained by Vita Nuova® as Free Software. Applications written in Inferno's concurrent programming language, Limbo, are compiled to its portable virtual machine code (Dis), to run anywhere on a network in the portable environment that Inferno provides. Unusually, that environment looks and acts like a complete operating system.

fish

fishfish

oil



mrsh

  • https://git.sr.ht/~emersion/mrsh - A minimal POSIX shell. POSIX compliant, no less, no more. Simple, readable code without magic. Library to build more elaborate shells

promptless

  • https://github.com/dylanaraps/promptless - A super fast and extremely minimal shell prompt. promptless follows the suckless philosophy and is simple by design. The code-base has a focus on elegance and clarity. The prompt is entirely hackable. Unnecessary and unworthy features need to be patched in.

dash

Xiki

  • Xiki: A shell console with GUI features. Xiki does what shell consoles do, but lets you edit everything at any time. It's trivial to make your own commands and menus to access other tools. [115]

Pdsh

  • Pdsh is a high-performance, parallel remote shell utility. It uses a sliding window of threads to execute remote commands, conserving socket resources while allowing some connections to timeout if needed. It was originally written as a replacement for IBM's DSH on clusters at LLNL.

shok

  • shok is a cross-platform non-POSIX interactive command shell. Similar to bash and cmd.exe, it allows the user to invoke programs and script their execution. shok will provide a secure, platform-agnostic, and user-friendly programming language that is designed to facilitate filesystem interaction and system management. [116]

Es

Scsh

psh

  • Perl Shell (psh) combines aspects of bash and other shells with the power of Perl scripting.

Elvish

Oh

  • Oh is a Unix shell. If you've used other Unix shells, oh should feel familiar. At its core, oh is a heavily modified dialect of the Scheme programming language complete with first-class continuations and proper tail recursion. Like early Scheme implementations, oh exposes environments as first-class values. Oh extends environments to allow both public and private members and uses these extended first-class environments as the basis for its prototype-based object system. Written in Go, oh is also a concurrent programming language. It exposes channels, in addition to pipes, as first-class values. As oh uses the same syntax for code and data, channels and pipes can, in many cases, be used interchangeably. This homoiconic nature also allows oh to support fexprs which, in turn, allow oh to be easily extended. In fact, much of oh is written in oh. [120]

xonsh

  • xonsh - a Python-ish, BASHwards-looking shell language and command prompt. The language is a superset of Python 3.4+ with additional shell primitives that you are used to from Bash and IPython. It works on all major systems including Linux, Mac OSX, and Windows. Xonsh is meant for the daily use of experts and novices alike. [121]

ngs

  • ngs - a Next Generation Shell. It has two main parts: the language and the interactive shell. NGS tries to fill the void between classical shells such as bash and general-purpose programming languages such as Ruby, Python, Perl, Go. The shells are domain-specific languages but the domain has changed so classical shells are not optimal for today's tasks. General-purpose languages on the other hand are not domain-specific so they are not good as shells and too verbose for system tasks scripting, not a good fit. [122]

cicada

viewglob

  • http://viewglob.sourceforge.net/ - a filesystem visualization add-on for Bash and Zsh. It tracks the command line and environment of any number of interactive shells (local and remote). A graphical display follows the currently active terminal, listing the contents of directories relevant to its shell and highlighting file selections and potential name completions dynamically.

dgsh

funky


janetsh


nushell

TopShell

qsh

Crush

Dune

janetsh

Commands


compgen -c | sort
  # list all commands

SSH

See SSH

General

  • Black Duck Open Hub (formerly Ohloh.net) is an online community and public directory of free and open source software (FOSS), offering analytics and search services for discovering, evaluating, tracking, and comparing open source code and projects. Open Hub Code Search is free code search engine indexing over 21,000,000,000 lines of open source code from projects on the Black Duck Open Hub.


  • fu - commandlinefu.com from the commandline. Nothing has made more sense!









exec

  • exec - shall open, close, and/or copy file descriptors as specified by any redirections as part of the command. If exec is specified without command or arguments, and any file descriptors with numbers greater than 2 are opened with associated redirection statements, it is unspecified whether those file descriptors remain open when the shell invokes another utility. Scripts concerned that child shells could misuse open file descriptors can always close them explicitly, as shown in one of the following examples. If exec is specified with command, it shall replace the shell with command without creating a new process. If arguments are specified, they shall be arguments to command. Redirection affects the current shell execution environment.
exec windowmanager
  # switch script shell process with windowmanager process


Daemons

A daemon is a program that runs in the background, waiting for events to occur and offering services. A good example is a web server that waits for a request to deliver a page or a ssh server waiting for someone trying to log in. While these are full featured applications, there are daemons whose work is not that visible. Daemons are for tasks like writing messages into a log file (e.g. syslog, metalog) or keeping your system time accurate (e.g. ntpd). For more information see man 7 daemon.

  • Daemon turns other process into daemons. There are many tasks that need to be performed to correctly set up a daemon process. This can be tedious. Daemon performs these tasks for other processes.

Programs

See also Regex for sed, awk, etc.

  • Vim - Text editor, etc.
  • Git - Distributed version control






GNU coreutils

cat

Use output as input. 

cat filename.txt

cat `locate file.txt`
  # or
cat $(locate file.txt)





tee
sponge
  • sponge(1) - soak up standard input and write to a file
basename
uniq
od

printf

tr

echo $PATH | tr \: \\n | sort

Diff




to sort

  • auditd - userspace component to the Linux Auditing System [136]


  • pp - a preprocessorpp(1). allows embedding sh(1) code in files of any type by nesting it inside the #!\n token, where \n is a new line. That means that if you'd like a simple loop or an if inside an HTML file for instance, you could use pp(1).




  • cut - remove sections from each line of files
  • shelr - console screencasting tool


  • https://github.com/colinhowe/monner - Allows you to monitor the CPU, memory and network usage when running a program. Output is tab-separated for easy loading into spreadsheet programs.


If you want to write daemons in languages that can link against C functions (e.g. C, C++), see libslack which contains the core functionality of daemon.



Expect

  • Expect - a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. Expect is also useful for testing these same applications. And by adding Tk, you can also wrap interactive applications in X11 GUIs. Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily.

Calculator

expr 2 \* 2
  # returns 4. doesn't do non-integers.

Terminal extras

Desktop entries


  • gendesk - Utility for generating .desktop files and download icons by specifying a minimum of information




Job schedulers

https://en.wikipedia.org/wiki/Job_scheduler

at


at and batch read commands from standard input or a specified file which are to be executed at a later time. 

at 5am Oct 20   
  # executes commands at "time am/pm" "month" "day"
atq
  # lists the user's pending jobs, unless the user is the superuser; in that case, everybody's jobs are listed. The format of the output lines (one for each job) is: Job number, date, hour, queue, and username.
atrm
  # deletes jobs, identified by their job number.
batch
  # executes commands when system load levels permit; in other words, when the load average drops below 0.8, or the value specified in the invocation of atd.

/var/spool/at/

cron

  • SUS: crontab - schedule periodic background work

A cronjob is a task that a Cron system is instructed to run periodically. The crontab file is a configuration file for a user that defines tasks to run under the user’s account. The systab file is a file that specifies cronjobs for the system.

Configuration
/var/spool/cron/
  # user crontabs

/etc/crontab
  # system-wide crontab

/etc/cron.d/*
  # or
/etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly
  # distro specific extra



crontab -l
  # view crontabs

crontab -e
  # edit their crontabs

crontab -r
 # remove their crontabs

crontab saved_crontab_filename
 # overwrite their old crontab with saved crontab


  1. m h dom mon dow user command

There are several special predefined values which can be used to substitute the CRON expression. 

Entry                  Description                                 Equivalent To
 @yearly (or @annually) Run once a year, midnight, Jan. 1st         0 0 1 1 *
 @monthly               Run once a month, midnight, first of month  0 0 1 * *
 @weekly                Run once a week, midnight on Sunday         0 0 * * 0
 @daily                 Run once a day, midnight                    0 0 * * *
 @hourly                Run once an hour, beginning of hour         0 * * * *
 @reboot                Run at startup                              @reboot 

*    *    *    *    *  command to be executed
┬    ┬    ┬    ┬    ┬
│    │    │    │    │
│    │    │    │    │
│    │    │    │    └───── day of week (0 - 6) (0 is Sunday, or use names)
│    │    │    └────────── month (1 - 12)
│    │    └─────────────── day of month (1 - 31)
│    └──────────────────── hour (0 - 23)
└───────────────────────── min (0 - 59)

0 0 * */2 * certbot certonly --webroot -n -w /var/www/example.org/ -d example.org
# “At 00:00 in every 2nd month.”


  • /etc/cron.allow - If this file exists, then you must be listed therein (your username must be listed) in order to be allowed to use cron jobs.
  • /etc/cron.deny - If the cron.allow file does not exist but the /etc/cron.deny file does exist, then you must not be listed in the /etc/cron.deny file in order to use cron jobs.
dcron

Vanilla dcron supports asynchronous job processing. Just put it with @hourly, @daily, @weekly or @monthly with a jobname.

dcron has @daily, @hourly and so on. In fcron, you can use standard crontab entries and add &bootrun to the beginning of the line to repeat "missed" cronjobs.

cronie

Arch default cron as dcron wasn't maintained at the time.

dcron and fcron are not under active development, cronie is. cronie is small - 0.20MB installed. developed by Red Hat - it is not going anywhere and we have a guaranteed upgrade path. As far as I can tell cronie has no deps beyond glibc and pam. cronie has /etc/cron.d support and anacron support via an anacrontab config file. cronie extends the original vixie cron package so the syntax, core feature set, etc are stable. implements advanced security hooks as well and can integrate with SELINUX.

anacron
  • anacron is a computer program that performs periodic command scheduling which is traditionally done by cron, but without assuming that the system is running continuously. Thus, it can be used to control the execution of daily, weekly, and monthly jobs (or anything with a period of n days) on systems that don't run 24 hours a day. 


/etc/anacrontab
  # and/or
/var/spool/anacron/*
fcron

Fcron’s own crontab system uses the fcrontab file for configuration information. The fcrontab syntax is similar but differs slightly from the classic Vixie/ISC Cron crontab notation. fcron lacks /etc/cron.d/ functionality.

Mcron
  • Mcron - a 100% compatible replacement for Vixie cron. It is written in pure Guile, and allows configuration files to be written in scheme (as well as Vixie's original format) for infinite flexibility in specifying when jobs should be run.
Other

systemd/Timers

  • https://wiki.archlinux.org/index.php/Systemd/Timers - Timers are systemd unit files whose name ends in .timer that control .service files or events. Timers can be used as an alternative to cron (read #As a cron replacement). Timers have built-in support for calendar time events, monotonic time events, and can be run asynchronously.

Batch system

  • https://en.wikipedia.org/wiki/Job_scheduler - a computer application for controlling unattended background program execution of jobs. This is commonly called batch scheduling, as execution of non-interactive jobs is often called batch processing, though traditional job and batch are distinguished and contrasted; see that page for details. Other synonyms include batch system, distributed resource management system (DRMS), distributed resource manager (DRM), and, commonly today, workload automation. The data structure of jobs to run is known as the job queue.

batch

Task Spooler

  • Task Spooler - a Unix batch system where the tasks spooled run one after the other. The amount of jobs to run at once can be set at any time. Each user in each system has his own job queue. The tasks are run in the correct context (that of enqueue) from any shell/process, and its output/results can be easily watched. It is very useful when you know that your commands depend on a lot of RAM, a lot of disk use, give a lot of output, or for whatever reason it's better not to run them all at the same time, while you want to keep your resources busy for maximum benfit. Its interface allows using it easily in scripts.

nq

  • https://github.com/chneukirchen/nq - These small utilities allow creating very lightweight job queue systems which require no setup, maintenance, supervision, or any long-running processes.

Slurm

TORQUE

pueue

File watchers

inotify

  • inotify-tools is a C library and a set of command-line programs for Linux providing a simple interface to inotify.
  • inotifywatch - gather filesystem access statistics using inotify
  • inotail is a replacement for the 'tail' program found in the base installation of every Linux/UNIX system. It makes use of the inotify infrastructure in recent versions of the Linux kernel to speed up tailing files in the follow mode (the '-f' option). Standard tail polls the file every second by default while inotail listens to special events sent by the kernel through the inotify API to determine whether a file needs to be reread. Currently inotail is not fully compatible to neither POSIX or GNU tail but might be in the future.
  • entr - Run arbitrary commands when files change
  • - Guard::Shell - automatically runs shell commands when watched files are modified.
  • incron - inotify cron system. This program is an "inotify cron" system. It consists of a daemon and a table manipulator. You can use it a similar way as the regular cron. The difference is that the inotify cron handles filesystem events rather than time periods.
  • nodemon will watch the files in the directory that nodemon was started, and if they change, it will automatically restart your node application.

Other

watch -n 2 ls -la





  • fswatch - A cross-platform file change monitor with multiple backends: Apple OS X File System Events API, *BSD kqueue, Solaris/Illumos File Events Notification, Linux inotify and a stat()-based backend.

chroot

Startup

  • e4rat ("Ext4 - Reducing Access Times") is a toolset to accelerate the boot process as well as application startups. Through physical file realloction e4rat eliminates both seek times and rotational delays. This leads to a high disk transfer rate. Placing files on disk in a sequentially ordered way allows to efficiently read-ahead files in parallel to the program startup.

Formatting


ncurses, etc.

See UI#TUI

CLI dashboard

  • WTF - a personal information dashboard for your terminal, developed for those who spend most of their day in the command line.It allows you to monitor systems, services, and important information that you otherwise might keep browser tabs open for, the kinds of things you don’t always need visible, but do check in on every now and then. [146]



Web


Configuration management

/etc

~/.config

/usr/local/etc/

etc.

See Dotfiles


Media

See Media, Playback

Misc.

fzf


fzy

  • https://github.com/jhawthorn/fzy - a simple, fast fuzzy finder for the terminal, fzy is faster and shows better results than other fuzzy finders. Most other fuzzy matchers sort based on the length of a match. fzy tries to find the result the user intended. It does this by favouring matches on consecutive letters and starts of words. This allows matching using acronyms or different parts of the path.


skim

pico

  • https://github.com/peco/peco - based on a python tool, percol. percol was darn useful, but I wanted a tool that was a single binary, and forget about python. peco is written in Go, and therefore you can just grab the binary releases and drop it in your $PATH. peco can be a great tool to filter stuff like logs, process stats, find files, because unlike grep, you can type as you think and look through the current results.


tosort


  • https://github.com/ysf/anewer - anewer appends lines from stdin to a file if they don't already exist in the file. This is a rust version of anew


  • https://github.com/p-gen/smenu - Terminal utility that allows you to use words coming from the standard input to create a nice selection window just below the cursor. Once done, your selection will be sent to standard output.


  • https://github.com/akavel/up - a tool for writing Linux pipes in a terminal-based UI interactively, with instant live preview of command results. The main goal of the Ultimate Plumber is to help interactively and incrementally explore textual data in Linux, by making it easier to quickly build complex pipelines, thanks to a fast feedback loop. This is achieved by boosting any typical Linux text-processing utils such as grep, sort, cut, paste, awk, wc, perl, etc., etc., by providing a quick, interactive, scrollable preview of their results. [147] [148]



https://github.com/zsh-users/zaw

Cute

See also GUI#Utils 



strfile -c % myfortunes myfortunes.dat




  • https://gitlab.com/jallbrit/cbonsai - a bonsai tree generator, written in C using ncurses. It intelligently creates, colors, and positions a bonsai tree, and is entirely configurable via CLI options-- see usage. There are 2 modes of operation: static (the default), and live.













hexdump -C /dev/urandom | pv -q -L 2k | grep "ca fe"








  • https://github.com/svenstaro/genact - Pretend to be busy or waiting for your computer when you should actually be doing real work! Impress people with your insane multitasking skills. Just open a few instances of genact and watch the show. genact has multiple scenes that pretend to be doing something exciting or useful when in reality nothing is happening at all.




  • https://github.com/OGoodness/Minesweeper-Login - Minesweeper that cannot be exited in any way other than completing the game. Make it run while logging in and it will work as a second form of "auth". Used as a joke for Cyber Team --- Bypass with code - 6969420

Windows

  • Wine - (originally an acronym for "Wine Is Not an Emulator") is a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems, such as Linux, macOS, & BSD. Instead of simulating internal Windows logic like a virtual machine or emulator, Wine translates Windows API calls into POSIX calls on-the-fly, eliminating the performance and memory penalties of other methods and allowing you to cleanly integrate Windows applications into your desktop.


Install a .msi 

wine msiexec /i xyz.msi



wineboot --update
  # speed up wine start time



  • ProtonDB - a new tool released by Valve Software that has been integrated with Steam Play to make playing Windows games on Linux as simple as hitting the Play button within Steam. Underneath the hood, Proton comprises other popular tools like Wine and DXVK among others that a gamer would otherwise have to install and maintain themselves. This greatly eases the burden for users to switch to Linux without having to learn the underlying systems or losing access to a large part of their library of games. Proton is still in its infancy so support is inconsistent, but regularly improving. [151]



OS X

Users


  • https://en.wikipedia.org/wiki/Name_Service_Switch - a facility in Unix-like operating systems that provides a variety of sources for common configuration databases and name resolution mechanisms. These sources include local operating system files (such as /etc/passwd, /etc/group, and /etc/hosts), the Domain Name System (DNS), the Network Information Service (NIS), and LDAP

Home directory

On Unix-like operating systems (including BSD, GNU/Linux and Mac OS X), tilde often indicates the current user's home directory: for example, if the current user's home directory is /home/bloggsj, then cd, cd ~, cd /home/bloggsj or cd $HOME are equivalent. This practice derives from the Lear-Siegler ADM-3A terminal in common use during the 1970s, which happened to have the tilde symbol and the word "Home" (for moving the cursor to the upper left) on the same key.

Files

Commands

useradd username
  create user with defaults (no password)

useradd -D
  show defaults that will be used

useradd -m -g [initial_group] -G [additional_groups] -s [login_shell] [username]
  -m - add home dir
  -d - non-default home dir path
  -c comment
  -e 2006-06-30 - expires

useradd -m -g users -G audio,lp,optical,storage,video,wheel,games,power,scanner -s /bin/bash username

adduser
  interactive tool

passwd
  # set password of current account

passwd username
  # set password of username

passwd -S username
  # see password status of username

userdel username
  # remove user

userdel -r username
  # remove user plus their home folder, mail spool

cat /etc/passwd | cut -d: -f 1,3,6 | grep "[5-9][0-9][0-9]"
  # List users on a system [153]

groups [username]
  # show users groups

Groups

groupadd [group]
  add group

gpasswd -a [user] [group]
  add user to group
gpasswd -d [user] [group]
  delete user from group

nix doesn't check the users groups, it checks the applications groups, so the shell doesn't automatically have the new group permission attribute. running "sudo login" or "newgrp [groupname]" gives a shell with the new group attribute. 


id [username]
  show user id and their groups ids

last
  lastb - show listing of last logged in users

Security

See also Security, Storage#Permissions


  • https://en.wikipedia.org/wiki/Setuid - setuid and setgid (short for "set user ID upon execution" and "set group ID upon execution", respectively) are Unix access rights flags that allow users to run an executable with the permissions of the executable's owner or group respectively and to change behaviour in directories. They are often used to allow users on a computer system to run programs with temporarily elevated privileges in order to perform a specific task. While the assumed user id or group id privileges provided are not always elevated, at a minimum they are specific. setuid and setgid are needed for tasks that require higher privileges than those which common users have, such as changing their login password. Some of the tasks that require elevated privileges may not immediately be obvious, though — such as the ping command, which must send and listen for control packets on a network interface.


su / sudo

Run things as root or another user. There are a number of methods to do higher privilege things, with differing resultant permissions. To refind good forum threads.. 

sudo -s
  open a new shell as root, but with your original user's environment variables

sudo -i
  open a new shell as if root had logged in; root's environment variables, root's .bash_profile, etc.

sudo su -
  same as above

snippet to source... 

sudo sh -c "echo 'something' >> /etc/privilegedfile"



sudo bash -c 'sleep 4h; command'
  # run something later
sudo bash -c 'sleep 4h; sudo -u dessert whoami; whoami'
  # run something later as another user

echo 'Defaults passwd_timeout=0' > "$(CreateFile /etc/sudoers.d/passwd_timeout  440)"

sudoers

visudo
  command to safely add sudoers rights using nano

user ALL=(ALL) ALL

doas

  • https://github.com/slicer69/doas - The doas utility is a program originally written for OpenBSD which allows a user to run a command as though they were another user. Typically doas is used to allow non-privleged users to run commands as though they were the root user. The doas program acts as an alternative to sudo, which is a popular method in the Linux community for granting admin access to specific users.

Other

Polkit

For non-root mount permissions, etc, i.e. to make USB drives, etc. accessible too non-root users.


  • polkit: polkit Reference Manual - provides an authorization API intended to be used by privileged programs (“MECHANISMS”) offering service to unprivileged programs (“SUBJECTS”) often through some form of inter-process communication mechanism. In this scenario, the mechanism typically treats the subject as untrusted. For every request from a subject, the mechanism needs to determine if the request is authorized or if it should refuse to service the subject. Using the polkit APIs, a mechanism can offload this decision to a trusted party: The polkit authority.

The polkit authority is implemented as an system daemon, polkitd(8), which itself has little privilege as it is running as the polkitd system user. Mechanisms, subjects and authentication agents communicate with the authority using the system message bus. In addition to acting as an authority, polkit allows users to obtain temporary authorization through authenticating either an administrative user or the owner of the session the client belongs to. This is useful for scenarios where a mechanism needs to verify that the operator of the system really is the user or really is an administrative user.


PAM

  • https://en.wikipedia.org/wiki/Pluggable_authentication_module - PAM) is a mechanism to integrate multiple low-level authentication schemes into a high-level application programming interface (API). It allows programs that rely on authentication to be written independently of the underlying authentication scheme. PAM was first proposed by Sun Microsystems in an Open Software Foundation Request for Comments (RFC) 86.0 dated October 1995. It was adopted as the authentication framework of the Common Desktop Environment. As a stand-alone open-source infrastructure, PAM first appeared in Red Hat Linux 3.0.4 in August 1996. PAM is currently supported in the AIX operating system, DragonFly BSD,[1] FreeBSD, HP-UX, Linux, Mac OS X, NetBSD and Solaris. Since no central standard of PAM behavior exists, there was a later attempt to standardize PAM as part of the X/Open UNIX standardization process, resulting in the X/Open Single Sign-on (XSSO) standard. This standard was not ratified, but the standard draft has served as a reference point for later PAM implementations (for example, OpenPAM).



Linux PAM

  • https://en.wikipedia.org/wiki/Linux_PAM - provide dynamic authentication support for applications and services in a Linux or GNU/kFreeBSD[1] system. Linux PAM is evolved from the Unix Pluggable Authentication Modules architecture.

Linux-PAM separates the tasks of authentication into four independent management groups:

  • account modules check that the specified account is a valid authentication target under current conditions. This may include conditions like account expiration, time of day, and that the user has access to the requested service.
  • authentication modules verify the user's identity, for example by requesting and checking a password or other secret. They may also pass authentication information on to other systems like a keyring.
  • password modules are responsible for updating passwords, and are generally coupled to modules employed in the authentication step. They may also be used to enforce strong passwords.
  • session modules define actions that are performed at the beginning and end of sessions. A session starts after the user has successfully authenticated.

Config format: 

module_type control_flag module_path arguments

Configuration file(s): 

/etc/pam.conf
  # or
/etc/pam.d/*

pam_access.so
  # restricts hosts

pam_console.so
  # privileges for users logged on via the console

pam_listfile.so
  # checks a file for a users privileges

pam_nologin.so
  # prevents users other than root when /etc/nologin exists

pam_securetty.so
  # restricts tty devices root can use

pam_time.so
  # restrict use availability times


  • https://github.com/mcdope/pam_usb - provides hardware authentication for Linux using ordinary USB Flash Drives. It works with any application supporting PAM, such as su and login managers (GDM, KDM).


  • https://github.com/nuvious/pam-duress - A Pluggable Authentication Module (PAM) which allows the establishment of alternate passwords that can be used to perform actions to clear sensitive data, notify IT/Security staff, close off sensitive network connections, etc if a user is coerced into giving a threat actor a password.


OpenPAM

  • https://en.wikipedia.org/wiki/OpenPAM - a BSD-licensed implementation of PAM used by FreeBSD, NetBSD, DragonFly BSD and OS X (starting with Snow Leopard), and offered as an alternative to Linux PAM in certain Linux distributions.

Packages and package management

See Packages

Time


echo "Today's date is `date -I`"

date
  Local datetime
date -d @1337000000
  Convert unixtime to datetime
date -ud
  Convert unixtime to UTC datetime



ntpdate pool.ntp.org && hwclock --systohc && hwclock --adjust
  Synchronize both your system clock and hardware clock and calculate/adjust time drift.
  Do not run this command if you already have ntpd running! [155]


Suspension, hibernation


  • https://wiki.archlinux.org/index.php/Pm-utils - a collection of shell scripts that wrap the kernel mode suspend/resume with the various hacks. These hacks are needed to work around bugs in drivers and subsystems that are not yet aware of suspend. It is easily extensible by putting custom hooks into a directory, which can either be done by the system administrator or those hooks can be part of a package, especially if this package needs special attention during a system suspend or power state transition.
sudo pm-suspend

/etc/sudoers 

username  ALL = NOPASSWD: /usr/sbin/pm-hibernate
username  ALL = NOPASSWD: /usr/sbin/pm-suspend

%group   ALL = NOPASSWD: /usr/sbin/pm-hibernate
%group   ALL = NOPASSWD: /usr/sbin/pm-suspend






  • https://autosuspend.readthedocs.io/en/3.0/ -a daemon that periodically suspends a system on inactivity and wakes it up again automatically in case it is needed. For this purpose, autosuspend periodically iterates a number of user-configurable activity checks, which indicate whether an activity on the host is currently present that should prevent the host from suspending. In case one of the checks indicates such activity, no action is taken and periodic checking continues. Otherwise, in case no activity can be detected, this state needs to be present for a specified amount of time before the host is suspended by autosuspend. In addition to the activity checks, wake up checks are used to determine planned future activities of the system (for instance, a TV recording or a periodic backup). In case such activities are known before suspending, autosuspend triggers a command to wake up the system automatically before the soonest activity.

Cleanup



Laptop

See also Computer#Laptop

Printing

CUPS

Printing system. 


lp filename
lpr filename

lpstat -p -d

lp -d printer filename
lpr -P printer filename

lpoptions -d printer

program | lp
program | lp -d printer
program | lpr
program | lpr -P printer

Overclocking

  • https://github.com/Lurkki14/tuxclocker - Qt5 overclocking tool. Currently supported cards are nvidia 600-series cards and newer, and AMD GPUs using the amdgpu driver until (not including) Radeon VII.



to sort

to move to Distros?



8-bit


Web

Retrieved from "https://wiki.thingsandstuff.org/index.php?title=*nix&oldid=58587"