CLI

From Things and Stuff Wiki
Jump to navigation Jump to search


General

See also Typography#Terminal, Terminals, TUI, *nix


  • https://en.wikipedia.org/wiki/In_the_Beginning..._Was_the_Command_Line - an essay by Neal Stephenson which was originally published online in 1999 and later made available in book form (November 1999, ISBN 978-0380815937). The essay is a commentary on why the proprietary operating systems business is unlikely to remain profitable in the future because of competition from free software. It also analyzes the corporate/collective culture of the Microsoft, Apple Computer, and free software communities.


Console

  • https://en.wikipedia.org/wiki/System_console - or computer console, root console, operator's console, or simply console is the text entry and display device for system administration messages, particularly those from the BIOS or boot loader, the kernel, from the init system and from the system logger. It is a physical device consisting of a keyboard and a screen, and traditionally is a text terminal, but may also be a graphical terminal. System consoles are generalized to computer terminals, which are abstracted respectively by virtual consoles and terminal emulators. Today communication with system consoles is generally done abstractly, via the standard streams (stdin, stdout, and stderr), but there may be system-specific interfaces, for example those used by the system kernel.

User space

  • https://en.wikipedia.org/wiki/User_space - A modern computer operating system usually segregates virtual memory into kernel space and user space. Primarily, this separation serves to provide memory protection and hardware protection from malicious or errant software behaviour. Kernel space is strictly reserved for running a privileged operating system kernel, kernel extensions, and most device drivers. In contrast, user space is the memory area where application software and some drivers execute.


Busybox

  • Busybox - combines tiny versions of many common UNIX utilities into a single small executable. It provides replacements for most of the utilities you usually find in GNU fileutils, shellutils, etc. The utilities in BusyBox generally have fewer options than their full-featured GNU cousins; however, the options that are included provide the expected functionality and behave very much like their GNU counterparts. BusyBox provides a fairly complete environment for any small or embedded system.

toybox

  • toybox - combines common Linux command line utilities together into a single BSD-licensed executable that's simple, small, fast, reasonably standards-compliant, and powerful enough to turn Android into a development environment. See the links on the left for details.

gokrazy

  • gokrazy - a pure-Go userland for your Raspberry Pi 3 appliances. For a long time, we were unhappy with having to care about security issues and Linux distribution maintenance on our various Raspberry Pis. Then, we had a crazy idea: what if we got rid of memory-unsafe languages and all software we don’t strictly need? Turns out this is feasible. gokrazy is the result. [1] [2]

Login

systemd-logind

  • systemd-logind.service - a system service that manages user logins. It is responsible for: Keeping track of users and sessions, their processes and their idle state. This is implemented by allocating a systemd slice unit for each user below user.slice, and a scope unit below it for each concurrent session of a user. Also, a per-user service manager is started as system service instance of user@.service for each logged in user; Generating and managing session IDs. If auditing is available and an audit session ID is already set for a session, then this ID is reused as the session ID. Otherwise, an independent session counter is used; Providing PolicyKit-based access for users for operations such as system shutdown or sleep; Implementing a shutdown/sleep inhibition logic for applications; Handling of power/sleep hardware keys; Multi-seat management; Session switch management; Device access management for users; Automatic spawning of text logins (gettys) on virtual console activation and user runtime directory management

elogind

  • https://github.com/elogind/elogind - the systemd project's "logind", extracted out to be a standalone daemon. It integrates with PAM to know the set of users that are logged in to a system and whether they are logged in graphically, on the console, or remotely. Elogind exposes this information via the standard org.freedesktop.login1 D-Bus interface, as well as through the file system using systemd's standard /run/systemd layout. Elogind also provides "libelogind", which is a subset of the facilities offered by "libsystemd". There is a "libelogind.pc" pkg-config file as well.

Session

sessiond

  • sessiond - a standalone X session manager that reports the idle status of a session to systemd-logind.service(8) and handles its lock, unlock, sleep, and shutdown signals. sessiond also provides hooks triggered by inactivity or a signal, automatic backlight dimming on idle, and optional management of DPMS settings.

Shell


/etc/shells
  # list of shells installed on system






General

to find;

  • way of making previous command screen output be pushed to a buffer that can be flipped through/forked?
  • paste pwd to readline/zle
  • make alt-delete not break things
Alt-left/right
  move forwards/back one word
  • chsh - change your login shell
    • https://en.wikipedia.org/wiki/Chsh - is a command on Unix-like operating systems that is used to change a login shell. Users can either supply the pathname of the shell that they wish to change to on the command line, or supply no arguments, in which case chsh allows the user to change the shell interactively. chsh is a setuid program that modifies the /etc/passwd file, and only allows ordinary users to modify their own login shells. The superuser can modify the shells of other users, by supplying the name of the user whose shell is to be modified as a command-line argument.


Environment

  • direnv - unclutter your .profile, an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.



Standard streams

  • https://en.wikipedia.org/wiki/Standard_streams - preconnected input and output communication channels[1] between a computer program and its environment when it begins execution. The three I/O connections are called standard input (stdin), standard output (stdout) and standard error (stderr). Originally I/O happened via a physically connected system console (input via keyboard, output via monitor), but standard streams abstract this. When a command is executed via an interactive shell, the streams are typically connected to the text terminal on which the shell is running, but can be changed with redirection, e.g. via a pipeline. More generally, a child process will inherit the standard streams of its parent process.

n the C programming language, the standard input, output, and error streams are attached to the existing Unix file descriptors 0, 1 and 2 respectively. In a POSIX environment the <unistd.h> definitions STDIN_FILENO, STDOUT_FILENO or STDERR_FILENO should be used instead rather than magic numbers. File pointers stdin, stdout, and stderr are also provided.

  • stdin - 0
  • stdout - 1
  • stderr - 2

Echo



Here document

tr a-z A-Z << END_TEXT
  one two three
  four five six
  END_TEXT

This yields the output: 

ONE TWO THREE
FOUR FIVE SIX

Redirection

The > and >> are redirection operators for FD's (File Descriptors)

In bash you have tree standard FD's that are the standard input (strin), the standard output (strout) and the standard error (strerr). These can also be called by FD 0, FD 1 and FD 2 respectively.

Normally you would have you would have all FD's pointing to the terminal, but this can be changed by using redirection.

For example, if you call: 

command > log.txt

You will redirect the output to the file log.txt This is similar as calling: 

command 1> log.txt

As this only redirects strout you will still be able to see the error in the terminal. In order to redirect strerr to you log.txt file you will have to run: 

command 2> log.txt

Again, this only redirects strerr. If you wish to redirect both stdout and stderr you need to duplicate your stderr output to stdout by using the >& command. 

command 1> log.txt 2>&1

To understand this command you need to read it form right to left, first a copy of stderr is made to stdout, then strout is redirected to the log.txt file.

When you use redirection in this way, bash will not look if the file exists or not and simply create one regardless if that implies erasing the existing one. If you want to avoid loosing what has already been written in your log file you can then use the >> command in the same ways explained above, but in this case all the outputs will be appended to existing files.

GNU rReadline

Used by bash and other programs. 


bind -p
  # list bindings readline is capable of




  • rlwrap(1) - Linux man page - runs the specified command, intercepting user input in order to provide readline's line editing, persistent history and completion.rlwrap tries to be completely transparent - you (or your shell) shouldn't notice any difference between command and rlwrap command - except the added readline functionality, of course. This should even hold true when you are re-directing, piping and sending signals from and to command, or when command manipulates its terminal settings.


IFS

Wildcards

Patterns

  • http://mywiki.wooledge.org/BashGuide/Patterns - BASH offers three different kinds of pattern matching. Pattern matching serves two roles in the shell: selecting filenames within a directory, or determining whether a string conforms to a desired format. On the command line you will mostly use globs. These are a fairly straight-forward form of patterns that can easily be used to match a range of files, or to check variables against simple rules. The second type of pattern matching involves extended globs, which allow more complicated expressions than regular globs. Since version 3.0, Bash also supports regular expression patterns. These will be useful mainly in scripts to test user input or parse data. (You can't use a regular expression to select filenames; only globs and extended globs can do that.)

Substitution

print \T\h\i\s\ \i\s\ \*\p\o\i\n\t\l\e\s\s\*\ \ 
     \-\ \b\u\t\ \v\a\l\i\d\!

% read string
  This is a *string* with various `special' characters
% print -r -- ${(q)string}
  This\ is\ a\ \*string\*\ with\ various\ \`special\'\ characters


Prompt

  • https://github.com/dolmen/angel-PS1 - your guardian angel for the Unix world. The implementation is a daemon, a background process, that will service your shell every time it needs to display the prompt.


  • https://github.com/starship/starship - minimal, blazing fast, and extremely customizable prompt for any shell!The prompt shows information you need while you're working, while staying sleek and out of the way. Rust.


History


  • https://github.com/cantino/mcfly - replaces your default ctrl-r Bash history search with an intelligent search engine that takes into account your working directory and the context of recently executed commands. McFly's suggestions are prioritized in real time with a small neural network. [11]

Other

  • https://github.com/neurobin/shc - A generic shell script compiler. Shc takes a script, which is specified on the command line and produces C source code. The generated source code is then compiled and linked to produce a stripped binary executable.The compiled binary will still be dependent on the shell specified in the first line of the shell code (i.e shebang) (i.e. #!/bin/sh), thus shc does not create completely independent binaries.



  • https://github.com/leahneukirchen/snooze - tool for waiting until a particular time and then running a command. Together with a service supervision system such as runit, this can be used to replace cron(8).


Thumpson shell / sh

  • https://en.wikipedia.org/wiki/Thompson_shell - the first Unix shell, introduced in the first version of Unix in 1971, and was written by Ken Thompson.[1] It was a simple command interpreter, not designed for scripting, but nonetheless introduced several innovative features to the command-line interface and led to the development of the later Unix shells.

Bourne shel / sh

The Bourne shell, sh, was written by Stephen Bourne at AT&T as the original Unix command line interpreter; it introduced the basic features common to all the Unix shells, including piping, here documents, command substitution, variables, control structures for condition-testing and looping and filename wildcarding. The language, including the use of a reversed keyword to mark the end of a block, was influenced by ALGOL 68.



bash









man: echo



Scripting



#!/bin/bash
set -euo pipefail
IFS=$'\n\t'

set -e
  # exit if any command finishes with non 0 exit code
set -u
  # exit if undefined variable is used
set -x
  # causes bash to print each command before executing it
set -o pipefail
  # prevents errors in a pipeline from being masked
IFS=$'\n\t'
  # word splitting will happen only on newlines and tab characters

Maths

a=$(( 5 + 3 ))

(( a = 23 ))

(( ++a ))

(( a++ ))


Options


Completion


Prompt


Aliases

\curl
  # bypass any "curl" alias [18]


Variables


  • declare or typeset - builtins, they are exact synonyms, permit modifying the properties of variables. This is a very weak form of the typing available in certain programming languages. The declare command is specific to version 2 or later of Bash. The typeset command also works in ksh scripts.


VARIABLE=test
VARIABLE="test one two three"


export VARIABLE="make global to current shell env"


TZ=:/etc/localtime
  # read in from file



for x in {1..9}; do echo mv myfile$x other_dir/$x.txt; done

Special Parameters


$_
  last word of previous line


Etc.






Trap


Other

  • https://github.com/niieani/bash-oo-framework - a standard library and a boilerplate framework for writing tools using bash. It's modular and lightweight, while managing to implement some concepts from C#, Java or JavaScript into bash. The Infinity Framework is also plug & play: include it at the beginning of your existing script to import any of the individual features such as error handling, and start using other features gradually. The aim of Bash Infinity is to maximize readability of bash scripts, minimize the amount of code repeat and create a central repository for well-written, and a well-tested standard library for bash. [23]


  • https://github.com/dylanaraps/bush - an experiment to see how many standard tools and functions we can re-implement in pure bash. No external processes are used, only shell builtins. Bush is not meant to be used for anything serious and there's probably edge cases and bugs throughout.


  • https://github.com/dylanaraps/pure-bash-bible - goal of this book is to document known and unknown methods of doing various tasks using only built-in bash features. Using the snippets from this bible can help remove unneeded dependencies from scripts and in most cases make them faster.


Testing


zsh



Configuration

  • zbkrc README file - This file describes startup files for Z-shell (zsh), bash and ksh

Commands are then read from $ZDOTDIR/.zshenv. If the shell is a login shell, commands are read from /etc/zprofile and then $ZDOTDIR/.zprofile. Then, if the shell is interactive, commands are read from /etc/zshrc and then $ZDOTDIR/.zshrc. Finally, if the shell is a login shell, /etc/zlogin and $ZDOTDIR/.zlogin are read.



Variables

ZLE

The ZSH equivalent of readline for bash. 


bindkey "\e[A" history-beginning-search-backward
bindkey "\e[B" history-beginning-search-forward

Expansion

$_
  last entered word

$?
  returned exit code of last exec

Prompt





reset-prompt-and-accept-line() {
    zle reset-prompt
    zle accept-line
}

zle -N reset-prompt-and-accept-line

bindkey '^m' reset-prompt-and-accept-line







Completion

setopt menu_complete
  # auto select first completion option

compdef g=git
  # basic alias completion [26]

compdef _files my-local-python-script
  # simple _files completion [27]

compdef '_files -g "*.(eps|ps|pdf)"' okular
  # restrict to some file extensions

compdef -e 'words[1]=(git pull origin); service=git; (( CURRENT+=2 )); _git' ggl
  # make alias use existing completion [28]


From http://news.ycombinator.com/item?id=4867369 :

Tab completion and aliases need not be mutually exclusive. For instance, in zsh you can use: 

alias gs='git status'
compdef _git gs=git-status

FYI: I also mapped g without parameters to `git status -sb` but g with parameters will simply execute everything as normal 

g () {
  if [ $# -eq 0 ]
  then
    git status -sb
  else
    git $*
  fi
}
compdef g=git


Regex

autoload -U regexp-replace

regexp-replace VARNAME REGEXP REPLACE

PATH="$PATH:/foo/bar"
echo $PATH
regexp-replace PATH ':/foo/bar'


Plugins






  • Antigen - A plugin manager for zsh, inspired by oh-my-zsh and vundle. Antigen is a small set of functions that help you easily manage your shell (zsh) plugins, called bundles. The concept is pretty much the same as bundles in a typical vim+pathogen setup. Antigen is to zsh, what Vundle is to vim.





  • https://github.com/marlonrichert/zsh-snap - the light-weight plugin manager & Git repo manager for Zsh that's easy to grok. While tailored for Zsh plugins specifically, Znap also functions as a general-pupose utility for managing Git repos.






  • https://github.com/oknowton/zsh-dwim - attempts to predict what you will want to do next. It provides a key binding (control-u) that will replace the current (or previous) command line with the command you will want to run next.



  • https://github.com/zsh-users/zsh-history-substring-search - This is a clean-room implementation of the Fish shell's history search feature, where you can type in any part of any previously entered command and press the UP and DOWN arrow keys to cycle through the matching commands. You can also use K and J in VI mode or ^P and ^N in EMACS mode for the same.


  • https://github.com/zsh-users/zsh-completions - Additional completion definitions for Zsh. This projects aims at gathering/developing new completion scripts that are not available in Zsh yet. The scripts are meant to be contributed to the Zsh project when stable enough.



  • https://github.com/zsh-users/fizsh - the Friendly Interactive ZSHell. It is a front end to ZSH. It provides the user of ZSH with interactive syntax-highting and Matlab-like history search. It also has a both short and informative prompt.




  • https://github.com/rimraf/k - a zsh script / plugin to make directory listings more readable, adding a bit of color and some git status information on files and directories.






Themes

Frameworks


  • Prezto is the configuration framework for Zsh; it enriches the command line interface environment with sane defaults, aliases, functions, auto completion, and prompt themes.


  • https://github.com/romkatv/zsh4humans - A turnkey configuration for Z shell that aims to work really well out of the box. It combines the best Zsh plugins into a coherent whole that feels like a finished product rather than a DIY starter kit.


Dotfiles

csh

  • https://en.wikipedia.org/wiki/C_shell - a Unix shell created by Bill Joy while he was a graduate student at University of California, Berkeley in the late 1970s. It has been widely distributed, beginning with the 2BSD release of the Berkeley Software Distribution (BSD) which Joy first distributed in 1978. Other early contributors to the ideas or the code were Michael Ubell, Eric Allman, Mike O'Brien and Jim Kulp. The C shell is a command processor typically run in a text window, allowing the user to type commands. The C shell can also read commands from a file, called a script. Like all Unix shells, it supports filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration. What differentiated the C shell from others, especially in the 1980s, were its interactive features and overall style. Its new features made it easier and faster to use. The overall style of the language looked more like C and was seen as more readable.On many systems, such as macOS and Red Hat Linux, csh is actually tcsh, an improved version of csh. Often one of the two files is either a hard link or a symbolic link to the other, so that either name refers to the same improved version of the C shell.


  • An Introduction to the C shell - William Joy, revised for 4.3BSD by Mark Seiden - a new command language interpreter for UNIXsystems. It incorporates good features of other shells and a history mechanism similar to the redo of INTERLISP. While incorporating many features of other shells which make writing shell programs (shell scripts) easier, most of the features unique to csh are designed more for the interactive UNIX user. UNIX users who have read a general introduction to the system will find a valuable basic explanation of the shell here. Simple terminal interaction with csh is possible after reading just the first section of this document. The second section describes the shell's capabilities which you can explore after you have begun to become acquainted with the shell. Later sections introduce features which are useful, but not necessary for all users of the shell. Additional information includes an appendix listing special characters of the shell and a glossary of terms and commands introduced in this manual.



ash / dash

  • https://en.wikipedia.org/wiki/Almquist_shell - The Almquist shell (also known as A Shell, ash and sh) is a lightweight Unix shell originally written by Kenneth Almquist in the late 1980s. Initially a clone of the System V.4 variant of the Bourne shell, it replaced the original Bourne shell in the BSD versions of Unix released in the early 1990s. Derivative versions of ash are still installed as the default shell (/bin/sh) on FreeBSD, NetBSD, DragonFly BSD, MINIX, and Android, and in some Linux distributions.

Debian and derived Linux distributions such as Ubuntu ship a version of ash, known as dash (Debian Almquist shell), as the default /bin/sh, although Bash is the default login shell for interactive use. The reason for using dash is faster shell script execution, especially during startup of the operating system, compared to previous versions of Debian and Ubuntu that used Bash for this purpose.

Ash is also fairly popular in embedded Linux systems; its code was incorporated into the BusyBox catch-all executable often employed in this area.

dash


  • DashAsBinSh - Ubuntu Wiki - In Ubuntu 6.10, the default system shell, /bin/sh, was changed to dash (the Debian Almquist Shell); previously it had been bash (the GNU Bourne-Again Shell). The same change will affect users of Ubuntu 6.06 LTS upgrading directly to Ubuntu 8.04 LTS. This document explains this change and what you should do if you encounter problems.

tcsh

  • https://en.wikipedia.org/wiki/tcsh - a Unix shell based on and compatible with the C shell (csh). It is essentially the C shell with programmable command-line completion, command-line editing, and a few other features. Unlike the other common shells, functions cannot be defined in a tcsh script and the user must use aliases instead (as in csh). It is the native root shell for BSD-based systems such as FreeBSD.tcsh added filename and command completion and command line editing concepts borrowed from the Tenex system, which is the source of the “t”. Because it only added functionality and did not change what was there, tcsh remained backward compatible with the original C shell. Though it started as a side branch from the original csh source tree that Bill Joy had created, tcsh is now the main branch for ongoing development. tcsh is very stable but new releases continue to appear roughly once a year, consisting mostly of minor bug fixes.

Korn

rc

lshell

  • lshell is a shell coded in Python, that lets you restrict a user's environment to limited sets of commands, choose to enable/disable any command over SSH (e.g. SCP, SFTP, rsync, etc.), log user's commands, implement timing restriction, and more.

git-shell

  • git-shell - Restricted login shell for Git-only SSH access

Inferno

  • Inferno is a distributed operating system, originally developed at Bell Labs, but now developed and maintained by Vita Nuova® as Free Software. Applications written in Inferno's concurrent programming language, Limbo, are compiled to its portable virtual machine code (Dis), to run anywhere on a network in the portable environment that Inferno provides. Unusually, that environment looks and acts like a complete operating system.

fish

fishfish

oil



mrsh

  • https://git.sr.ht/~emersion/mrsh - A minimal POSIX shell. POSIX compliant, no less, no more. Simple, readable code without magic. Library to build more elaborate shells

promptless

  • https://github.com/dylanaraps/promptless - A super fast and extremely minimal shell prompt. promptless follows the suckless philosophy and is simple by design. The code-base has a focus on elegance and clarity. The prompt is entirely hackable. Unnecessary and unworthy features need to be patched in.

Xiki

  • Xiki: A shell console with GUI features. Xiki does what shell consoles do, but lets you edit everything at any time. It's trivial to make your own commands and menus to access other tools. [32]

Pdsh

  • Pdsh is a high-performance, parallel remote shell utility. It uses a sliding window of threads to execute remote commands, conserving socket resources while allowing some connections to timeout if needed. It was originally written as a replacement for IBM's DSH on clusters at LLNL.

shok

  • shok is a cross-platform non-POSIX interactive command shell. Similar to bash and cmd.exe, it allows the user to invoke programs and script their execution. shok will provide a secure, platform-agnostic, and user-friendly programming language that is designed to facilitate filesystem interaction and system management. [33]

Es

Scsh

psh

  • Perl Shell (psh) combines aspects of bash and other shells with the power of Perl scripting.

Elvish

Oh

  • Oh is a Unix shell. If you've used other Unix shells, oh should feel familiar. At its core, oh is a heavily modified dialect of the Scheme programming language complete with first-class continuations and proper tail recursion. Like early Scheme implementations, oh exposes environments as first-class values. Oh extends environments to allow both public and private members and uses these extended first-class environments as the basis for its prototype-based object system. Written in Go, oh is also a concurrent programming language. It exposes channels, in addition to pipes, as first-class values. As oh uses the same syntax for code and data, channels and pipes can, in many cases, be used interchangeably. This homoiconic nature also allows oh to support fexprs which, in turn, allow oh to be easily extended. In fact, much of oh is written in oh. [37]

Xonsh

  • Xonsh - a Python-powered, cross-platform, Unix-gazing shell language and command prompt. The language is a superset of Python 3.6+ with additional shell primitives that you are used to from Bash and IPython. It works on all major systems including Linux, OSX, and Windows. Xonsh is meant for the daily use of experts and novices. [38]

ngs

  • ngs - a Next Generation Shell. It has two main parts: the language and the interactive shell. NGS tries to fill the void between classical shells such as bash and general-purpose programming languages such as Ruby, Python, Perl, Go. The shells are domain-specific languages but the domain has changed so classical shells are not optimal for today's tasks. General-purpose languages on the other hand are not domain-specific so they are not good as shells and too verbose for system tasks scripting, not a good fit. [39]

cicada

viewglob

  • http://viewglob.sourceforge.net/ - a filesystem visualization add-on for Bash and Zsh. It tracks the command line and environment of any number of interactive shells (local and remote). A graphical display follows the currently active terminal, listing the contents of directories relevant to its shell and highlighting file selections and potential name completions dynamically.

dgsh

funky


janetsh


nushell

TopShell

qsh

Crush

Dune

janetsh

nsh

gosh

Commands


compgen -c | sort
  # list all commands


History

  • https://github.com/ellie/atuin - replaces your existing shell history with a SQLite database, and records additional context for your commands. Additionally, it provides optional and fully encrypted synchronisation of your history between machines, via an Atuin server.


SSH

See SSH

General

  • Black Duck Open Hub (formerly Ohloh.net) is an online community and public directory of free and open source software (FOSS), offering analytics and search services for discovering, evaluating, tracking, and comparing open source code and projects. Open Hub Code Search is free code search engine indexing over 21,000,000,000 lines of open source code from projects on the Black Duck Open Hub.


  • fu - commandlinefu.com from the commandline. Nothing has made more sense!









exec

  • exec - shall open, close, and/or copy file descriptors as specified by any redirections as part of the command. If exec is specified without command or arguments, and any file descriptors with numbers greater than 2 are opened with associated redirection statements, it is unspecified whether those file descriptors remain open when the shell invokes another utility. Scripts concerned that child shells could misuse open file descriptors can always close them explicitly, as shown in one of the following examples. If exec is specified with command, it shall replace the shell with command without creating a new process. If arguments are specified, they shall be arguments to command. Redirection affects the current shell execution environment.
exec windowmanager
  # switch script shell process with windowmanager process


Daemons

A daemon is a program that runs in the background, waiting for events to occur and offering services. A good example is a web server that waits for a request to deliver a page or a ssh server waiting for someone trying to log in. While these are full featured applications, there are daemons whose work is not that visible. Daemons are for tasks like writing messages into a log file (e.g. syslog, metalog) or keeping your system time accurate (e.g. ntpd). For more information see man 7 daemon.

  • Daemon turns other process into daemons. There are many tasks that need to be performed to correctly set up a daemon process. This can be tedious. Daemon performs these tasks for other processes.

Programs

See also Regex for sed, awk, etc.

  • Vim - Text editor, etc.
  • Git - Distributed version control






GNU coreutils

cat

Use output as input. 

cat filename.txt

cat `locate file.txt`
  # or
cat $(locate file.txt)





tee
sponge
  • sponge(1) - soak up standard input and write to a file
basename
uniq
od

printf

tr

echo $PATH | tr \: \\n | sort

Diff





to sort

  • auditd - userspace component to the Linux Auditing System [54]


  • pp - a preprocessorpp(1). allows embedding sh(1) code in files of any type by nesting it inside the #!\n token, where \n is a new line. That means that if you'd like a simple loop or an if inside an HTML file for instance, you could use pp(1).




  • cut - remove sections from each line of files
  • shelr - console screencasting tool


  • https://github.com/colinhowe/monner - Allows you to monitor the CPU, memory and network usage when running a program. Output is tab-separated for easy loading into spreadsheet programs.


If you want to write daemons in languages that can link against C functions (e.g. C, C++), see libslack which contains the core functionality of daemon.




  • https://github.com/tomnomnom/anew - Append lines from stdin to a file, but only if they don't already appear in the file. Outputs new lines to stdout too, making it a bit like a tee -a that removes duplicates.



Expect

  • Expect - a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. Expect is also useful for testing these same applications. And by adding Tk, you can also wrap interactive applications in X11 GUIs. Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily.

Calculator

expr 2 \* 2
  # returns 4. doesn't do non-integers.

Terminal extras


Desktop entries


  • gendesk - Utility for generating .desktop files and download icons by specifying a minimum of information




Job schedulers

https://en.wikipedia.org/wiki/Job_scheduler

at


at and batch read commands from standard input or a specified file which are to be executed at a later time. 

at 5am Oct 20   
  # executes commands at "time am/pm" "month" "day"
atq
  # lists the user's pending jobs, unless the user is the superuser; in that case, everybody's jobs are listed. The format of the output lines (one for each job) is: Job number, date, hour, queue, and username.
atrm
  # deletes jobs, identified by their job number.
batch
  # executes commands when system load levels permit; in other words, when the load average drops below 0.8, or the value specified in the invocation of atd.

/var/spool/at/

cron

  • SUS: crontab - schedule periodic background work

A cronjob is a task that a Cron system is instructed to run periodically. The crontab file is a configuration file for a user that defines tasks to run under the user’s account. The systab file is a file that specifies cronjobs for the system.

Configuration
/var/spool/cron/
  # user crontabs

/etc/crontab
  # system-wide crontab

/etc/cron.d/*
  # or
/etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, /etc/cron.monthly
  # distro specific extra



crontab -l
  # view crontabs

crontab -e
  # edit their crontabs

crontab -r
 # remove their crontabs

crontab saved_crontab_filename
 # overwrite their old crontab with saved crontab


  1. m h dom mon dow user command

There are several special predefined values which can be used to substitute the CRON expression. 

Entry                  Description                                 Equivalent To
 @yearly (or @annually) Run once a year, midnight, Jan. 1st         0 0 1 1 *
 @monthly               Run once a month, midnight, first of month  0 0 1 * *
 @weekly                Run once a week, midnight on Sunday         0 0 * * 0
 @daily                 Run once a day, midnight                    0 0 * * *
 @hourly                Run once an hour, beginning of hour         0 * * * *
 @reboot                Run at startup                              @reboot 

*    *    *    *    *  command to be executed
┬    ┬    ┬    ┬    ┬
│    │    │    │    │
│    │    │    │    │
│    │    │    │    └───── day of week (0 - 6) (0 is Sunday, or use names)
│    │    │    └────────── month (1 - 12)
│    │    └─────────────── day of month (1 - 31)
│    └──────────────────── hour (0 - 23)
└───────────────────────── min (0 - 59)

0 0 * */2 * certbot certonly --webroot -n -w /var/www/example.org/ -d example.org
# “At 00:00 in every 2nd month.”


  • /etc/cron.allow - If this file exists, then you must be listed therein (your username must be listed) in order to be allowed to use cron jobs.
  • /etc/cron.deny - If the cron.allow file does not exist but the /etc/cron.deny file does exist, then you must not be listed in the /etc/cron.deny file in order to use cron jobs.
dcron

Vanilla dcron supports asynchronous job processing. Just put it with @hourly, @daily, @weekly or @monthly with a jobname.

dcron has @daily, @hourly and so on. In fcron, you can use standard crontab entries and add &bootrun to the beginning of the line to repeat "missed" cronjobs.

cronie

Arch default cron as dcron wasn't maintained at the time.

dcron and fcron are not under active development, cronie is. cronie is small - 0.20MB installed. developed by Red Hat - it is not going anywhere and we have a guaranteed upgrade path. As far as I can tell cronie has no deps beyond glibc and pam. cronie has /etc/cron.d support and anacron support via an anacrontab config file. cronie extends the original vixie cron package so the syntax, core feature set, etc are stable. implements advanced security hooks as well and can integrate with SELINUX.

anacron
  • anacron is a computer program that performs periodic command scheduling which is traditionally done by cron, but without assuming that the system is running continuously. Thus, it can be used to control the execution of daily, weekly, and monthly jobs (or anything with a period of n days) on systems that don't run 24 hours a day. 


/etc/anacrontab
  # and/or
/var/spool/anacron/*
fcron

Fcron’s own crontab system uses the fcrontab file for configuration information. The fcrontab syntax is similar but differs slightly from the classic Vixie/ISC Cron crontab notation. fcron lacks /etc/cron.d/ functionality.

Mcron
  • Mcron - a 100% compatible replacement for Vixie cron. It is written in pure Guile, and allows configuration files to be written in scheme (as well as Vixie's original format) for infinite flexibility in specifying when jobs should be run.
Other

systemd/Timers

  • https://wiki.archlinux.org/index.php/Systemd/Timers - Timers are systemd unit files whose name ends in .timer that control .service files or events. Timers can be used as an alternative to cron (read #As a cron replacement). Timers have built-in support for calendar time events, monotonic time events, and can be run asynchronously.

Batch system

  • https://en.wikipedia.org/wiki/Job_scheduler - a computer application for controlling unattended background program execution of jobs. This is commonly called batch scheduling, as execution of non-interactive jobs is often called batch processing, though traditional job and batch are distinguished and contrasted; see that page for details. Other synonyms include batch system, distributed resource management system (DRMS), distributed resource manager (DRM), and, commonly today, workload automation. The data structure of jobs to run is known as the job queue.

batch

Task Spooler

  • Task Spooler - a Unix batch system where the tasks spooled run one after the other. The amount of jobs to run at once can be set at any time. Each user in each system has his own job queue. The tasks are run in the correct context (that of enqueue) from any shell/process, and its output/results can be easily watched. It is very useful when you know that your commands depend on a lot of RAM, a lot of disk use, give a lot of output, or for whatever reason it's better not to run them all at the same time, while you want to keep your resources busy for maximum benfit. Its interface allows using it easily in scripts.

nq

  • https://github.com/chneukirchen/nq - These small utilities allow creating very lightweight job queue systems which require no setup, maintenance, supervision, or any long-running processes.

Slurm

TORQUE

pueue

Luigi

  • https://github.com/spotify/luigi - Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

File watchers

inotify

  • inotify-tools is a C library and a set of command-line programs for Linux providing a simple interface to inotify.
  • inotifywatch - gather filesystem access statistics using inotify
  • inotail is a replacement for the 'tail' program found in the base installation of every Linux/UNIX system. It makes use of the inotify infrastructure in recent versions of the Linux kernel to speed up tailing files in the follow mode (the '-f' option). Standard tail polls the file every second by default while inotail listens to special events sent by the kernel through the inotify API to determine whether a file needs to be reread. Currently inotail is not fully compatible to neither POSIX or GNU tail but might be in the future.
  • entr - Run arbitrary commands when files change
  • - Guard::Shell - automatically runs shell commands when watched files are modified.
  • incron - inotify cron system. This program is an "inotify cron" system. It consists of a daemon and a table manipulator. You can use it a similar way as the regular cron. The difference is that the inotify cron handles filesystem events rather than time periods.
  • nodemon will watch the files in the directory that nodemon was started, and if they change, it will automatically restart your node application.

Other

watch -n 2 ls -la





  • fswatch - A cross-platform file change monitor with multiple backends: Apple OS X File System Events API, *BSD kqueue, Solaris/Illumos File Events Notification, Linux inotify and a stat()-based backend.


chroot

Startup

  • e4rat ("Ext4 - Reducing Access Times") is a toolset to accelerate the boot process as well as application startups. Through physical file realloction e4rat eliminates both seek times and rotational delays. This leads to a high disk transfer rate. Placing files on disk in a sequentially ordered way allows to efficiently read-ahead files in parallel to the program startup.

Formatting


Text user interface

See TUI

CLI dashboard

  • WTF - a personal information dashboard for your terminal, developed for those who spend most of their day in the command line.It allows you to monitor systems, services, and important information that you otherwise might keep browser tabs open for, the kinds of things you don’t always need visible, but do check in on every now and then. [64]



Web


Configuration management

See Dotfiles, Stack 

/etc

~/.config

/usr/local/etc/

etc.



Media

See Media, Playback

Misc.

fzf


fzy

  • https://github.com/jhawthorn/fzy - a simple, fast fuzzy finder for the terminal, fzy is faster and shows better results than other fuzzy finders. Most other fuzzy matchers sort based on the length of a match. fzy tries to find the result the user intended. It does this by favouring matches on consecutive letters and starts of words. This allows matching using acronyms or different parts of the path.


skim

pico

  • https://github.com/peco/peco - based on a python tool, percol. percol was darn useful, but I wanted a tool that was a single binary, and forget about python. peco is written in Go, and therefore you can just grab the binary releases and drop it in your $PATH. peco can be a great tool to filter stuff like logs, process stats, find files, because unlike grep, you can type as you think and look through the current results.


tosort


  • https://github.com/ysf/anewer - anewer appends lines from stdin to a file if they don't already exist in the file. This is a rust version of anew


  • https://github.com/p-gen/smenu - Terminal utility that allows you to use words coming from the standard input to create a nice selection window just below the cursor. Once done, your selection will be sent to standard output.


  • https://github.com/akavel/up - a tool for writing Linux pipes in a terminal-based UI interactively, with instant live preview of command results. The main goal of the Ultimate Plumber is to help interactively and incrementally explore textual data in Linux, by making it easier to quickly build complex pipelines, thanks to a fast feedback loop. This is achieved by boosting any typical Linux text-processing utils such as grep, sort, cut, paste, awk, wc, perl, etc., etc., by providing a quick, interactive, scrollable preview of their results. [65] [66]





McFly

  • https://github.com/cantino/mcfly - replaces your default ctrl-r shell history search with an intelligent search engine that takes into account your working directory and the context of recently executed commands. McFly's suggestions are prioritized in real time with a small neural network.

Cute

See also GUI#Utils 



strfile -c % myfortunes myfortunes.dat




  • https://gitlab.com/jallbrit/cbonsai - a bonsai tree generator, written in C using ncurses. It intelligently creates, colors, and positions a bonsai tree, and is entirely configurable via CLI options-- see usage. There are 2 modes of operation: static (the default), and live.













hexdump -C /dev/urandom | pv -q -L 2k | grep "ca fe"








  • https://github.com/svenstaro/genact - Pretend to be busy or waiting for your computer when you should actually be doing real work! Impress people with your insane multitasking skills. Just open a few instances of genact and watch the show. genact has multiple scenes that pretend to be doing something exciting or useful when in reality nothing is happening at all.




  • https://github.com/OGoodness/Minesweeper-Login - Minesweeper that cannot be exited in any way other than completing the game. Make it run while logging in and it will work as a second form of "auth". Used as a joke for Cyber Team --- Bypass with code - 6969420


Windows

  • Wine - (originally an acronym for "Wine Is Not an Emulator") is a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems, such as Linux, macOS, & BSD. Instead of simulating internal Windows logic like a virtual machine or emulator, Wine translates Windows API calls into POSIX calls on-the-fly, eliminating the performance and memory penalties of other methods and allowing you to cleanly integrate Windows applications into your desktop.


Install a .msi 

wine msiexec /i xyz.msi



wineboot --update
  # speed up wine start time



  • ProtonDB - a new tool released by Valve Software that has been integrated with Steam Play to make playing Windows games on Linux as simple as hitting the Play button within Steam. Underneath the hood, Proton comprises other popular tools like Wine and DXVK among others that a gamer would otherwise have to install and maintain themselves. This greatly eases the burden for users to switch to Linux without having to learn the underlying systems or losing access to a large part of their library of games. Proton is still in its infancy so support is inconsistent, but regularly improving. [69]



OS X

Retrieved from "https://wiki.thingsandstuff.org/index.php?title=CLI&oldid=63927"