*nix

From Things and Stuff Wiki
Revision as of 19:44, 20 December 2013 by Milk (talk | contribs) (→‎other)
Jump to navigation Jump to search


Mainly Linux. See also GUI, etc.

Guides

Articles

System

See also Distros

Standards

  • POSIX, an acronym for "Portable Operating System Interface", is a family of standards specified by the IEEE for maintaining compatibility between operating systems. POSIX defines the application programming interface (API), along with command line shells and utility interfaces, for software compatibility with variants of Unix and other operating systems.
  • The Single UNIX Specification (SUS) is the collective name of a family of standards for computer operating systems to qualify for the name "Unix". The SUS is developed and maintained by the Austin Group, based on earlier work by the IEEE and The Open Group.
  • Linux Standard Base (LSB) is a joint project by several Linux distributions under the organizational structure of the Linux Foundation to standardize the software system structure, including the filesystem hierarchy, used with Linux operating system. The LSB is based on the POSIX specification, the Single UNIX Specification, and several other open standards, but extends them in certain areas.
  • freedesktop.org is open source / open discussion software projects working on interoperability and shared technology for X Window System desktops. The most famous X desktops are GNOME and KDE, but developers working on any Linux/UNIX GUI technology are welcome to participate. freedesktop.org is building a base platform for desktop software on Linux and UNIX. The elements of this platform have become the backend for higher-level application-visible APIs such as Qt, GTK+, XUL, VCL, WINE, GNOME, and KDE. The base platform is both software and specifications.

UNIX

Linux

Tools

Modules

  • Kernel modules are pieces of code that can be loaded and unloaded into the kernel upon demand. They extend the functionality of the kernel without the need to reboot the system.
  • modprobe is a Linux program originally written by Rusty Russell and used to add a loadable kernel module (LKM) to the Linux kernel or to remove an LKM from the kernel. It is commonly used indirectly: udev relies upon modprobe to load drivers for automatically detected hardware.
  • Dynamic Kernel Module Support (DKMS) is a framework used to generate Linux kernel modules whose sources do not generally reside in the Linux kernel source tree. DKMS enables kernel device drivers to be automatically rebuilt when a new kernel is installed. An essential feature of DKMS is that it automatically recompiles all DKMS modules if a new kernel version is installed. This allows drivers and devices outside of the mainline kernel to continue working after a Linux kernel upgrade. Another benefit of DKMS is that it allows the installation of a new driver on an existing system, running an arbitrary kernel version, without any need for manual compilation or precompiled packages provided by the vendor.

BSD

Paramaters

cat /proc/sys/net/ipv4/ip_forward

Boot

Network

Boot loaders

"In order to boot Arch Linux, a Linux-capable boot loader such as GRUB(2), Syslinux, LILO or GRUB Legacy must be installed to the Master Boot Record or the GUID Partition Table. The boot loader is responsible for loading the kernel and initial ramdisk before initiating the boot process."

GRUB2

Syslinux

rEFInd

Kernal loading

initrd

  • initrd (initial ramdisk) is a scheme for loading a temporary root file system into memory in the boot process of the Linux kernel. initrd and refer to two different methods of achieving this. Both are commonly used to make preparations before the real root file system can be mounted.

initramfs

  • mkinitcpio is the next generation of initramfs creation.
ps -p 1 -o comm=

init

  • init (short for initialization) is a daemon process that is the direct or indirect ancestor of all other processes. It automatically adopts all orphaned processes. Init is the first process started during booting, and is typically assigned PID number 1. It is started by the kernel using a hard-coded filename, and if the kernel is unable to start it, a kernel panic will result. Init continues running until the system is shut down. The design of init has diverged in Unix systems such as System III and System V, from the functionality provided by the init in Research Unix and its BSD derivatives. The usage on most Linux distributions is compatible with System V, but some distributions, such as Slackware, use a BSD-style and others, such as Gentoo, have their own customized version. Several replacement init implementations have been written which attempt to address design limitations in the standard versions. These include systemd and Upstart, the latter being used by Ubuntu and some other Linux distributions.

SysVinit

OpenRC

systemd

  • systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux control groups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. It can work as a drop-in replacement for sysvinit.

Upstart

runit

Supervisor

  • Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. It shares some of the same goals of programs like launchd, daemontools, and runit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. Instead it is meant to be used to control processes related to a project or a customer, and is meant to start like any other program at boot time.

udev

  • udev is a generic kernel device manager. It runs as a daemon on a Linux system and listens (via netlink socket) to uevents the kernel sends out if a new device is initialized or a device is removed from the system. Loads kernel modules by utilizing coding parallelism to provide a potential performance advantage versus loading these modules serially. The modules are therefore loaded asynchronously. The inherent disadvantage of this method is that udev does not always load modules in the same order on each boot. If the machine has multiple block devices, this may manifest itself in the form of device nodes changing designations randomly. For example, if the machine has two hard drives, /dev/sda may randomly become /dev/sdb. now part of systemd code tree [4]

Manages /dev. Rules in /etc/udev/rules.d/ and /usr/lib/udev/rules.d/ 

udevadm info -a -n [device path]

Reboot and shutdown

Command-line

TTY / console

See also Typography#Terminal

Getty

getty is the generic name for a program which manages a terminal line and its connected terminal. Its purpose is to protect the system from unauthorized access. Generally, each getty process is started by init and manages a single terminal line. Within the context of a typical Arch Linux installation, the terminals managed by the getty processes are implemented as virtual consoles.

agetty

mgetty

fgetty

Qingu

PYT / terminal emulator

fbterm

Xterm

  • How to change the title of an xterm - This document explains how to use escape sequences to dynamically change window and icon titles of an xterm. Examples are given for several shells, and the appendix gives escape sequences for some other terminal types.

urxvt

if not using a tiling window manager, tabbed is good + remote tmux. urxvt has overhead because of unicode. 

urxvtd -q -f -o
  • Arch Forum: [Solved Change urxvt's font on the fly: Can YOU do it?]
 printf '\33]50;%s\007' "xft:Terminus:pixelsize=16"

urxvt sets char width as widest char in font. there's a patched package for that.

st

-*-terminus-medium-r-*-*-10-*-*-*-*-*-*-*

other

  • FireSSH is a free, cross-platform SSH terminal client for Mozilla Firefox and Google Chrome. Written entirely in Javascript!
  • oterm is a web browser unix terminal. It serves a console so you can access your server from anywhere in the world where you have an Internet connection. It effectively is an xterm in a browser. [6]

retro; 

/usr/lib/xscreensaver/apple2 -text -fast -program bash

Terminal multiplexing

Screen

config goes in ~/.screenrc 

escape ^Ww
  change escape key to w

Tmux

Config goes in ~/.tmux.conf, which can be symlinked to a hidden git repo folder.

Better than screen.

Articles
Commands
tmux lsc
  list clients

tmux detach-client -t /dev/pts/26
  remove other clients from session (if screensize is fucked)

split-window

bind-key -n C-S-Left swap-window -t -1
bind-key -n C-S-Right swap-window -t +1

open new window in current pwd; 

bind <key of your choice> default-path $(pwd) \; split-window\; set default-path ~/

tmux_pwd () {
   [ -z "${TMUX}" ] && return
   tmux set default-path $(pwd) > /dev/null && tmux new-window
   (( sleep 300;
   tmux set default-path ~/ > /dev/null; ) & ) > /dev/null 2>&1
}

alias tpwd="tmux_pwd"
Additions

dvtm

Misc

  • Mosh (mobile shell) - Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes. Mosh is a replacement for SSH. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.

Colour

General

dircolors

config file;
dir_colors

print dircolor config with colours; 

echo $LS_COLORS | tr : '\n' | sed 's/\(.*\)=\(.*\)/\x1b[\2m\1\t\2\x1b[0m/'

Articles

Themes

For ZSH, Vim, Tmux, etc.

Tools

Threads

Clipboard

The PRIMARY selection is used when you select some text with the mouse. You usually paste it using the middle button. The CLIPBOARD selection is used when you copy text by using, for example, the Edit/Copy menu. You may paste it using the Edit/Paste menu. 

Shift-Insert
  paste clipboard selection (where ctrl-v isn't supported)

Mouse

  • gpm is the mouse support for Linux on the console

Shell

/etc/shells

General

to find;

  • way of making previous command screen output be pushed to a buffer that can be flipped through/forked.
  • paste pwd to readline or whatnot
Alt-left/right
  move forwards/back one word

Pipes

Readline

Used by bash and other programs.

sh

The Bourne shell, sh, was written by Stephen Bourne at AT&T as the original Unix command line interpreter; it introduced the basic features common to all the Unix shells, including piping, here documents, command substitution, variables, control structures for condition-testing and looping and filename wildcarding. The language, including the use of a reversed keyword to mark the end of a block, was influenced by ALGOL 68.

bash

man: echo

Basics

Options

Completion

Prompt

Etc.

$_
  last entered word

$?
  returned exit code of last exec

zsh

Configuration

  • zbkrc README file - This file describes startup files for Z-shell (zsh), bash and ksh
Completion
ZLE

ZSH readline

Prompt

Plugins

  • zsh-history-substring-search - This is a clean-room implementation of the Fish shell's history search feature, where you can type in any part of any previously entered command and press the UP and DOWN arrow keys to cycle through the matching commands. You can also use K and J in VI mode or ^P and ^N in EMACS mode for the same.
  • zsh-completions - Additional completion definitions for Zsh. This projects aims at gathering/developing new completion scripts that are not available in Zsh yet. The scripts are meant to be contributed to the Zsh project when stable enough.
  • FIZSH is the Friendly Interactive ZSHell. It is a front end to ZSH. It provides the user of ZSH with interactive syntax-highting and Matlab-like history search. It also has a both short and informative prompt.
  • Antigen - A plugin manager for zsh, inspired by oh-my-zsh and vundle. Antigen is a small set of functions that help you easily manage your shell (zsh) plugins, called bundles. The concept is pretty much the same as bundles in a typical vim+pathogen setup. Antigen is to zsh, what Vundle is to vim.

lshell

  • lshell is a shell coded in Python, that lets you restrict a user's environment to limited sets of commands, choose to enable/disable any command over SSH (e.g. SCP, SFTP, rsync, etc.), log user's commands, implement timing restriction, and more.

git-shell

  • git-shell - Restricted login shell for Git-only SSH access

Inferno

  • Inferno is a distributed operating system, originally developed at Bell Labs, but now developed and maintained by Vita Nuova® as Free Software. Applications written in Inferno's concurrent programming language, Limbo, are compiled to its portable virtual machine code (Dis), to run anywhere on a network in the portable environment that Inferno provides. Unusually, that environment looks and acts like a complete operating system.

Fish

fishfish

Other

  • Xiki: A shell console with GUI features. Xiki does what shell consoles do, but lets you edit everything at any time. It's trivial to make your own commands and menus to access other tools. [11]
  • Pdsh is a high-performance, parallel remote shell utility. It uses a sliding window of threads to execute remote commands, conserving socket resources while allowing some connections to timeout if needed. It was originally written as a replacement for IBM's DSH on clusters at LLNL.

Files

File systems

Ext

kjournald is responsible for the journal of ext3 [12]

Btrfs

See also Backup#Btrfs

General

Subvolumes appear like directories. inode is different.

"Btrfs support is included in the linux package (as a module). Needs a reboot after installing before btrfs recognised. User space utilities are available in btrfs-progs. For multi-devices support (RAID like feature of btrfs) aka btrfs volume in early boot, you have to enable btrfs mkinitcpio hook (provided by mkinitcpio package) to be able to use, for example, a root btrfs volume. If the btrfs volume is a non-system volume, one only needs to set USEBTRFS="yes" in /etc/rc.conf. However, if you only use bare btrfs partition, such options are not needed."

"The btrfs scrub command reads redundant data and validates all the checksums, correcting any errors it finds along the way, using the checksum to determine which copy is the valid one. But with a single drive, how can it correct anything? The metadata - the file system overhead that is used to manage your data - is always stored in a redundant manner by default, even on a single drive. As a result, any corrupted metadata can be corrected, on the fly."

"EXT4 checksums its journal, which AFAIK will protect against errors caused by sync failures (ie. power failure during disk I/O). But it’s not going to protect against latent sector errors. To do that, you need checksumming on all the file data, along the lines of what ZFS or BTRFS provides."

A cross-subvolume copy patch has made it into 3.6_rc. This patch will allow cp --reflink across subvolumes, as long as the copy does not cross mount points.

Commands
mkfs.btrfs -L [label] /dev/[device]

btrfs filesystem df /media/[drive]
btrfs filesystem show

btrfs-debug-tree -R /dev/sdg
  show drive/subvolume infos, unmounted

btrfs subvolume create [<dest>/]
btrfs subvolume snapshot /mnt/btrfs /mnt/btrfs/snapshot_of_root
btrfs subvolume delete [<dest>/]

mount -o remount,compress=lzo /dev/sdXY /mnt/target

btrfs filesystem defragment /

btrfs device add /dev/sdc /mnt/btrfs

Cloning a file between subvolumes; 

cp --reflink /mnt/MYFILES/myfile1 /mnt/MYFILES/myfile3

copy-on-write, without the ram requirement of zsf snapshots every 30 seconds, ability to mount from previous gen

GUI
  • btrfs-gui is a graphical user interface tool for inspecting and managing btrfs filesystems. It is capable of managing filesystems on the local machine, and filesystems on remote network-accessible machines. It requires root access to the machine to perform most of its tasks (but separates the root-access part from the GUI).
  • Snapper is a tool for managing btrfs snapshots. Apart from the obvious creation and deletion of snapshots it can compare snapshots and revert differences between snapshots. In simple terms, this allows users to view older versions of files and revert changes. Snapper is available as a command line interface tool and a YaST module. Both make use of the C++ library libsnapper which is also available to other programs.
Articles

Ceph

  • Ceph is a free software distributed file system. Ceph's main goals are to be POSIX-compatible, and completely distributed without a single point of failure. The data is seamlessly replicated, making it fault tolerant. Clients mount the file system using a Linux kernel client. On March 19, 2010, Linus Torvalds merged the Ceph client for Linux kernel 2.6.34 which was released on May 16, 2010. An older FUSE-based client is also available. The servers run as regular Unix daemons.

aufs

Other

  • Opendedup Develops SDFS, a file-system that does inline deduplication.

RAID

  • unRAID is an embedded Network Attached Storage (NAS) server operating system, designed for digital media storage. It allows you to build an array of hard drives and share the data from those drives across the local network (typically within a house or business). Importantly, it protects all the data on the drives if one should fail.

LVM

ZFS

"FreeBSD ZFS tuning guide wiki indicates you'll need about 5GB of ram per 1TB of saved disk space"

Other

Image files

Tools


Directory structure

See LSB, etc.

  • rmshit - Keep $HOME or other dir clean from unwanted tempfiles, configs and other crap you'll never use that's autocreated upon execution of bad behaving applications
  • BleachBit quickly frees disk space and tirelessly guards your privacy. Free cache, delete cookies, clear Internet history, shred temporary files, delete logs, and discard junk you didn't know was there. Designed for Linux and Windows systems, it wipes clean a thousand applications including Firefox, Internet Explorer, Adobe Flash, Google Chrome, Opera, Safari,and more. Beyond simply deleting files, BleachBit includes advanced features such as shredding files to prevent recovery, wiping free disk space to hide traces of files deleted by other applications, and vacuuming Firefox to make it faster.

Mount

lsbkl
cat /proc/partitions
blkid

sd[a,b,etc]
  drive
sda[1,2,etc]
  partition of drive

mount /dev/sdxY /some/directory

umount /some/directory

mount -o remount /
  remount partition after /etc/fstab change

mount -o loop example.img /home/you/dir

Partitions

  • GNU Parted manipulates partition tables. This is useful for creating space for new operating systems, reorganizing disk usage, copying data on hard disks and disk imaging. The package contains a library, libparted, as well as well as a command-line frontend, parted, which can also be used in scripts.
  • http://gparted.sourceforge.net/

dd

  • dd - Copy a file, converting and formatting according to the options.
    • dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data.
dd if=/dev/sr0 of=myCD.iso bs=2048 conv=noerror,sync
  create an ISO disk image from a CD-ROM.
dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=noerror
  Clone one partition to another
dd if=/dev/ad0 of=/dev/ad1 bs=1M conv=noerror
  Clone a hard disk "ad0" to "ad1".

dd if=/dev/zero bs=1024 count=1000000 of=file_1GB
dd if=file_1GB of=/dev/null bs=64k
  drive benchmark test and analyze the sequential read and write performance for 1024 byte blocks

du (disk usage)

du -sh
  size of a folder
du -S
  size of files in a folder

du -aB1m|awk '$1 >= 100'
  everything over 100Mb

cd / | sudo du -khs *
  show root folder size

sudo du -a --max-depth=1 /usr/lib | sort -n -r | head -n 20
  size of program folders /usr/lib

du -sk ./* | sort -nr | awk 'BEGIN{ pref[1]="K"; pref[2]="M"; pref[3]="G";} { total = total + $1;
x = $1; y = 1;  while( x > 1024 ) { x = (x + 1023)/1024; y++; }
printf("%g%s\t%s\n",int(x*10)/10,pref[y],$2); } END { y = 1; while( total > 1024 )
{ total = (total + 1023)/1024; y++; } printf("Total: %g%s\n",int(total*10)/10,pref[y]); }'

ncdu

  • ncdu - ncurses disk usage
ncdu / --exclude /home --exclude /media --exclude /run/media
  check everything apart from home and external drives

ncdu / --exclude /home --exclude /media --exclude /run/media
  check everything apart from external drives

ncdu / --exclude /home --exclude /media --exclude /run/media --exclude /boot
--exclude /tmp --exclude /dev --exclude /proc
 just the root partition

df

df -h
  human readable

di

  • di is a disk information utility, displaying everything (and more) that your 'df' command does. It features the ability to display your disk usage in whatever format you prefer. It also checks the user and group quotas, so that the user sees the space available for their use, not the system wide disk space.

du

Baobab

Other

todo; source aliases.zsh

  • Filelight creates an interactive map of concentric, segmented rings that help visualise disk usage on your computer.

Files and directories

touch

touch filename
  create a file or update timestamp

ls

ls
  list in row
ls -l
  long list

ls *
  files in directory and immediate subdiretories

just names; 

ls -m1
  -m fill width with a comma separated list of entries ??
ls --format single-column
  column of names only
ls -l | grep - | awk '{print $9}'
  using awk to show the 9th word (name). strips colour.
ls -l | cut -f9 -s -d" "
  using cut to cut from the 9th word, using space as a delimiter. strips colour.
ls | cat
  neat

ls -a
  show hidden files
ls  -A
  show hidden files, exclude . and ..

cd

cd change/directory/path

[16]


mv

mv position1 ~/position2
  basic move

mkdir

mkdir directory
mkdir directory -p
  no error if existing, make parent directories as needed

ln

symlink 

ln -s {target-filename}
ln -s {target-filename} {symbolic-filename}
  create soft link

other

stat .
  display file or file system status
stat -c "%n %a" * | column -t
  directory files + octal

v def conf       =>     vim /some/awkward/path/to/type/default.conf
j abc            =>     cd /hell/of/a/awkward/path/to/get/to/abcdef
m movie          =>     mplayer /whatever/whatever/whatever/awesome_movie.mp4
o eng paper      =>     xdg-open /you/dont/remember/where/english_paper.pdf
vim `f rc lo`    =>     vim /etc/rc.local
vim `f rc conf`  =>     vim /etc/rc.conf

alias defaults; 

alias a='fasd -a'        # any
alias s='fasd -si'       # show / search / select
alias d='fasd -d'        # directory
alias f='fasd -f'        # file
alias sd='fasd -sid'     # interactive directory selection
alias sf='fasd -sif'     # interactive file selection
alias z='fasd_cd -d'     # cd, same functionality as j in autojump
alias zz='fasd_cd -d -i' # cd with interactive selection

File types

xdg-mime default Thunar.desktop inode/directory
  to make Thunar the default file-browser
xdg-mime default xpdf.desktop application/pdf
  to use xpdf as the default PDF viewer

/usr/share/applications/defaults.list      (global)
~/.local/share/applications/defaults.list  (per user, overrides global)

[Default Applications]
mimetype=desktopfile1;desktopfile2;...;desktopfileN

Copying files

cp - copy files and directories

scp -P 2264 foobar.txt your_username@remotehost.edu:/some/remote/directory
scp -rP 2264 folder your_username@remotehost.edu:/some/remote/directory

Removing files

rm file

rm -rf directory

find * -maxdepth 0 -name 'keepthis' -prune -o -exec rm -rf '{}' ';'
  remove all but keepthis [20]

Viewing files

<filename

cat

cat filename
  output file to screen
cat -n filename
  output file to screen w/ line numbers
cat filename1 filename2
  output two files (concatinate)
cat filename1 > filename2
  overwrite filename2 with filename1
cat filename1 >> filename2
  append filename1 to filename2
cat filename{1,2} > filename2
  add filename1 and filename2 together into filename3

other

head filename
  top 10 lines of file
head -23 filename
  top 23 lines of file
tail filename
  bottom 10 lines of file
tail -23 filename
  bottom 23 lines of file

sed -n 20,30p filename
  print lines 20..30 of file [21]

Pagers

more

more is a filter for paging through text one screenful at a time. This version is especially primitive. Users should realize that less(1) provides more(1) emulation plus extensive enhancements.

less

less is an improvement on more and a funny name. if you want the output to be left on screen, use more.

vimpager

other

Managing files

setup ~/.config/ranger/ with defaults; 

ranger --copy-config=all

Finding files

find

find /usr/share -name README
find ~/Journalism -name '*.txt'
find ~/Programming -path '*/src/*.c'

find ~/Journalism -name '*.txt' -exec cat {} ;
  exec command on result path (aliases don't work in exec argument) 

find ~/Images/Screenshots -size +500k -iname '*.jpg'
find ~/Journalism -name '*.txt' -print0 | xargs -0 cat   (faster than above)

find / -group [group]
find / -user [user]

find . -mtime -[n]
  File's data was last modified n*24 hours ago

find . -mtime +5 -exec rm {} \;
  remove files older than 5 days

find . -type f -print0 | xargs -0 stat -c "%y %s %n"

find . -type f -links +1
  list hard links

locate

creates smaller Mb db of paths that can be queried with locate

PRUNEPATHS in /etc/updatedb.conf

to sort

Editing files

See Vim, Emacs

Finding file content

grep

grep "stuff" sqldump.sql | fold -w 200 | grep -C 1 "stuff"

The first grep gets the (mile-wide) line that has the match, then fold will split the mile-wide line into 200 char long lines, and "grep -C 1" will show only the one 200 char wide line where the match is + 1 line of context before and after. [22]

  • sgrep - search a file for a structured pattern

Compression

tar

tar -xvf foo.tar
  verbosely extract foo.tar
tar -xzf foo.tar.gz
  extract gzipped foo.tar.gz

tar -xjf foo.tar.bz2 -C bar/
  extract bzipped foo.tar.bz2 after changing directory to bar
tar -xzf foo.tar.gz blah.txt
  extract the file blah.txt from foo.tar.gz

tar -cjf foo.tar.bz2 bar/
  create bzipped tar archive of the directory bar called foo.tar.bz2

zip

zip

unzip

unzip -l archive.zip
  list files in archive
gzip
  • gzip, gunzip, zcat - compress or expand files
bzip2
7z
  • 7-Zip is a file archiver with the highest compression ratio. The program supports 7z (that implements LZMA compression algorithm), ZIP, CAB, ARJ, GZIP, BZIP2, TAR, CPIO, RPM and DEB formats. Compression ratio in the new 7z format is 30-50% better than ratio in ZIP format.
    • p7zip is a port of 7za.exe for POSIX systems like Unix (Linux, Solaris, OpenBSD, FreeBSD, Cygwin, AIX, ...), MacOS X and also for BeOS and Amiga. 7za.exe is the command line version of 7-zip, see http://www.7-zip.org/. 7-Zip is a file archiver with highest compression ratio.
    • man z7 (p7zip)
    • p7zip-light in AUR
7z x filename
  extract archive with directories
xz

pax

  • pax will read, write, and list the members of an archive file, and will copy directory hierarchies. pax operation is independent of the specific archive format, and supports a wide variety of different archive formats. A list of supported archive formats can be found under the description of the -x option. [23]

Other

Generic function

# Extract Files
extract() {
 if [ -f $1 ] ; then
     case $1 in
         *.tar.bz2)   tar xvjf $1    ;;
         *.tar.gz)    tar xvzf $1    ;;
         *.tar.xz)    tar xvJf $1    ;;
         *.bz2)       bunzip2 $1     ;;
         *.rar)       unrar x $1     ;;
         *.gz)        gunzip $1      ;;
         *.tar)       tar xvf $1     ;;
         *.tbz2)      tar xvjf $1    ;;
         *.tgz)       tar xvzf $1    ;;
         *.zip)       unzip $1       ;;
         *.Z)         uncompress $1  ;;
         *.7z)        7z x $1        ;;
         *.xz)        unxz $1        ;;
         *.exe)       cabextract $1  ;;
         *)           echo "\`$1': unrecognized file compression" ;;
     esac
 else
     echo "\`$1' is not a valid file"
 fi
}

Encryption

eCrypt

TrueCrypt

Rubberhose

Emulation

Users

Files

Commands

useradd username
  create user with defaults (no password)
useradd -D
  show defaults that will be used

useradd -m -g [initial_group] -G [additional_groups] -s [login_shell] [username]
  -m - add home dir
  -d - non-default home dir path
  -c comment
  -e 2006-06-30 - expires
useradd -m -g users -G audio,lp,optical,storage,video,wheel,games,power,scanner -s /bin/bash username

adduser
  interactive tool

passwd
  set password of current account
passwd username
  set password of username
passwd -S username
  see password status of username

userdel username
  remove user
userdel -r username
  remove user plus their home folder, mail spool

cat /etc/passwd | cut -d: -f 1,3,6 | grep "[5-9][0-9][0-9]"
  List users on a system [24]

groups [username]
  show users groups

Groups

groupadd [group]
  add group

gpasswd -a [user] [group]
  add user to group
gpasswd -d [user] [group]
  delete user from group

User must relogin for new group to take effect. 

id [username]
  show user id and their groups ids

last
  lastb - show listing of last logged in users

Permissions

stat -c '%A %a %n' *
  list permissions in octal

chmod

chmod
  change file mode bits

or

Users;

  • u - the user who owns the file.
  • g - other users who are in the file's group.
  • o - all other users.
  • a - all users; the same as ugo.

Operation;

  • + - to add the permissions to whatever permissions the users already have for the file.
  • - - to remove the permissions from whatever permissions the users already have for the file.
  • = - to make the permissions the only permissions that the users have for the file.

Permissions; r - the permission the users have to read the file. w - the permission the users have to write to the file. x - the permission the users have to execute the file, or search it if it is a directory.

chown

For changing user and group ownership of file 

chown -R user:group .
  change all and subitems [25]

su / sudo

Run things as root or another user. There are a number of methods to do higher privilege things, with differing resultant permissions. To refind good forum threads.. 

sudo -s
  open a new shell as root, but with your original user's environment variables

sudo -i
  open a new shell as if root had logged in; root's environment variables, root's .bash_profile, etc.

sudo su -
  same as above

snippet to source... 

sudo sh -c "echo 'something' >> /etc/privilegedfile"

sudoers

visudo
  command to safely add sudoers rights using nano

user ALL=(ALL) ALL

ACLs

Partition must have acl set in /etc/fstab (and be remounted after). 

setfacl -m "u:username:permissions"
setfacl -m "u:uid:permissions"
  add permissions for user

setfacl -m "g:groupname:permissions"
setfacl -m "g:gid:permissions"
  add permissions for group

setfacl -m "u:user:rwx" file
  add read, write, execure perms for user for file
setfacl -Rm "u:user:rw" /dir
  add recursive read, write perms for user for dir
setfacl -Rdm "u:user:rw" /dir
  add recursive read, write perms for user for dir and make them default for future changes

Session permission management

For non-root mount permissions, etc.

Commands

Packages

  • pkgs.org - Linux Software Catalog and Packages Search. RPM, DEB, TGZ, TXZ packages from well-known repositories of the Archlinux, Fedora, CentOS, RHEL, Scientific Linux, Debian, Ubuntu, openSUSE, Mandriva, Mageia and Slackware distributions; packages search by name, filename, summary, description, requires, provides, files and directories; pPowerful packages browser (summary, description, files, requires, provides, changelog, etc.)

Basic

Debian

including ubuntu, linux mint and others 

dpkg --get-selections > installed-software
  create list of installed software

dpkg --set-selections < installed-software
dselect
  reinstall from list

dpkg --list
  to check
Apt
apt-get install package
etc.  
apt-cache search 'web server'
etc.

apt-find show python
 show files installed by package
GUI

Arch

pacman
pacman -Sy [package]
  install [package]
pacman -Syu
  upgrade system

pacman -Qo [file]
  check what package owns a file
pacman -Qi
  list installed programms

pacman -Qqtd
  check whether there are any orphaned packages
pacman -Rsn packagename
  remove orphaned packages
pacman -Rc packagename
  remove package and deps ('cascade')

pacman -Rdd qt
pacman -Syuuu
  when qt4 replaced qt, but qt had deps.
  • cacheclean - Cleans up pacman packages. Users selects how many old versions to keep.
cacheclean {-p} {-v} <# of copies to keep>
# of copies to keep - (required) how many generations of each package.
-p - (optional) preview what would be deleted.
-v - (optional) show deleted packages.

for i in `pacman -Qdt | awk '{print $1}'`; do pacman -R $i ; done

Helpers
Packages
 makepkg -g >> PKGBUILD && makepkg

"ERROR: PKGBUILD contains CRLF characters and cannot be sourced." Solution: 

sed -i 's/^M//' PKGBUILD

[ctrl+v][ctrl+m] for the ^M symbol. 

pacman -Qdt
  list all orphans
pacman -Rsn $(pacman -Qdtq)
  remove them all
Repos
Other
  • gist: unused.sh: finds unused package on your Arch Linux box

RPM

7z also extracts rpm to cpio, and cpio to files.

Yum

Gentoo Portage

Guix

Ports Collection

Other

Resources

Use output as input. 

cat `locate file.txt`

or 

cat $(locate file.txt)

todo; piping and redirection

  • inotail is a replacement for the 'tail' program found in the base installation of every Linux/UNIX system. It makes use of the inotify infrastructure in recent versions of the Linux kernel to speed up tailing files in the follow mode (the '-f' option). Standard tail polls the file every second by default while inotail listens to special events sent by the kernel through the inotify API to determine whether a file needs to be reread.

Currently inotail is not fully compatible to neither POSIX or GNU tail but might be in the future.

Configuration

/etc
~/.config

/usr/local/etc/

etc.

Management

  • etckeeper is a collection of tools to let /etc be stored in a git, mercurial, darcs, or bzr repository. It hooks into apt (and other package managers including yum and pacman-g2) to automatically commit changes made to /etc during package upgrades. It tracks file metadata that revison control systems do not normally support, but that is important for /etc, such as the permissions of /etc/shadow. It's quite modular and configurable, while also being simple to use if you understand the basics of working with revision control.
  • Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files. seems for /etc
  • fresh is a tool to source shell configuration (aliases, functions, etc) from others into your own configuration files. We also support files such as ackrc and gitconfig. Think of it as Bundler for your dot files (from github)
  • dots - Modular dotfile manager. Symlinks dotfiles from plugins to HOME.
  • Homesick is sorta like rip, but for dotfiles. It uses git to clone a repository containing dotfiles, and saves them in ~/.homesick. It then allows you to symlink all the dotfiles into place with a single command. We call a repository that is compatible with homesick to be a 'castle'. To act as a castle, a repository must be organized like so: Contains a 'home' directory; 'home' contains any number of files and directories that begin with '.'
  • vcsh - Version Control System for $HOME - multiple Git repositories in $HOME

Dotfiles

  • http://dotfiles.github.io/
  • dotphiles - A community driven framework of dotfiles, for the usual terminal apps and shells, designed to work across multiple platforms and degrade for older versions of software or O/S, allowing you to use the same settings on all your machines.

Processes

at bg chroot cron fg kill killall nice pgrep pidof pkill ps pstree time top

kill

In Unix and Unix-like operating systems, kill is a command used to send a signal to a process. By default, the message sent is the termination signal, which requests that the process exit. But kill is something of a misnomer; the signal sent may have nothing to do with process killing. The kill command is a wrapper around the kill() system call, which sends signals to processes or process groups on the system, referenced by their numeric process IDs (PIDs) or process group IDs (PGIDs). kill is always provided as a standalone utility as defined by the POSIX standard. However, most shells have built-in kill commands that may slightly differ from it.

A process can be sent a SIGTERM signal in four ways (the process ID is '1234' in this case): 

kill 1234
kill -s TERM 1234
kill -TERM 1234
kill -15 1234

The process can be sent a SIGKILL signal in three ways: 

kill -s KILL 1234
kill -KILL 1234
kill -9 1234



killall -u username
  terminate all username's applications to log them out

nohup

  • nohup - Run a command immune to hangups, runs the given command with hangup signals ignored, so that the command can continue running in the background after you log out.

Daemons

A daemon is a program that runs in the background, waiting for events to occur and offering services. A good example is a web server that waits for a request to deliver a page or a ssh server waiting for someone trying to log in. While these are full featured applications, there are daemons whose work is not that visible. Daemons are for tasks like writing messages into a log file (e.g. syslog, metalog) or keeping your system time accurate (e.g. ntpd). For more information see man 7 daemon.

  • Daemon turns other process into daemons. There are many tasks that need to be performed to correctly set up a daemon process. This can be tedious. Daemon performs these tasks for other processes.

Programs

  • Vim - Text editor, etc.
  • Git - Distributed version control

sed

echo "test string oldWord yadayada" | sed 's/oldWord/newWord/g'

find . -name "*.html" -exec sed -i "s/oldWord/newWord/g" '{}' \;
  replace text in multiple files [31]

 echo "<a href="index.html"><img src="logo.svg" id="site-logo"></a>
          <h1>Site Title</h1>" | sed 'N; s@</a>\
          <h1>Site Title</h1>@\
          <h1>Site Title</h1></a>@g'
   multiline replacement

AWK

ack

to sort

  • auditd - userspace component to the Linux Auditing System [32]
  • nice - run a program with modified scheduling priority
nice -n 19 [command]
  run at lowest priority
  • shelr - console screencasting tool

~/.local/share/applications/mimeinfo.cache

If you want to write daemons in languages that can link against C functions (e.g. C, C++), see libslack which contains the core functionality of daemon.

Finding programs

whereis

apropos apropos vim
  search the whatis database for strings

Aliases

From http://news.ycombinator.com/item?id=4867369 :

Tab completion and aliases need not be mutually exclusive. For instance, in zsh you can use: 

alias gs='git status'
compdef _git gs=git-status

FYI: I also mapped g without parameters to `git status -sb` but g with parameters will simply execute everything as normal 

g () {
  if [ $# -eq 0 ]
  then
    git status -sb
  else
    git $*
  fi
}
compdef g=git

Monitoring

See also Server#Monitoring

  • perf (sometimes "Perf Events" or perf tools, originally "Performance Counters for Linux", PCL) - is a user-space performance analyzing tool in Linux with git-like interface and subcommands. It is capable of statistical profiling of entire system (both kernel and user code), single CPU or severals threads. 
time [command]
  return with command execution time

lsof

  • lsof - "lists open files" (lots, given "everything" is a file)
lsof -i :[port]
  what application using a specific port

top

Part of the procps package.

Current stats only, no disk or network. Sorts by cpu by default. A multicore box can show more than 100%. 

z
  colour!
d
  change update interval
return, space
  refresh

H
  show separate threads of each process
i
  show run queue (R) or waiting on disk io (D, red)
1
  show serperate cpus

q
  quit

htop

atop

Shows short lived processes that run and finish between update interval times.

lines;

  • PRC: Total CPU time in system and user mode, total number of processes and of zombie processes, and the number of processes that exited during the polling interval.
  • CPU and CPL: CPU utilization and load (averaged over 1, 5 and 15 minutes).
  • cpu: individual CPU usage.
  • MEM and SWP: Amount of memory and swap space that is available and where it’s allocated. vmcom and vmlim show how much virtual memory space is committed and what the limit is.
  • DSK: disk utilization. avio shows the average number of milliseconds per request.
  • NET: Network utilization for the TCP layer (“transport”), the IP layer (“network”) and each interface.
Ctrl-f
  forward a page
Ctrl-b
  back a page

C
  sort by cpu activity
M
  sort by memory consumption
D
  sort by disk activity
N
  sort by network activity
A
  sort by most active system resource (auto mode)

P
  focus on process (regex)
y
  show all threads (inc. idle)

s
 Scheduling information
m
  Memory consumption
d
  Disk utilization
v
 Variable information
c
  Command line
p
 Accumulated per program
u
  Accumulated per user
n
 Network utilization (patched kernel)

i
  interval timer
t
  trigger update

Other system

  • free - Display amount of free and used memory in the system
  • vpsinfo is a Linux server monitoring script, written in PHP, that provides web access to system status information. It gathers the output from several common Linux commands into one web page, providing a quick overview of the system's current state.
Disk activity

Cron

A cronjob is a task that a Cron system is instructed to run periodically. The crontab file is a configuration file for a user that defines tasks to run under the user’s account. The systab file is a file that specifies cronjobs for the system.

Config

crontab -l
  view crontabs
crontab -e
  edit their crontabs
crontab -r
 remove their crontabs
crontab saved_crontab_filename
  overwrite their old crontab with saved crontab

There are several special predefined values which can be used to substitute the CRON expression. 

Entry                  Description                                 Equivalent To
 @yearly (or @annually) Run once a year, midnight, Jan. 1st         0 0 1 1 *
 @monthly               Run once a month, midnight, first of month  0 0 1 * *
 @weekly                Run once a week, midnight on Sunday         0 0 * * 0
 @daily                 Run once a day, midnight                    0 0 * * *
 @hourly                Run once an hour, beginning of hour         0 * * * *
 @reboot                Run at startup                              @reboot 

*    *    *    *    *  command to be executed
┬    ┬    ┬    ┬    ┬
│    │    │    │    │
│    │    │    │    │
│    │    │    │    └───── day of week (0 - 6) (0 is Sunday, or use names)
│    │    │    └────────── month (1 - 12)
│    │    └─────────────── day of month (1 - 31)
│    └──────────────────── hour (0 - 23)
└───────────────────────── min (0 - 59)
  • /etc/cron.allow - If this file exists, then you must be listed therein (your username must be listed) in order to be allowed to use cron jobs.
  • /etc/cron.deny - If the cron.allow file does not exist but the /etc/cron.deny file does exist, then you must not be listed in the /etc/cron.deny file in order to use cron jobs.

dcron

Vanilla dcron supports asynchronous job processing. Just put it with @hourly, @daily, @weekly or @monthly with a jobname.

dcron has @daily, @hourly and so on. In fcron, you can use standard crontab entries and add &bootrun to the beginning of the line to repeat "missed" cronjobs.

cronie

Arch default cron as dcron wasn't maintained at the time.

dcron and fcron are not under active development, cronie is. cronie is small - 0.20MB installed. developed by Red Hat - it is not going anywhere and we have a guaranteed upgrade path. As far as I can tell cronie has no deps beyond glibc and pam. cronie has /etc/cron.d support and anacron support via an anacrontab config file. cronie extends the original vixie cron package so the syntax, core feature set, etc are stable. implements advanced security hooks as well and can integrate with SELINUX.

anacron

  • anacron is a computer program that performs periodic command scheduling which is traditionally done by cron, but without assuming that the system is running continuously. Thus, it can be used to control the execution of daily, weekly, and monthly jobs (or anything with a period of n days) on systems that don't run 24 hours a day.

fcron

Fcron’s own crontab system uses the fcrontab file for configuration information. The fcrontab syntax is similar but differs slightly from the classic Vixie/ISC Cron crontab notation. fcron lacks /etc/cron.d/ functionality.

Wildcards

MIME

chroot

Startup

  • e4rat ("Ext4 - Reducing Access Times") is a toolset to accelerate the boot process as well as application startups. Through physical file realloction e4rat eliminates both seek times and rotational delays. This leads to a high disk transfer rate. Placing files on disk in a sequentially ordered way allows to efficiently read-ahead files in parallel to the program startup.

ncurses, etc.

Cute

See also GUI#Utils

Windows

  • WINE

OS X

Documentation

man

Time

echo "Today's date is `date -I`"

date
  Local datetime
date -d @1337000000
  Convert unixtime to datetime
date -ud
  Convert unixtime to UTC datetime

ntpdate pool.ntp.org && hwclock --systohc && hwclock --adjust
  Synchronize both your system clock and hardware clock and calculate/adjust time drift.
  Do not run this command if you already have ntpd running! [35]

Swap

swapon -s
free -m

Suspension, hibernation

sudo pm-suspend

Performance

  • http://openbenchmarking.org/ OpenBenchmarking.org is an open, collaborative testing platform designed by Phoronix Media and the developers behind the Phoronix Test Suite, the most comprehensive benchmarking platform for Linux and other operating systems. OpenBenchmarking.org makes the Phoronix Test Suite an even more extensible platform for conducting automated tests with complete integration into Phoronix Test Suite 3.0-Iveland as well as within Phoromatic, an online test remote management system designed for managing test farms within enterprise environments.

Other software

Printing

CUPS

Printing system. 


lp filename
lpr filename

lpstat -p -d

lp -d printer filename
lpr -P printer filename

lpoptions -d printer

program | lp
program | lp -d printer
program | lpr
program | lpr -P printer

Inferno

Retrieved from "https://wiki.thingsandstuff.org/index.php?title=*nix&oldid=20532"