*nix
Mainly Linux.
- Vim - Text editor, etc.
- Git - Distributed version control
- IRC - Internet Relat Chat
- Bitlbee - Instant Messaging
- Security
Guides
- The Art of Unix Programming Eric Steven Raymond
- The Linux Command Line - A Book By William E. Shotts, Jr.
Articles
- I introduced my 5-year-old and 2-year-old to startx and xmonad. They’re DELIGHTED! - June 20th, 2012
- http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html
- http://www.pgrs.net/2007/09/06/useful-unix-tricks/
Dotfiles
System
Standards
- POSIX, an acronym for "Portable Operating System Interface", is a family of standards specified by the IEEE for maintaining compatibility between operating systems. POSIX defines the application programming interface (API), along with command line shells and utility interfaces, for software compatibility with variants of Unix and other operating systems.
- Linux Standard Base (LSB) is a joint project by several Linux distributions under the organizational structure of the Linux Foundation to standardize the software system structure, including the filesystem hierarchy, used with Linux operating system. The LSB is based on the POSIX specification, the Single UNIX Specification, and several other open standards, but extends them in certain areas.
- freedesktop.org is open source / open discussion software projects working on interoperability and shared technology for X Window System desktops. The most famous X desktops are GNOME and KDE, but developers working on any Linux/UNIX GUI technology are welcome to participate. freedesktop.org is building a base platform for desktop software on Linux and UNIX. The elements of this platform have become the backend for higher-level application-visible APIs such as Qt, GTK+, XUL, VCL, WINE, GNOME, and KDE. The base platform is both software and specifications.
Boot
- e4rat - reduce boot time (into X) by some 50% for ext4
"In order to boot Arch Linux, a Linux-capable boot loader such as GRUB(2), Syslinux, LILO or GRUB Legacy must be installed to the Master Boot Record or the GUID Partition Table. The boot loader is responsible for loading the kernel and initial ramdisk before initiating the boot process."
- https://wiki.archlinux.org/index.php/Arch_Boot_Process
- https://wiki.archlinux.org/index.php/Init_and_inittab
init is the first process that is executed once the Linux kernel loads. The default init program Arch uses is /sbin/init provided by sysvinit. The word init will always refer to sysvinit in this article. inittab is the startup configuration file for init located in /etc. It contains directions for init on what programs and scripts to run when entering a specfic runlevel.
Monitoring
- lsof - "lists open files" (lots, given "everything" is a file)
lsof -i :[port] what application using a specific port
- ps aux - list all processes
- pstree - ascii process tree
- free - Display amount of free and used memory in the system
UNIX =
Linux
- http://studyhat.blogspot.co.uk/2009/12/identifying-linux-bottlenecks.html
- modprobe is a Linux program originally written by Rusty Russell and used to add a loadable kernel module (LKM) to the Linux kernel or to remove an LKM from the kernel. It is commonly used indirectly: udev relies upon modprobe to load drivers for automatically detected hardware.
BSD
- http://openindiana.org/ - solaris based
- http://smartos.org/
- https://www.illumos.org/projects/illumian
Terminal
Basics
- Wikipedia:Pseudo terminal, Wikipedia:Virtual console (computer user-interface), Wikipedia:Terminal emulator, Wikipedia:Control character, Wikipedia:Escape sequence, Wikipedia:Escape character, Wikipedia:C0 and C1 control codes, Wikipedia:Terminfo
- ASCII Characters for MPE Users (control char info)
Shift-Insert paste clipboard
See also Typography#Terminal
Xterm
- How to change the title of an xterm - This document explains how to use escape sequences to dynamically change window and icon titles of an xterm. Examples are given for several shells, and the appendix gives escape sequences for some other terminal types.
Urxvt
if not using a tiling window manager, tabbed is good + remote tmux. urxvt has overhead because of unicode.
- http://linux.die.net/man/1/urxvtd - a backend daemon. lighter.
urxvtd -q -f -o
- http://linux.die.net/man/1/urxvtc - client
- Arch Forum: [Solved Change urxvt's font on the fly: Can YOU do it?]
printf '\33]50;%s\007' "xft:Terminus:pixelsize=16"
! :Bind C-0, C-+ and C-= to activate small, medium, and big font size resp. URxvt.keysym.C-0: command:\033]710;-*-dina-medium-r-normal--13-*-*-*-*-*-*-*\007 URxvt.keysym.C-minus: command:\033]710;-*-dina-medium-r-normal--15-*-*-*-*-*-*-*\007 URxvt.keysym.C-equal: command:\033]710;-*-dina-medium-r-normal--16-*-*-*-*-*-*-*\007
Screen
config goes in ~/.screenrc
escape ^Ww change escape key to w
Tmux
Better than screen, if available.
Config goes in ~/.tmux.conf, which can be symlinked to a hidden git repo folder.
Articles
- TMUX – The Terminal Multiplexer (Part 1) - June 28, 2010
- TMUX – The Terminal Multiplexer (Part 2) - July 2, 2010
- switching from gnu screen to tmux (updated) - May 5, 2010
Commands
tmux lsc list clients tmux detach-client -t /dev/pts/26 remove other clients from session (if screensize is fucked)
split-window
Additions
- wemux enhances tmux to make multi-user terminal multiplexing both easier and more powerful. It allows users to host a wemux server and have clients join in either:
- https://github.com/lmartinking/tmux-applets
- https://github.com/aziz/tmuxinator
- https://github.com/remiprev/teamocil
Misc
- ArchWiki: Improve Boot Performance#TTY terminal management
- StackOverflow: Can terminals detect <Shift-Enter> or <Control-Enter>?
- Home and End keys not working
- That 256 Color Thing, P.C. Shyamshankar
- Super User: TTY with 256 colors?
- The Text Pistols
- Wikipedia:Plan 9 from Bell Labs
- http://awesome.naquadah.org/wiki/Terminals
- Gate One is an HTML5-powered terminal emulator and SSH client]
- http://www.brain-dump.org/projects/dvtm/ - ncurses tiling manager
Mouse
- gpm is the mouse support for Linux on the console
Unicode
- The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky
- Codepoint, n. the position of a character in an encoding system.
- Charbase - A visual unicode database
- http://www.charset.org/
- http://unicode.org/charts/
- http://sheet.shiar.nl/unicode
- http://en.wikipedia.org/wiki/List_of_Unicode_characters
= Shell
/etc/shells
Bash
- Bash Reference Manual
- Builtin Commands
- Shell variables: Bourne, Bash
- Conditional Expressions
- Bash Guide for Beginners
- Advanced Bash-Scripting Guide by Mendel Cooper
- .bashrc, etc
- Getting Started with BASH
- The Command Line Crash Course (for cli newbs)
- Writing Robust Bash Shell Scripts
- http://blog.commandlinekungfu.com/p/index-of-tips-and-tricks.html Unix Command-Line Kung Fu] Tip index
- http://mywiki.wooledge.org/BashGuide/Practices
- http://www.aboutlinux.info/2005/10/10-seconds-guide-to-bash-shell.html
man: echo
Basics
- IBM - Linux tip: Bash parameters and parameter expansions
- Debugging a script Bash Hackers Wiki
- Bash Initialisation Files
Options
- http://rsalveti.wordpress.com/2007/04/03/bash-parsing-arguments-with-getopts/
- http://wiki.bash-hackers.org/howto/getopts_tutorial
- shFlags - getop wrapper for long flags with fallback for non gnu-getopt
Etc.
$_ last entered word
$? returned exit code of last exec
Zsh
- Github: P.C. Shyamshankar's Zsh configs
- https://gist.github.com/914831
- https://github.com/olivierverdier/zsh-git-prompt or https://github.com/jcorbin/zsh-git
- http://www.techrepublic.com/blog/opensource/enhanced-history-searching-in-zsh/1347
- https://github.com/zsh-users/zsh-history-substring-search
- http://quasimal.com/posts/2012-05-21-funsh.html - functional programming in zsh
Configuration
lshell
- lshell is a shell coded in Python, that lets you restrict a user's environment to limited sets of commands, choose to enable/disable any command over SSH (e.g. SCP, SFTP, rsync, etc.), log user's commands, implement timing restriction, and more.
git-shell
- git-shell - Restricted login shell for Git-only SSH access
Inferno
- Inferno is a distributed operating system, originally developed at Bell Labs, but now developed and maintained by Vita Nuova® as Free Software. Applications written in Inferno's concurrent programming language, Limbo, are compiled to its portable virtual machine code (Dis), to run anywhere on a network in the portable environment that Inferno provides. Unusually, that environment looks and acts like a complete operating system.
Fish
Other
- Xiki: A shell console with GUI features. Xiki does what shell consoles do, but lets you edit everything at any time. It's trivial to make your own commands and menus to access other tools. [5]
Files
File systems
Ext
- http://en.wikipedia.org/wiki/Ext3 - w/ journaling
- http://en.wikipedia.org/wiki/Ext4 - (ext3.5)
Btrfs
See also Backup#Btrfs
General
- http://en.wikipedia.org/wiki/Btrfs
- https://wiki.archlinux.org/index.php/Btrfs
- https://btrfs.wiki.kernel.org/index.php/Main_Page
- http://www.funtoo.org/wiki/BTRFS_Fun
- https://help.ubuntu.com/community/btrfs
- http://www.rkeene.org/projects/info/wiki.cgi/165 - terminology
Subvolumes appear like directories. inode is different.
"Btrfs support is included in the linux package (as a module). Needs a reboot after installing before btrfs recognised. User space utilities are available in btrfs-progs. For multi-devices support (RAID like feature of btrfs) aka btrfs volume in early boot, you have to enable btrfs mkinitcpio hook (provided by mkinitcpio package) to be able to use, for example, a root btrfs volume. If the btrfs volume is a non-system volume, one only needs to set USEBTRFS="yes" in /etc/rc.conf. However, if you only use bare btrfs partition, such options are not needed."
"The btrfs scrub command reads redundant data and validates all the checksums, correcting any errors it finds along the way, using the checksum to determine which copy is the valid one. But with a single drive, how can it correct anything? The metadata - the file system overhead that is used to manage your data - is always stored in a redundant manner by default, even on a single drive. As a result, any corrupted metadata can be corrected, on the fly."
"EXT4 checksums its journal, which AFAIK will protect against errors caused by sync failures (ie. power failure during disk I/O). But it’s not going to protect against latent sector errors. To do that, you need checksumming on all the file data, along the lines of what ZFS or BTRFS provides."
A cross-subvolume copy patch has made it into 3.6_rc. This patch will allow cp --reflink across subvolumes, as long as the copy does not cross mount points.
Commands
mkfs.btrfs -L [label] /dev/[device] btrfs filesystem df /media/[drive] btrfs filesystem show btrfs-debug-tree -R /dev/sdg show drive/subvolume infos, unmounted btrfs subvolume create [<dest>/] btrfs subvolume snapshot /mnt/btrfs /mnt/btrfs/snapshot_of_root btrfs subvolume delete [<dest>/] mount -o remount,compress=lzo /dev/sdXY /mnt/target btrfs filesystem defragment /
btrfs device add /dev/sdc /mnt/btrfs
Cloning a file between subvolumes;
cp --reflink /mnt/MYFILES/myfile1 /mnt/MYFILES/myfile3
copy-on-write, without the ram requirement of zsf snapshots every 30 seconds, ability to mount from previous gen
GUI
- btrfs-gui is a graphical user interface tool for inspecting and managing btrfs filesystems. It is capable of managing filesystems on the local machine, and filesystems on remote network-accessible machines. It requires root access to the machine to perform most of its tasks (but separates the root-access part from the GUI).
- Snapper is a tool for managing btrfs snapshots. Apart from the obvious creation and deletion of snapshots it can compare snapshots and revert differences between snapshots. In simple terms, this allows users to view older versions of files and revert changes. Snapper is available as a command line interface tool and a YaST module. Both make use of the C++ library libsnapper which is also available to other programs.
Articles
- http://www.oracle.com/technetwork/articles/servers-storage-admin/gettingstarted-btrfs-1695246.html
- http://www.oracle.com/technetwork/articles/servers-storage-admin/advanced-btrfs-1734952.html
- http://www.linux.com/learn/tutorials/371623-weekend-project-get-started-with-btrfs
- https://bbs.archlinux.org/viewtopic.php?id=146150
- http://askubuntu.com/questions/124075/how-to-make-a-btrfs-snapshot
- Using btrfs snapshots for incremental backup - Feb 1, 2010
- http://newbrightidea.com/2010/12/13/home-nas-with-btrfs/
- http://permalink.gmane.org/gmane.comp.file-systems.btrfs/13767
- http://comments.gmane.org/gmane.comp.file-systems.btrfs/16412
Other
- Opendedup Develops SDFS, a file-system that does inline deduplication.
RAID
LVM
- http://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)
- https://wiki.archlinux.org/index.php/LVM
ZFS
- http://en.wikipedia.org/wiki/ZFS - GPL incompatibility, CDDL license, Sun
- https://wiki.ubuntu.com/ZFS/ZPool
"FreeBSD ZFS tuning guide wiki indicates you'll need about 5GB of ram per 1TB of saved disk space"
- http://rudd-o.com/linux-and-free-software/ways-in-which-zfs-is-better-than-btrfs
- http://www.unixconsult.org/zfs_vs_lvm.html
- https://wiki.archlinux.org/index.php/Installing_Arch_Linux_on_ZFS
- Arch Forum: ZFS-FUSE vs. BTRFS, for an arch backup RAID array
- https://wiki.archlinux.org/index.php/User_talk:Wolfdogg
- http://news.ycombinator.com/item?id=4122937
- http://zfsonlinux.org
- http://strotmann.de/roller/cas/entry/timemachine_ish_backup_with_zfs
- http://www.thegeekprophet.com/zfs-usb-mount-linux-mint
- http://icesquare.com/wordpress/zfsusb-building-a-super-large-server-using-usb-memory-cf-card-and-sd-card/
File structures
See LSB, etc.
- Linux Directory Structure (File System Structure) Explained with Examples
- http://hivelogic.com/articles/using_usr_local/
- Point/Counterpoint - /opt vs. /usr/local - March 2010
- Understanding the bin, sbin, usr/bin , usr/sbin split [6] - Dec 9, 2010
- -arch-dev-public- -RFC- merge /bin, /sbin, /lib into /usr/bin and /usr/lib - Mar 2nd, 2012
- on ., .., .dotfiles - Rob Pike, Aug 3rd, 2012
Mount
lsbkl cat /proc/partitions blkid
sd[a,b,etc] drive sda[1,2,etc] partition of drive
mount /dev/sdxY /some/directory umount /some/directory mount -o remount / remount partition after /etc/fstab change
Partitions
- Arch Wiki: Partitioning
- GNU Parted manipulates partition tables. This is useful for creating space for new operating systems, reorganizing disk usage, copying data on hard disks and disk imaging. The package contains a library, libparted, as well as well as a command-line frontend, parted, which can also be used in scripts.
- http://gparted.sourceforge.net/
dd
- dd - Copy a file, converting and formatting according to the options.
- dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data.
dd if=/dev/sr0 of=myCD.iso bs=2048 conv=noerror,sync create an ISO disk image from a CD-ROM. dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=noerror Clone one partition to another dd if=/dev/ad0 of=/dev/ad1 bs=1M conv=noerror Clone a hard disk "ad0" to "ad1".
dd if=/dev/zero bs=1024 count=1000000 of=file_1GB dd if=file_1GB of=/dev/null bs=64k drive benchmark test and analyze the sequential read and write performance for 1024 byte blocks
du (disk usage)
du -sh size of a folder du -S size of files in a folder du -aB1m|awk '$1 >= 100' everything over 100Mb
cd / | sudo du -khs * show root folder size sudo du -a --max-depth=1 /usr/lib | sort -n -r | head -n 20 size of program folders /usr/lib du -sk ./* | sort -nr | awk 'BEGIN{ pref[1]="K"; pref[2]="M"; pref[3]="G";} { total = total + $1; x = $1; y = 1; while( x > 1024 ) { x = (x + 1023)/1024; y++; } printf("%g%s\t%s\n",int(x*10)/10,pref[y],$2); } END { y = 1; while( total > 1024 ) { total = (total + 1023)/1024; y++; } printf("Total: %g%s\n",int(total*10)/10,pref[y]); }'
ncdu
- ncdu - ncurses disk usage
ncdu / --exclude /home --exclude /media --exclude /run/media check everything apart from home and external drives ncdu / --exclude /home --exclude /media --exclude /run/media check everything apart from external drives
ncdu / --exclude /home --exclude /media --exclude /run/media --exclude /boot --exclude /tmp --exclude /dev --exclude /proc just the root partition
df
- df - report file system disk space usage
Baobab
- Baobab - gnome app
Other
todo; source aliases.zsh
- Filelight creates an interactive map of concentric, segmented rings that help visualise disk usage on your computer.
Disk activity
badly sorted
- http://linux.die.net/man/1/dstat
- http://linux.die.net/man/1/iostat
- http://linux.die.net/man/8/vmstat (virtual mem)
- http://linux.die.net/man/1/sar
- http://linuxpoison.blogspot.co.uk/2009/02/how-to-measure-and-read-disk-activity.html
- iotop
- nmon
- http://collectl.sourceforge.net/Process.html
- http://sebastien.godard.pagesperso-orange.fr/
Files and directories
ls list in row ls -l long list ls * files in directory and immediate subdiretories
just names;
ls -m1 -m fill width with a comma separated list of entries ?? ls --format single-column column of names only ls -l | grep - | awk '{print $9}' using awk to show the 9th word (name). strips colour. ls -l | cut -f9 -s -d" " using cut to cut from the 9th word, using space as a delimiter. strips colour. ls | cat neat ls -a show hidden files ls -A show hidden files, exclude . and ..
stat . display file or file system status stat -c "%n %a" * | column -t directory files + octal
mkdir directory mkdir directory -p no error if existing, make parent directories as needed
ln -s {target-filename} ln -s {target-filename} {symbolic-filename} create soft link
find . -type f -links +1 list hard links
cd change/directory/path
mv position1 ~/position2 basic move
- http://superuser.com/questions/187866/unix-shell-scripting-how-to-recursively-move-files-up-one-directory
- http://serverfault.com/questions/122233/how-to-recursively-move-all-files-including-hidden-in-a-subfolder-into-a-paren
pax - read and write file archives and copy directory hierarchies
find . -type f -print0 | xargs -0 stat -c "%y %s %n"
v def conf => vim /some/awkward/path/to/type/default.conf j abc => cd /hell/of/a/awkward/path/to/get/to/abcdef m movie => mplayer /whatever/whatever/whatever/awesome_movie.mp4 o eng paper => xdg-open /you/dont/remember/where/english_paper.pdf vim `f rc lo` => vim /etc/rc.local vim `f rc conf` => vim /etc/rc.conf
alias defaults;
alias a='fasd -a' # any alias s='fasd -si' # show / search / select alias d='fasd -d' # directory alias f='fasd -f' # file alias sd='fasd -sid' # interactive directory selection alias sf='fasd -sif' # interactive file selection alias z='fasd_cd -d' # cd, same functionality as j in autojump alias zz='fasd_cd -d -i' # cd with interactive selection
File types
xdg-mime default Thunar.desktop inode/directory to make Thunar the default file-browser xdg-mime default xpdf.desktop application/pdf to use xpdf as the default PDF viewer
/usr/share/applications/defaults.list (global) ~/.local/share/applications/defaults.list (per user, overrides global) [Default Applications] mimetype=desktopfile1;desktopfile2;...;desktopfileN
Copying files
cp - copy files and directories
scp -P 2264 foobar.txt your_username@remotehost.edu:/some/remote/directory scp -rP 2264 folder your_username@remotehost.edu:/some/remote/directory
wget -O myzip.zip https://github.com/zeromq/jzmq/zipball/master wget -k --convert-links
Viewing files
cat
cat filename output file to screen cat -n filename output file to screen w/ line numbers cat filename1 filename2 output two files (concatinate) cat filename1 > filename2 overwrite filename2 with filename1 cat filename1 >> filename2 append filename1 to filename2 cat filename{1,2} > filename2 add filename1 and filename2 together into filename3
etc
head filename top 10 lines of file head -23 filename top 23 lines of file tail filename bottom 10 lines of file tail -23 filename bottom 23 lines of file
more is a filter for paging through text one screenful at a time. This version is especially primitive. Users should realize that less(1) provides more(1) emulation plus extensive enhancements.
sed -n 20,30p filename print lines 20..30 of file [9]
Finding files
find /usr/share -name README find ~/Journalism -name '*.txt' find ~/Programming -path '*/src/*.c' find ~/Images/Screenshots -size +500k -iname '*.jpg' find ~/Journalism -name '*.txt' -exec cat {} ; find ~/Journalism -name '*.txt' -print0 | xargs -0 cat (faster than above) find / -group [group] find / -user [user]
- sgrep - search a file for a structured pattern
Compression
tar
tar <operation> [options] Operations: [-]A --catenate --concatenate [-]c --create [-]d --diff --compare [-]r --append [-]t --list [-]u --update [-]x --extract --get --delete Common Options: -C, --directory DIR -f, --file F -j, --bzip2 -p, --preserve-permissions -v, --verbose -z, --gzip
zip
gzip
bzip
7z
- 7-Zip is a file archiver with the highest compression ratio. The program supports 7z (that implements LZMA compression algorithm), ZIP, CAB, ARJ, GZIP, BZIP2, TAR, CPIO, RPM and DEB formats. Compression ratio in the new 7z format is 30-50% better than ratio in ZIP format.
- p7zip is a port of 7za.exe for POSIX systems like Unix (Linux, Solaris, OpenBSD, FreeBSD, Cygwin, AIX, ...), MacOS X and also for BeOS and Amiga. 7za.exe is the command line version of 7-zip, see http://www.7-zip.org/. 7-Zip is a file archiver with highest compression ratio.
- man z7 (p7zip)
- p7zip-light in AUR
7z x filename extract archive with directories
xz
pax
- pax will read, write, and list the members of an archive file, and will copy directory hierarchies. pax operation is independent of the specific archive format, and supports a wide variety of different archive formats. A list of supported archive formats can be found under the description of the -x option. [10]
Other
Generic function
# Extract Files extract() { if [ -f $1 ] ; then case $1 in *.tar.bz2) tar xvjf $1 ;; *.tar.gz) tar xvzf $1 ;; *.tar.xz) tar xvJf $1 ;; *.bz2) bunzip2 $1 ;; *.rar) unrar x $1 ;; *.gz) gunzip $1 ;; *.tar) tar xvf $1 ;; *.tbz2) tar xvjf $1 ;; *.tgz) tar xvzf $1 ;; *.zip) unzip $1 ;; *.Z) uncompress $1 ;; *.7z) 7z x $1 ;; *.xz) unxz $1 ;; *.exe) cabextract $1 ;; *) echo "\`$1': unrecognized file compression" ;; esac else echo "\`$1' is not a valid file" fi }
Commands
Resources
- commandlinefu.com [11]
- LINFO: Index of Linux Commands
- Awesome but often unknown Linux commands and tools - August 10th, 2011
- LinuxCommand.org is a Linux education and advocacy site devoted to helping users of legacy operating systems migrate into the future.
- WikiBooks: Guide to Unix/Commands
- SHELLdorado - your UNIX shell scripting resource
Use output as input.
cat `locate file.txt`
or
cat $(locate file.txt)
todo; piping and redirection
- inotail is a replacement for the 'tail' program found in the base installation of every Linux/UNIX system. It makes use of the inotify infrastructure in recent versions of the Linux kernel to speed up tailing files in the follow mode (the '-f' option). Standard tail polls the file every second by default while inotail listens to special events sent by the kernel through the inotify API to determine whether a file needs to be reread.
Currently inotail is not fully compatible to neither POSIX or GNU tail but might be in the future.
Programs
- World's best introduction to sed [12]
- awk
- ack - ack is a perl tool like grep for programmers
- https://github.com/rupa/z / https://github.com/joelthelion/autojump - verrry handy for moving about
- nice - run a program with modified scheduling priority
- cut - remove sections from each line of files
- shelr - console screencasting tool
~/.local/share/applications/mimeinfo.cache
- http://furius.ca/xxdiff/
- http://meldmerge.org/ - wow, nice three pane diff merging gui app
- http://diffuse.sourceforge.net/screenshots.html
- http://caca.zoy.org/wiki/toilet - like figlet but w/ colours
Finding programs
whereis
apropos apropos vim search the whatis database for strings
Cron
- Wikipedia: cron
- Arch Wiki: cron
- Gentoo Linux Cron Guide
A cronjob is a task that a Cron system is instructed to run periodically. The crontab file is a configuration file for a user that defines tasks to run under the user’s account. The systab file is a file that specifies cronjobs for the system.
Config
crontab -l view crontabs crontab -e edit their crontabs crontab -r remove their crontabs crontab saved_crontab_filename overwrite their old crontab with saved crontab
There are several special predefined values which can be used to substitute the CRON expression.
Entry Description Equivalent To @yearly (or @annually) Run once a year, midnight, Jan. 1st 0 0 1 1 * @monthly Run once a month, midnight, first of month 0 0 1 * * @weekly Run once a week, midnight on Sunday 0 0 * * 0 @daily Run once a day, midnight 0 0 * * * @hourly Run once an hour, beginning of hour 0 * * * * @reboot Run at startup @reboot
* * * * * command to be executed ┬ ┬ ┬ ┬ ┬ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───── day of week (0 - 6) (0 is Sunday, or use names) │ │ │ └────────── month (1 - 12) │ │ └─────────────── day of month (1 - 31) │ └──────────────────── hour (0 - 23) └───────────────────────── min (0 - 59)
- /etc/cron.allow - If this file exists, then you must be listed therein (your username must be listed) in order to be allowed to use cron jobs.
- /etc/cron.deny - If the cron.allow file does not exist but the /etc/cron.deny file does exist, then you must not be listed in the /etc/cron.deny file in order to use cron jobs.
- http://xyne.archlinux.ca/projects/cronwhip/ - run cron tasks that were missed due to power down. uses crond config unlike anacron.
dcron
Vanilla dcron supports asynchronous job processing. Just put it with @hourly, @daily, @weekly or @monthly with a jobname.
dcron has @daily, @hourly and so on. In fcron, you can use standard crontab entries and add &bootrun to the beginning of the line to repeat "missed" cronjobs.
cronie
Arch default cron as dcron wasn't maintained at the time.
dcron and fcron are not under active development, cronie is. cronie is small - 0.20MB installed. developed by Red Hat - it is not going anywhere and we have a guaranteed upgrade path. As far as I can tell cronie has no deps beyond glibc and pam. cronie has /etc/cron.d support and anacron support via an anacrontab config file. cronie extends the original vixie cron package so the syntax, core feature set, etc are stable. implements advanced security hooks as well and can integrate with SELINUX.
anacron
- anacron is a computer program that performs periodic command scheduling which is traditionally done by cron, but without assuming that the system is running continuously. Thus, it can be used to control the execution of daily, weekly, and monthly jobs (or anything with a period of n days) on systems that don't run 24 hours a day.
fcron
- fcron is a computer program with a GNU GPL license that performs periodic command scheduling. It has been developed on Linux and should work on POSIX systems. As with Anacron, it does not assume that the system is running continuously, and can run in systems that do not run all the time or regularly. It aims to replace Vixie-cron and Anacron with a single integrated program, providing many features missing from the original Cron daemon.
Fcron’s own crontab system uses the fcrontab file for configuration information. The fcrontab syntax is similar but differs slightly from the classic Vixie/ISC Cron crontab notation. fcron lacks /etc/cron.d/ functionality.
Wildcards
- tuxfiles.org: wildcards
MIME
chroot
- http://linux.die.net/man/1/chroot
- http://en.wikipedia.org/wiki/Chroot
- https://wiki.archlinux.org/index.php/Change_Root
- https://help.ubuntu.com/community/BasicChroot
- http://www.unixwiz.net/techtips/chroot-practices.html
Startup
- e4rat ("Ext4 - Reducing Access Times") is a toolset to accelerate the boot process as well as application startups. Through physical file realloction e4rat eliminates both seek times and rotational delays. This leads to a high disk transfer rate. Placing files on disk in a sequentially ordered way allows to efficiently read-ahead files in parallel to the program startup.
ncurses, etc.
Users
- /etc/passwd - local user information
- account:password:UID:GID:GECOS:directory:shell
- /etc/shadow - restricted access encrypted password file
useradd username create user with defaults (no password) useradd -D show defaults that will be used useradd -m -g [initial_group] -G [additional_groups] -s [login_shell] [username] -m - add home dir -d - non-default home dir path -c comment -e 2006-06-30 - expires useradd -m -g users -G audio,lp,optical,storage,video,wheel,games,power,scanner -s /bin/bash username
adduser interactive tool
userdel username remove user userdel -r username remove user plus their home folder, mail spool
cat /etc/passwd | cut -d: -f 1,3,6 | grep "[5-9][0-9][0-9]" List users on a system [14]
- last, lastb - show listing of last logged in users
Groups
groupadd [group] add group gpasswd -a [user] [group] add user to group gpasswd -d [user] [group] delete user from group
User must relogin for new group to take effect.
Permissions
- chmod - change file mode bits
- Chmod, Umask, Stat, Fileperms, and File Permissions - 0000 to 0777 list, etc.
sudoers
ACLs
Partition must have acl set in /etc/fstab (and be remounted after).
setfacl -m "u:username:permissions" setfacl -m "u:uid:permissions" add permissions for user setfacl -m "g:groupname:permissions" setfacl -m "g:gid:permissions" add permissions for group setfacl -m "u:user:rwx" file add read, write, execure perms for user for file setfacl -Rm "u:user:rw" /dir add recursive read, write perms for user for dir setfacl -Rdm "u:user:rw" /dir add recursive read, write perms for user for dir and make them default for future changes
Session permission management
For non-root mount permissions, etc.
- https://wiki.archlinux.org/index.php/ConsoleKit
- https://wiki.archlinux.org/index.php/Systemd
- https://wiki.archlinux.org/index.php/Udev
- https://wiki.archlinux.org/index.php/Udiskie
- https://wiki.archlinux.org/index.php/Polkit
Networking
sudo /etc/init.d/<service> restart ubuntu, restart a service sudo /etc/rc.d/<service> stop | start | restart arch, service things
sudo sh -c "echo 'something' >> /etc/privilegedfile" chown -R user:group . - change all and subitems [15]
route show network routes avaliable route -n show network routes avaliable, just ip
- Arch Wiki: Configuring Network
- Linux DNS Lookup Command - host & dig
- route -n - display the host's networks and gateway
- /etc/hosts
- /etc/network/interfaces - ubuntu network interface settings
- /etc/resolv.conf - dns settings
- /etc/host.conf - dns resolve order
- /etc/dhcp3/dhclient.conf - overrides ubuntu server dns settings. change prepend option.
- /etc/dnsmasq.conf - dnsmasq settings
- dnsmasq configuration
- Local DNS cache in Linux using dnsmasq
- View The Daily Show, etc. in the UK, etc. Mofity HTTP headers; X-Forwarded-For "12.13.14.15" [17]
curl
- curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.
curl http://www.google.com/search.js -o /path/to/local/file.js
curl http://site.{one,two,three}.com
curl ftp://ftp.numericals.com/file[1-100].txt ftp://ftp.numericals.com/file[001-100].txt (with leading zeros) ftp://ftp.letters.com/file[a-z].txt sequences of alphanumeric series by using []
curl http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html Nested sequences are not supported, but you can use several ones next to each other:
curl http://www.numericals.com/file[1-100:10].txt http://www.letters.com/file[a-z:2].txt multiple urls + specify a step counter for the ranges to get every Nth number or letter:
ftp
nmap
nmap -sT -sU -O -p 1-65535 localhost full port scan
VOIP
File sharing
Media
Package management
Apt
dpkg --get-selections > installed-software create list of installed software dpkg --set-selections < installed-software dselect reinstall from list
dpkg --list to check
Pacman
pacman -Sy [package] install [package] pacman -Syu upgrade system pacman -Qo [file] check what package owns a file pacman -Qi list installed programms
pacman -Qqtd check whether there are any orphaned packages pacman -Rsn packagename remove orphaned packages pacman -Rc packagename remove package and deps ('cascade')
- cacheclean - Cleans up pacman packages. Users selects how many old versions to keep.
cacheclean {-p} {-v} <# of copies to keep> # of copies to keep - (required) how many generations of each package. -p - (optional) preview what would be deleted. -v - (optional) show deleted packages.
for i in `pacman -Qdt | awk '{print $1}'`; do pacman -R $i ; done
Helpers
- packer - all repos
- aurget - slow, aur only
Packages
makepkg -g >> PKGBUILD && makepkg
"ERROR: PKGBUILD contains CRLF characters and cannot be sourced." Solution:
sed -i 's/^M//' PKGBUILD
[ctrl+v][ctrl+m] for the ^M symbol.
Other
- http://www.nongnu.org/archup/ - on screen notifications
- http://kmkeen.com/pacmatic/ - e-mail notifications
Documentation
man
Time
echo "Today's date is `date -I`"
date Local datetime date -d @1337000000 Convert unixtime to datetime date -ud Convert unixtime to UTC datetime
- Linux, Clocks, and Time
- tzconfig / dpkg-reconfigure tzdata
ntpdate pool.ntp.org && hwclock --systohc && hwclock --adjust Synchronize both your system clock and hardware clock and calculate/adjust time drift. Do not run this command if you already have ntpd running! [19]
Swap
swapon -s free -m
Suspension, hibernation
Other software
- http://kmandla.wordpress.com/software/
- http://blog.chavezgu.com/2012/03/07/the-command-line-challenge/
- shortcutworld.com - keyboard shortcuts wiki
- https://www.shortcutfoo.com/
CUPS
Printing system.
Regex
- http://refiddle.com/
- http://regex.learncodethehardway.org/book/
- http://www.regextester.com/index2.html
- http://myregextester.com/
- http://regexpal.com/
Cisco
Storage
Plan 9
Unix done better.