*nix

From Things and Stuff Wiki
Revision as of 08:56, 29 August 2012 by Milk (talk | contribs) (→‎Networking)
Jump to navigation Jump to search


Mainly linux, some unix-like.

Guides

Dotfiles

System

Standards

  • POSIX, an acronym for "Portable Operating System Interface", is a family of standards specified by the IEEE for maintaining compatibility between operating systems. POSIX defines the application programming interface (API), along with command line shells and utility interfaces, for software compatibility with variants of Unix and other operating systems.
  • Linux Standard Base (LSB) is a joint project by several Linux distributions under the organizational structure of the Linux Foundation to standardize the software system structure, including the filesystem hierarchy, used with Linux operating system. The LSB is based on the POSIX specification, the Single UNIX Specification, and several other open standards, but extends them in certain areas.
  • freedesktop.org is open source / open discussion software projects working on interoperability and shared technology for X Window System desktops. The most famous X desktops are GNOME and KDE, but developers working on any Linux/UNIX GUI technology are welcome to participate. freedesktop.org is building a base platform for desktop software on Linux and UNIX. The elements of this platform have become the backend for higher-level application-visible APIs such as Qt, GTK+, XUL, VCL, WINE, GNOME, and KDE. The base platform is both software and specifications.

Boot

  • e4rat - reduce boot time (into X) by some 50% for ext4

"In order to boot Arch Linux, a Linux-capable boot loader such as GRUB(2), Syslinux, LILO or GRUB Legacy must be installed to the Master Boot Record or the GUID Partition Table. The boot loader is responsible for loading the kernel and initial ramdisk before initiating the boot process."

Monitoring

  • lsof - "lists open files" (lots, given "everything" is a file)
lsof -i :[port]
  what application using a specific port

ps aux
  list all processes
pstree
  ascii process tree

iostat, vmstat, free

Linux

BSD

Terminals

Basics

Shift-Insert
  paste x clipboard

Unicode

Urxvt

if not using a tiling window manager, tabbed is good + remote tmux. 

urxvtd -q -f -o

Screen

config goes in ~/.screenrc 

escape ^Ww
  change escape key to w

Tmux

Better than screen, if available.

Config goes in ~/.tmux.conf, which can be symlinked to a hidden git repo folder.

Articles

Commands

tmux lsc
  list clients

tmux detach-client -t /dev/pts/26
  remove other clients from session (if screensize is fucked)

split-window

Additions

  • wemux enhances tmux to make multi-user terminal multiplexing both easier and more powerful. It allows users to host a wemux server and have clients join in either:

Misc

Shell

$?
  returned exit code of last exec

Bash

man: echo

Basics

Options

More

Zsh

Configuration

lshell

  • lshell is a shell coded in Python, that lets you restrict a user's environment to limited sets of commands, choose to enable/disable any command over SSH (e.g. SCP, SFTP, rsync, etc.), log user's commands, implement timing restriction, and more.

git-shell

  • git-shell - Restricted login shell for Git-only SSH access

Inferno

  • Inferno is a distributed operating system, originally developed at Bell Labs, but now developed and maintained by Vita Nuova® as Free Software. Applications written in Inferno's concurrent programming language, Limbo, are compiled to its portable virtual machine code (Dis), to run anywhere on a network in the portable environment that Inferno provides. Unusually, that environment looks and acts like a complete operating system.

Fish

Other

Files

File systems

Ext

Btrfs

General

copy-on-write, without the ram requirement of zsf snapshots every 30 seconds, ability to mount from previous gen

  • Snapper is a tool for managing btrfs snapshots. Apart from the obvious creation and deletion of snapshots it can compare snapshots and revert differences between snapshots. In simple terms, this allows users to view older versions of files and revert changes. Snapper is available as a command line interface tool and a YaST module. Both make use of the C++ library libsnapper which is also available to other programs.

Other

  • Opendedup Develops SDFS, a file-system that does inline deduplication.

RAID

LVM

ZFS

"FreeBSD ZFS tuning guide wiki indicates you'll need about 5GB of ram per 1TB of saved disk space"

File structures

See LSB, etc.

Mount

lsbkl
cat /proc/partitions

sd[a,b,etc]
  drive
sda[1,2,etc]
  partition of drive

mount /dev/sdxY /some/directory

umount /some/directory

mount -o remount /
  remount partition after /etc/fstab change

Partitions

  • GNU Parted manipulates partition tables. This is useful for creating space for new operating systems, reorganizing disk usage, copying data on hard disks and disk imaging. The package contains a library, libparted, as well as well as a command-line frontend, parted, which can also be used in scripts.
  • http://gparted.sourceforge.net/

dd

  • dd - Copy a file, converting and formatting according to the options.
    • dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data.
dd if=/dev/sr0 of=myCD.iso bs=2048 conv=noerror,sync
  create an ISO disk image from a CD-ROM.
dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=noerror
  Clone one partition to another
dd if=/dev/ad0 of=/dev/ad1 bs=1M conv=noerror
  Clone a hard disk "ad0" to "ad1".

dd if=/dev/zero bs=1024 count=1000000 of=file_1GB
dd if=file_1GB of=/dev/null bs=64k
  drive benchmark test and analyze the sequential read and write performance for 1024 byte blocks

du (disk usage)

du -sh
  size of a folder
du -S
  size of files in a folder

du -aB1m|awk '$1 >= 100'
  everything over 100Mb

cd / | sudo du -khs *
  show root folder size

sudo du -a --max-depth=1 /usr/lib | sort -n -r | head -n 20
  size of program folders /usr/lib

du -sk ./* | sort -nr | awk 'BEGIN{ pref[1]="K"; pref[2]="M"; pref[3]="G";} { total = total + $1;
x = $1; y = 1;  while( x > 1024 ) { x = (x + 1023)/1024; y++; }
printf("%g%s\t%s\n",int(x*10)/10,pref[y],$2); } END { y = 1; while( total > 1024 )
{ total = (total + 1023)/1024; y++; } printf("Total: %g%s\n",int(total*10)/10,pref[y]); }'

ncdu

  • ncdu - ncurses disk usage
ncdu / --exclude /home --exclude /media --exclude /run/media
  check everything apart from home and external drives

ncdu / --exclude /home --exclude /media --exclude /run/media
  check everything apart from external drives

ncdu / --exclude /home --exclude /media --exclude /run/media --exclude /boot
--exclude /tmp --exclude /dev --exclude /proc
 just the root partition

df

  • df - report file system disk space usage

Baobab

Other

todo; source aliases.zsh

  • Filelight creates an interactive map of concentric, segmented rings that help visualise disk usage on your computer.

Files and directories

ls
  list in row
ls -l
  long list

ls *
  files in directory and immediate subdiretories

just names; 

ls -m1
  -m fill width with a comma separated list of entries ??
ls --format single-column
  column of names only
ls -l | grep - | awk '{print $9}'
  using awk to show the 9th word (name). strips colour.
ls -l | cut -f9 -s -d" "
  using cut to cut from the 9th word, using space as a delimiter. strips colour.
ls | cat
  neat

ls -a
  show hidden files
ls  -A
  show hidden files, exclude . and ..

stat .
  display file or file system status
stat -c "%n %a" * | column -t
  directory files + octal

mkdir directory
mkdir directory -p
  no error if existing, make parent directories as needed

ln -s {target-filename}
ln -s {target-filename} {symbolic-filename}



cd change/directory/path

[4] 

mv position1 ~/position2
  basic move

pax - read and write file archives and copy directory hierarchies 

find . -type f -print0 | xargs -0 stat -c "%y %s %n"

File types

xdg-mime default Thunar.desktop inode/directory
  to make Thunar the default file-browser
xdg-mime default xpdf.desktop application/pdf
  to use xpdf as the default PDF viewer

/usr/share/applications/defaults.list      (global)
~/.local/share/applications/defaults.list  (per user, overrides global)

[Default Applications]
mimetype=desktopfile1;desktopfile2;...;desktopfileN

Copying files

cp - copy files and directories

scp -P 2264 foobar.txt your_username@remotehost.edu:/some/remote/directory
scp -rP 2264 folder your_username@remotehost.edu:/some/remote/directory

wget -O myzip.zip https://github.com/zeromq/jzmq/zipball/master
wget -k
  --convert-links

Viewing files

less is better than more 

cat filename
  output file to screen
cat -n filename
  output file to screen w/ line numbers
cat filename1 filename2
  output two files (concatinate)
cat filename1 > filename2
  overwrite filename2 with filename1
cat filename1 >> filename2
  append filename1 to filename2
cat filename{1,2} > filename2
  add filename1 and filename2 together into filename3

head filename
  top 10 lines of file
head -23 filename
  top 23 lines of file
tail filename
  bottom 10 lines of file
tail -23 filename
  bottom 23 lines of file

sed -n 20,30p filename
  print lines 20..30 of file [5]

Finding files

find /usr/share -name README
find ~/Journalism -name '*.txt'
find ~/Programming -path '*/src/*.c'

find ~/Images/Screenshots -size +500k -iname '*.jpg'
find ~/Journalism -name '*.txt' -exec cat {} ;
find ~/Journalism -name '*.txt' -print0 | xargs -0 cat   (faster than above)

find / -group [group]
find / -user [user]

http://arstechnica.com/information-technology/2011/07/ask-ars-how-to-use-the-find-command-in-a-pipeline/

  • sgrep - search a file for a structured pattern

Compression

tar

tar <operation> [options]
Operations:
  [-]A --catenate --concatenate
  [-]c --create
  [-]d --diff --compare
  [-]r --append
  [-]t --list
  [-]u --update
  [-]x --extract --get
  --delete

Common Options:
  -C, --directory DIR
  -f, --file F
  -j, --bzip2
  -p, --preserve-permissions
  -v, --verbose
  -z, --gzip

zip

gzip
bzip
7z
  • 7-Zip is a file archiver with the highest compression ratio. The program supports 7z (that implements LZMA compression algorithm), ZIP, CAB, ARJ, GZIP, BZIP2, TAR, CPIO, RPM and DEB formats. Compression ratio in the new 7z format is 30-50% better than ratio in ZIP format.
    • p7zip is a port of 7za.exe for POSIX systems like Unix (Linux, Solaris, OpenBSD, FreeBSD, Cygwin, AIX, ...), MacOS X and also for BeOS and Amiga. 7za.exe is the command line version of 7-zip, see http://www.7-zip.org/. 7-Zip is a file archiver with highest compression ratio.
    • man z7 (p7zip)
    • p7zip-light in AUR
7z x filename
  extract archive with directories
xz

pax

  • pax will read, write, and list the members of an archive file, and will copy directory hierarchies. pax operation is independent of the specific archive format, and supports a wide variety of different archive formats. A list of supported archive formats can be found under the description of the -x option. [6]

Other

Generic function

# Extract Files
extract() {
 if [ -f $1 ] ; then
     case $1 in
         *.tar.bz2)   tar xvjf $1    ;;
         *.tar.gz)    tar xvzf $1    ;;
         *.tar.xz)    tar xvJf $1    ;;
         *.bz2)       bunzip2 $1     ;;
         *.rar)       unrar x $1     ;;
         *.gz)        gunzip $1      ;;
         *.tar)       tar xvf $1     ;;
         *.tbz2)      tar xvjf $1    ;;
         *.tgz)       tar xvzf $1    ;;
         *.zip)       unzip $1       ;;
         *.Z)         uncompress $1  ;;
         *.7z)        7z x $1        ;;
         *.xz)        unxz $1        ;;
         *.exe)       cabextract $1  ;;
         *)           echo "\`$1': unrecognized file compression" ;;
     esac
 else
     echo "\`$1' is not a valid file"
 fi
}

Commands

Resources

Use output as input. 

cat `locate file.txt`

or 

cat $(locate file.txt)

todo; piping and redirection

Programs

  • auditd - userspace component to the Linux Auditing System [9]
  • nice - run a program with modified scheduling priority
  • cut - remove sections from each line of files
  • shelr - console screencasting tool

~/.local/share/applications/mimeinfo.cache

Finding programs

whereis

apropos apropos vim
  search the whatis database for strings

Cron

crontab -l
  view crontabs
crontab -e
  edit their crontabs
crontab -r
 remove their crontabs
crontab saved_crontab_filename
  overwrite their old crontab with saved crontab

There are several special predefined values which can be used to substitute the CRON expression. 

Entry                  Description                                 Equivalent To
 @yearly (or @annually) Run once a year, midnight, Jan. 1st         0 0 1 1 *
 @monthly               Run once a month, midnight, first of month  0 0 1 * *
 @weekly                Run once a week, midnight on Sunday         0 0 * * 0
 @daily                 Run once a day, midnight                    0 0 * * *
 @hourly                Run once an hour, beginning of hour         0 * * * *
 @reboot                Run at startup                              @reboot 

*    *    *    *    *  command to be executed
┬    ┬    ┬    ┬    ┬
│    │    │    │    │
│    │    │    │    │
│    │    │    │    └───── day of week (0 - 6) (0 is Sunday, or use names)
│    │    │    └────────── month (1 - 12)
│    │    └─────────────── day of month (1 - 31)
│    └──────────────────── hour (0 - 23)
└───────────────────────── min (0 - 59)
  • /etc/cron.allow - If this file exists, then you must be listed therein (your username must be listed) in order to be allowed to use cron jobs.
  • /etc/cron.deny - If the cron.allow file does not exist but the /etc/cron.deny file does exist, then you must not be listed in the /etc/cron.deny file in order to use cron jobs.

Wildcards

MIME

chroot

Startuup

  • e4rat ("Ext4 - Reducing Access Times") is a toolset to accelerate the boot process as well as application startups. Through physical file realloction e4rat eliminates both seek times and rotational delays. This leads to a high disk transfer rate. Placing files on disk in a sequentially ordered way allows to efficiently read-ahead files in parallel to the program startup.

Users

  • /etc/passwd - local user information
    • account:password:UID:GID:GECOS:directory:shell
  • /etc/shadow - restricted access encrypted password file
useradd username
  create user with defaults (no password)
useradd -D
  show defaults that will be used

useradd -m -g [initial_group] -G [additional_groups] -s [login_shell] [username]
  -m - add home dir
  -d - non-default home dir path
  -c comment
  -e 2006-06-30 - expires
useradd -m -g users -G audio,lp,optical,storage,video,wheel,games,power,scanner -s /bin/bash username

adduser
  interactive tool

userdel username
  remove user
userdel -r username
  remove user plus their home folder, mail spool

cat /etc/passwd | cut -d: -f 1,3,6 | grep "[5-9][0-9][0-9]"
  List users on a system [10]

Groups

groupadd [group]
  add group
gpasswd -a [user] [group]
  add user to group
gpasswd -d [user] [group]
  delete user from group

User must relogin for new group to take effect.

Permissions

sudoers

ACLs

Partition must have acl set in /etc/fstab (and be remounted after). 

setfacl -m "u:username:permissions"
setfacl -m "u:uid:permissions"
  add permissions for user

setfacl -m "g:groupname:permissions"
setfacl -m "g:gid:permissions"
  add permissions for group

setfacl -m "u:user:rwx" file
  add read, write, execure perms for user for file
setfacl -Rm "u:user:rw" /dir
  add recursive read, write perms for user for dir
setfacl -Rdm "u:user:rw" /dir
  add recursive read, write perms for user for dir and make them default for future changes

Session permission management

For non-root mount permissions, etc.

Networking

sudo /etc/init.d/<service> restart
  ubuntu, restart a service
sudo /etc/rc.d/<service> stop | start | restart
  arch, service things

sudo sh -c "echo 'something' >> /etc/privilegedfile"
chown -R user:group . - change all and subitems [11]
  • route -n - display the host's networks and gateway
  • View The Daily Show, etc. in the UK, etc. Mofity HTTP headers; X-Forwarded-For "12.13.14.15" [13]

curl

  • curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.
curl http://www.google.com/search.js -o /path/to/local/file.js

curl http://site.{one,two,three}.com

curl ftp://ftp.numericals.com/file[1-100].txt
     ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)
     ftp://ftp.letters.com/file[a-z].txt
 sequences of alphanumeric series by using []

curl http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html
  Nested sequences are not supported, but you can use several ones next to each other:

curl http://www.numericals.com/file[1-100:10].txt  http://www.letters.com/file[a-z:2].txt
  multiple urls + specify a step counter for the ranges to get every Nth number or letter:

ftp

nmap

nmap -sT -sU -O -p 1-65535 localhost
  full port scan

VOIP

Media

Package management

Apt

dpkg --get-selections > installed-software
  create list of installed software

dpkg --set-selections < installed-software
dselect
  reinstall from list

dpkg --list
  to check

Pacman

pacman -Sy [package]
  install [package]
pacman -Syu
  upgrade system

pacman -Qo [file]
  check what package owns a file
pacman -Qi
  list installed programms

pacman -Qqtd
  check whether there are any orphaned packages
pacman -Rsn packagename
  remove orphaned packages
pacman -Rc packagename
  remove package and deps ('cascade')
  • cacheclean - Cleans up pacman packages. Users selects how many old versions to keep.
cacheclean {-p} {-v} <# of copies to keep>
# of copies to keep - (required) how many generations of each package.
-p - (optional) preview what would be deleted.
-v - (optional) show deleted packages.

for i in `pacman -Qdt | awk '{print $1}'`; do pacman -R $i ; done

= Helpers

= Packages

 makepkg -g >> PKGBUILD && makepkg

"ERROR: PKGBUILD contains CRLF characters and cannot be sourced." Solution: 

sed -i 's/^M//' PKGBUILD

[ctrl+v][ctrl+m] for the ^M symbol.

Time

echo "Today's date is `date -I`"

Convert unixtime into date; 

date -d @1337000000

ntpdate pool.ntp.org && hwclock --systohc && hwclock --adjust
  Synchronize both your system clock and hardware clock and calculate/adjust time drift.
  Do not run this command if you already have ntpd running! [15]

Swap

swapon -s
free -m

Suspension, hibernation

Other software

CUPS

Printing system.

Regex

Cisco

Storage

Plan 9

Unix done better.

Retrieved from "https://wiki.thingsandstuff.org/index.php?title=*nix&oldid=4111"