Debugging

General

• https://en.wikipedia.org/wiki/Software_bug

• https://en.wikipedia.org/wiki/Debugging

• https://en.wikipedia.org/wiki/Debug_symbol - a special kind of symbol that attaches additional information to the symbol table of an object file, such as a shared library or an executable. This information allows a symbolic debugger to gain access to information from the source code of the binary, such as the names of identifiers, including variables and routines. The symbolic information may be compiled together with the module's binary file, or distributed in a separate file, or simply discarded during the compilation and/or linking. This information can be helpful while trying to investigate and fix a crashing application or any other fault.

• Debugging Information in Separate Files - GDB allows you to put a program’s debugging information in a file separate from the executable itself, in a way that allows GDB to find and load the debugging information automatically. Since debugging information can be very large—sometimes larger than the executable code itself—some systems distribute debugging information for their executables in separate files, which users can install only when they need to debug a problem.

• https://en.wikipedia.org/wiki/Dynamic_program_analysis

• https://wiki.archlinux.org/index.php/Debug_-_Getting_Traces

• https://en.wikipedia.org/wiki/Tracing_(software)

• https://jvns.ca/blog/2017/07/05/linux-tracing-systems [1]

• Debug your programs like they're closed source! [2] [3]

• ltrace, strace, truss, pstack

• YouTube: Sysdig CCWFS - A brief history of syscalls - Adam Leventhal

• https://en.wikipedia.org/wiki/Instrumentation_(computer_programming)

• https://en.wikipedia.org/wiki/stabs - a debugging data format for storing information about computer programs for use by symbolic and source-level debuggers. (The information is stored in symbol table entries; hence the name "stabs".) It "was apparently invented by Peter Kessler at the University of California, Berkeley" for use with a Pascal compiler. When stabs was created in the 1980s, the dominant object file format was a.out, which (unlike more recent formats such as ELF) makes no provision for storing debugging information. Stabs works around this problem by encoding the information using special entries in the symbol table. At one stage stabs was widely used on Unix systems, but the newer DWARF format has largely supplanted it.

• https://en.wikipedia.org/wiki/DWARF - a widely used, standardized debugging data format. DWARF was originally designed along with Executable and Linkable Format (ELF), although it is independent of object file formats. The name is a medieval fantasy complement to "ELF" that has no official meaning, although the backronym 'Debugging With Attributed Record Formats' was later proposed.

truss

• https://en.wikipedia.org/wiki/truss_(Unix) - system tool available on some Unix-like operating systems. When invoked with an additional executable command-line argument, truss makes it possible to print out the system calls made by and the signals received by this executable command-line argument. As of version IEEE Std 1003.1-2008, truss is not part of the Single UNIX Specification (POSIX). The truss command was originally developed by Roger Faulkner and Ron Gomes as part of the development of Procfs for System V Release 4. While several names were considered, “truss” was chosen for being non-ambiguous and easily pronounceable, with multiple meanings, including as an abbreviation for TRace Unix Syscalls and Signals or in the sense of “If your program doesn’t work, put it in a truss.”

gdb

• https://en.wikipedia.org/wiki/GNU_Debugger

• https://en.wikipedia.org/wiki/Nemiver - gdb frontend

• https://www.gnu.org/software/gdb/
• https://stackoverflow.com/questions/2308653/can-i-use-gdb-to-debug-a-running-process
• https://blogs.oracle.com/dbx/entry/gdb_vs_dbx_commands_mapping

• “Give me 15 minutes and I'll change your view of GDB” [4]

• https://github.com/rogerhu/gdb-heap

Valgrind

• Valgrind - an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.
  • https://en.wikipedia.org/wiki/Valgrind - a programming tool for memory debugging, memory leak detection, and profiling. Valgrind was originally designed to be a free memory debugging tool for Linux on x86, but has since evolved to become a generic framework for creating dynamic analysis tools such as checkers and profilers.

• http://maintainablecode.logdown.com/posts/245425-valgrind-is-not-a-leak-checker

• Callgrind / KCachegrind - profiling tool and the profile data visualization Both are licensed under GPL V2. Callgrind uses runtime instrumentation via the Valgrind framework for its cache simulation and call-graph generation. This way, even shared libraries and dynamically opened plugins can be profiled. The data files generated by Callgrind can be loaded into KCachegrind for browsing the performance results. But there is also a command line tool in the package to get ASCII reports from data files without the need to use KCachegrind. The format of Callgrind output is documented here. With conversion scripts, KCachegrind is able to visualize output of other profilers like OProfile, a system-wide profiler for Linux using statistical sampling with hardware performance counters. There also exist converters for profiling output of Python, PHP and PERL.

rr

• rr - aspires to be your primary C/C++ debugging tool for Linux, replacing — well, enhancing — gdb. You record a failure once, then debug the recording, deterministically, as many times as you want. The same execution is replayed every time. rr also provides efficient reverse execution under gdb. Set breakpoints and data watchpoints and quickly reverse-execute to where they were hit.
  • http://robert.ocallahan.org/2016/04/using-rr-to-debug-rr.html

QIRA

• QIRA - a timeless debugger. All state is tracked while a program is running, so you can debug in the past.
  • https://www.youtube.com/watch?v=eGl6kpSajag [5]

radare2

• https://github.com/radare/radare2 - Radare project started as a forensics tool, a scriptable commandline hexadecimal editor able to open disk files, but later support for analyzing binaries, disassembling code, debugging programs, attaching to remote gdb servers, .. r2 is a rewrite from scratch of radare in order to provide a set of libraries and tools to work with binary files.

voltron

• https://github.com/snare/voltron - an extensible debugger UI toolkit written in Python. It aims to improve the user experience of various debuggers (LLDB, GDB, VDB and WinDbg) by enabling the attachment of utility views that can retrieve and display data from the debugger host. By running these views in other TTYs, you can build a customised debugger user interface to suit your needs. [6]

ftrace

• Ftrace - an internal tracer designed to help out developers and designers of systems to find what is going on inside the kernel. It can be used for debugging or analyzing latencies and performance issues that take place outside of user-space. Although ftrace is typically considered the function tracer, it is really a frame work of several assorted tracing utilities. There's latency tracing to examine what occurs between interrupts disabled and enabled, as well as for preemption and from a time a task is woken to the task is actually scheduled in.
  • https://en.wikipedia.org/wiki/ftrace
  • ftrace: trace your kernel functions!

DTrace

• http://www.brendangregg.com/dtrace.html

• https://en.wikipedia.org/wiki/DTrace - a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License (CDDL) and has been ported to several other Unix-like systems. In December 2012, Oracle announced the general availability of DTrace for Oracle Linux.

• https://gnu.wildebeest.org/blog/mjw/2018/02/14/dtrace-for-linux-oracle-does-the-right-thing/

strace

• https://en.wikipedia.org/wiki/strace

• http://chadfowler.com/blog/2014/01/26/the-magic-of-strace/ [7]

• https://news.ycombinator.com/item?id=16353054

ltrace

• ltrace - intercepts and records dynamic library calls which are called by an executed process and the signals received by that process. It can also intercept and print the system calls executed by the program.
  • http://linux.die.net/man/1/ltrace
  • https://en.wikipedia.org/wiki/Ltrace
  • http://blog.packagecloud.io/eng/2016/03/14/how-does-ltrace-work/

ktrace

• https://en.wikipedia.org/wiki/ktrace - a utility included with certain versions of BSD Unix and Mac OS X that traces kernel interaction with a program and dumps it to disk for the purposes of debugging and analysis. Traced kernel operations include system calls, namei translations, signal processing, and I/O. ktrace is somewhat similar to Linux's strace, except for being much faster – with strace, every system call executed by the traced program requires context switch to the tracing program and back, while the tracing with ktrace is actually performed by the kernel, so no additional context switches are required.

KProbes

• lwn: An introduction to KProbes

eBPF / BCC

• BCC - a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of eBPF (Extended Berkeley Packet Filters), a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above. [8]
  • https://github.com/iovisor/bcc

• A thorough introduction to eBPF

• https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise [9]

SystemTap

• SystemTap - provides free software (GPL) infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data. SystemTap provides a simple command line interface and scripting language for writing instrumentation for a live running kernel plus user-space applications. We are publishing samples, as well as enlarging the internal "tapset" script library to aid reuse and abstraction. Among other tracing/probing tools, SystemTap is the tool of choice for complex tasks that may require live analysis, programmable on-line response, and whole-system symbolic access. SystemTap can also handle simple tracing jobs.
  • https://en.wikipedia.org/wiki/SystemTap

• https://blog.cloudflare.com/how-to-monkey-patch-the-linux-kernel/

LTT / LTTng

• https://en.wikipedia.org/wiki/Linux_Trace_Toolkit - a set of tools that is designed to log program execution details from a patched Linux kernel and then perform various analyses on them, using console-based and graphical tools. LTT has been mostly superseded by its successor LTTng (Linux Trace Toolkit Next Generation).

• LTTng - an open source tracing framework for Linux.

Apport

• https://wiki.ubuntu.com/Apport - Apport is a system which: intercepts crashes right when they happen the first time, gathers potentially useful information about the crash and the OS environment, can be automatically invoked for unhandled exceptions in other programming languages (e. g. in Ubuntu this is done for Python), can be automatically invoked for other problems that can be automatically detected (e. g. Ubuntu automatically detects and reports package installation/upgrade failures from update-manager), presents a UI that informs the user about the crash and instructs them on how to proceed, and is able to file non-crash bug reports about software, so that developers still get information about package versions, OS version etc.
  • https://github.com/rickysarraf/apport

coala

• coala - provides a unified interface for linting and fixing code with a single configuration file, regardless of the programming languages used. You can use coala from within your favorite editor, integrate it with your CI, get the results as JSON, or customize it to your needs with its flexible configuration syntax. [10]

SystemTap

• SystemTap - provides free software (GPL) infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data.
  • https://en.wikipedia.org/wiki/SystemTap - a scripting language and tool for dynamically instrumenting running production Linux kernel-based operating systems. System administrators can use SystemTap to extract, filter and summarize data in order to enable diagnosis of complex performance or functional problems.

Senseye

• https://github.com/letoram/senseye - Dynamic Visual Debugging / Reverse Engineering Toolsuite. a set of data providers, parsers that work with the Arcan display servers IPC subsystem, and as a set of extension script for the Durden desktop environment/window management scheme.

to sort

• Recall class

• http://blogs.msdn.com/b/xweb/archive/2008/09/19/the-secret-life-of-software-bugs-exposed.aspx

• https://github.com/AutomatedTester/Bugsy - A python library for interacting with Bugzilla

• http://lemire.me/blog/2016/06/21/i-do-not-use-a-debugger/ [11]

• Flame - graphs are a visualization of profiled software, allowing the most frequent code-paths to be identified quickly and accurately. They can be generated using my open source programs on github.com/brendangregg/FlameGraph, which create interactive SVGs. See the Updates section for other implementations.
  • https://github.com/brendangregg/FlameGraph - Stack trace visualizer

• jtool - a command meant to meet and exceed the functionality to XCode's otool(1), picking up along the way additional Mach-O commands such as atos(1), dyldinfo(1), nm(1), segedit(1), pagestuff(1), strings(1) , and even codesign(1) and the informal ldid. jtool also provides novel features, such in-binary search functionality, symbol injection and a disassembler functionality with (limited but constantly improving) emulation capabilities. It also provides color output. Most importantly, it can be run on a variety of platforms - OS X, iOS, and even Linux. It is ENTIRELY FREE for use of any type, and the latest version

• Forkstat - a program that logs process fork(), exec() and exit() activity. It is useful for monitoring system behaviour and to track down rogue processes that are spawning off processes and potentially abusing the system.

• ps-watcher – A Perl Program for Monitoring a System via ps-like Commands.
  • Process monitoring with ps-watcher

• http://people.redhat.com/sgrubb/audit/

• https://en.wikipedia.org/wiki/Ptrace - system call found in several Unix and Unix-like operating systems. By using ptrace (the name is an abbreviation of "process trace") one process can control another, enabling the controller to inspect and manipulate the internal state of its target. ptrace is used by debuggers and other code-analysis tools, mostly as aids to software development.

• https://github.com/dellis23/ispy - Monitor the output of terminals and processes. [12]

• https://github.com/p-e-w/maybe [13]

• https://www.gnu.org/software/acct/ - psacct, etc.

• linux process monitoring (exec, fork, exit, set*uid, set*gid)

• https://unix.stackexchange.com/questions/64611/how-to-track-the-number-of-processes-and-forks-per-user

• https://github.com/columbia/libtrack

• Dwarf Home - a debugging file format used by many compilers and debuggers to support source level debugging. It addresses the requirements of a number of procedural languages, such as C, C++, and Fortran, and is designed to be extensible to other languages. DWARF is architecture independent and applicable to any processor or operating system. It is widely used on Unix, Linux and other operating systems, as well as in stand-alone environments.

ProcDump-for-Linux

• https://github.com/Microsoft/ProcDump-for-Linux - a Linux reimagining of the classic ProcDump tool from the Sysinternals suite of tools for Windows. ProcDump provides a convenient way for Linux developers to create core dumps of their application based on performance triggers. [14]