Debugging

From Things and Stuff Wiki
Jump to navigation Jump to search


General

See *nix#System calls, Development#Testing, Server#Monitoring


this article is still fragmented




Julia Evans:


  • https://en.wikipedia.org/wiki/Debug_symbol - a special kind of symbol that attaches additional information to the symbol table of an object file, such as a shared library or an executable. This information allows a symbolic debugger to gain access to information from the source code of the binary, such as the names of identifiers, including variables and routines. The symbolic information may be compiled together with the module's binary file, or distributed in a separate file, or simply discarded during the compilation and/or linking. This information can be helpful while trying to investigate and fix a crashing application or any other fault.


  • Debugging Information in Separate Files - GDB allows you to put a program’s debugging information in a file separate from the executable itself, in a way that allows GDB to find and load the debugging information automatically. Since debugging information can be very large—sometimes larger than the executable code itself—some systems distribute debugging information for their executables in separate files, which users can install only when they need to debug a problem.


  • https://en.wikipedia.org/wiki/Dynamic_program_analysis - the analysis of computer software that is performed by executing programs on a real or virtual processor. For dynamic program analysis to be effective, the target program must be executed with sufficient test inputs to produce interesting behavior. Use of software testing measures such as code coverage helps ensure that an adequate slice of the program's set of possible behaviors has been observed. Also, care must be taken to minimize the effect that instrumentation has on the execution (including temporal properties) of the target program. Dynamic analysis is in contrast to static program analysis. Unit tests, integration tests, system tests and acceptance tests use dynamic testing.

System information

time [command]
  return with command execution time



uname -a





oof where did i put the rest of these


Formatted

  • inxi - A newer, better system information script for irc, administration, and system troubleshooters.

screenFetch

archey

alsi

ufetch


Neofetch

  • https://github.com/dylanaraps/neofetch - a command-line system information tool written in bash 3.2+. Neofetch displays information about your operating system, software and hardware in an aesthetic and visually pleasing way. The overall purpose of Neofetch is to be used in screen-shots of your system. Neofetch shows the information other people want to see. There are other tools available for proper system statistic/diagnostics.

pfetch

coffeetch

  • https://github.com/clieg/coffeetch - a minimal command line system information tool written in C. It displays the username and hostname, OS, kernel, machine model, CPU, RAM, shell, number of packages installed, and uptime information.

cpufetch

Hardware

  • PCI Utilities - a collection of programs for inspecting and manipulating configuration of PCI devices, all based on a common portable library libpci which offers access to the PCI configuration space on a variety of operating systems.

The utilities include:

  • lspci - displays detailed information about all PCI buses and devices in the system
  • setpci - allows reading from and writing to PCI device configuration registers. For example, you can adjust the latency timers with it.


  • Dmidecode - reports information about your system's hardware as described in your system BIOS according to the SMBIOS/DMI standard (see a sample output). This information typically includes system manufacturer, model name, serial number, BIOS version, asset tag as well as a lot of other details of varying level of interest and reliability depending on the manufacturer. This will often include usage status for the CPU sockets, expansion slots (e.g. AGP, PCI, ISA) and memory module slots, and the list of I/O ports (e.g. serial, parallel, USB).


  • lshw - Hardware Lister, a small tool to provide detailed information on the hardware configuration of the machine. It can report exact memory configuration, firmware version, mainboard configuration, CPU version and speed, cache configuration, bus speed, etc. on DMI-capable x86 or EFI (IA-64) systems and on some PowerPC machines (​PowerMac G4 is known to work).


lstopo --of txt


  • https://github.com/0vm/Simple-HWID-Checker - a hardware identification or HWID tool that collects system-specific information such as hard drive data, computer name, CPU hash, GPU information, total system memory, MAC address, and BIOS serial number.

Instrumentation

  • https://en.wikipedia.org/wiki/Instrumentation_(computer_programming) - refers to an ability to monitor or measure the level of a product's performance, to diagnose errors, and to write trace information. Programmers implement instrumentation in the form of code instructions that monitor specific components in a system (for example, instructions may output logging information to appear on the screen). When an application contains instrumentation code, it can be managed by using a management tool. Instrumentation is necessary to review the performance of the application. Instrumentation approaches can be of two types: source instrumentation and binary instrumentation.


  • https://en.wikipedia.org/wiki/Hardware_performance_counter - or hardware counters are a set of special-purpose registers built into modern microprocessors to store the counts of hardware-related activities within computer systems. Advanced users often rely on those counters to conduct low-level performance analysis or tuning.






procfs

  • https://en.wikipedia.org/wiki/Procfs - a special filesystem in UNIX-like operating systems that presents information about processes and other system information in a hierarchical file-like structure, providing a more convenient and standardized method for dynamically accessing process data held in the kernel than traditional tracing methods or direct access to kernel memory. Typically, it is mapped to a mount point named /proc at boot time. The proc file system acts as an interface to internal data structures in the kernel. It can be used to obtain information about the system and to change certain kernel parameters at runtime (sysctl).

More than a few operating systems support the proc filesystem, including Solaris, IRIX, Tru64 UNIX, BSD, Linux (Linux kernel extends it to non–process-related data), IBM AIX (which bases its implementation on Linux to improve compatibility), QNX, and Plan 9 from Bell Labs. The proc filesystem provides a method of communication between kernel space and user space. For example, the GNU version of ps uses the procfs to obtain its data, without using any specialized system calls.


procps

  • free - Report the amount of free and used memory in the system
  • kill - Send a signal to a process based on PID
  • pgrep - List processes based on name or other attributes
  • pkill - Send a signal to a process based on name or other attributes
  • pmap - Report memory map of a process
  • ps - Report information of processes
  • pwdx - Report current directory of a process
  • skill - Obsolete version of pgrep/pkill
  • slabtop - Display kernel slab cache information in real time
  • snice - Renice a process
  • sysctl - Read or Write kernel parameters at run-time
  • tload - Graphical representation of system load average
  • top - Dynamic real-time view of running processes
  • uptime - Display how long the system has been running
  • vmstat - Report virtual memory statistics
  • w - Report logged in users and what they are doing
  • watch - Execute a program periodically, showing output fullscreen
( eval $(TERM=xterm dircolors); watch -c ls --color )



ps ax | grep -process-





  • fuser - identifies what processes are using files.
  • killall - kills a process by its name, similar to a pkill found in some other Unices.
  • pstree - Shows currently running processes in a tree format.
  • peekfd - Peek at file descriptors of running processes.




ps


pstree


truss

  • https://en.wikipedia.org/wiki/truss_(Unix) - system tool available on some Unix-like operating systems. When invoked with an additional executable command-line argument, truss makes it possible to print out the system calls made by and the signals received by this executable command-line argument. As of version IEEE Std 1003.1-2008, truss is not part of the Single UNIX Specification (POSIX). The truss command was originally developed by Roger Faulkner and Ron Gomes as part of the development of Procfs for System V Release 4. While several names were considered, “truss” was chosen for being non-ambiguous and easily pronounceable, with multiple meanings, including as an abbreviation for TRace Unix Syscalls and Signals or in the sense of “If your program doesn’t work, put it in a truss.”


top

Part of the procps package.

Current stats only, no disk or network. Sorts by cpu by default. A multicore box can show more than 100%. 

z
  colour!
d
  change update interval
return, space
  refresh

H
  show separate threads of each process
i
  show run queue (R) or waiting on disk io (D, red)
1
  show serperate cpus

q
  quit


htop


atop

Shows short lived processes that run and finish between update interval times.

lines;

  • PRC: Total CPU time in system and user mode, total number of processes and of zombie processes, and the number of processes that exited during the polling interval.
  • CPU and CPL: CPU utilization and load (averaged over 1, 5 and 15 minutes).
  • cpu: individual CPU usage.
  • MEM and SWP: Amount of memory and swap space that is available and where it’s allocated. vmcom and vmlim show how much virtual memory space is committed and what the limit is.
  • DSK: disk utilization. avio shows the average number of milliseconds per request.
  • NET: Network utilization for the TCP layer (“transport”), the IP layer (“network”) and each interface.
Ctrl-f
  forward a page
Ctrl-b
  back a page

C
  sort by cpu activity
M
  sort by memory consumption
D
  sort by disk activity
N
  sort by network activity
A
  sort by most active system resource (auto mode)

P
  focus on process (regex)
y
  show all threads (inc. idle)

s
 Scheduling information
m
  Memory consumption
d
  Disk utilization
v
 Variable information
c
  Command line
p
 Accumulated per program
u
  Accumulated per user
n
 Network utilization (patched kernel)

i
  interval timer
t
  trigger update

atopsar -c 60 5
  current CPU utilization, five times with an interval of sixty seconds

atopsar -A -b 13:00 -e 13:35
  all available reports are generated, starting from 13:00 till 13:35


bb

btop

sysfs

  • https://en.wikipedia.org/wiki/sysfs - a pseudo file system provided by the Linux kernel that exports information about various kernel subsystems, hardware devices, and associated device drivers from the kernel's device model to user space through virtual files. In addition to providing information about various devices and kernel subsystems, exported virtual files are also used for their configuration. sysfs provides functionality similar to the sysctl mechanism found in BSD operating systems, with the difference that sysfs is implemented as a virtual file system instead of being a purpose-built kernel mechanism, and that, in Linux, sysctl configuration parameters are made available at /proc/sys/ as part of procfs, not sysfs which is mounted at /sys/


Tools

perf

  • perf (sometimes "Perf Events" or perf tools, originally "Performance Counters for Linux", PCL) - is a user-space performance analyzing tool in Linux with git-like interface and subcommands. It is capable of statistical profiling of entire system (both kernel and user code), single CPU or severals threads.




lsof

  • lsof - "lists open files" (lots, given "everything" is a file)
lsof -i :[port]
  what application using a specific port



SYSSTAT

iostat, isag, mpstat, pidstat, sadf, sar 



iostat -d -x 2


nmon


vmstat


collectl


free

  • free - Display amount of free and used memory in the system


glances

bpytop

bashtop

gtop

gotop

bottom

  • https://github.com/clementtsang/bottom - A cross-platform graphical process/system monitor with a customizable interface and a multitude of features. Supports Linux, macOS, and Windows. Inspired by both gtop and gotop.

bpytop

s-tui

ttyload


Dstat

  • Dstat - a versatile replacement for vmstat, iostat, netstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting. Dstat allows you to view all of your system resources in real-time, you can eg. compare disk utilization in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval). Dstat gives you detailed selective information in columns and clearly indicates in what magnitude and unit the output is displayed. Less confusion, less mistakes. And most importantly, it makes it very easy to write plugins to collect your own counters and extend in ways you never expected. Dstat's output by default is designed for being interpreted by humans in real-time, however you can export details to CSV output to a file to be imported later into Gnumeric or Excel to generate graphs.


rtop

  • rtop - Remote Server Monitoring over SSH [15]


vtop


zenith


Other



input/output

Memory and/or disk. 

iotop -obt -d2


  • Bonnie++ - a benchmark suite that is aimed at performing a number of simple tests of hard drive and file system performance. Then you can decide which test is important and decide how to compare different systems after running it. I have no plans to ever have it produce a single number, because I don't think that a single number can be useful when comparing such things. The main program tests database type access to a single file (or a set of files if you wish to test more than 1G of storage), and it tests creation, reading, and deleting of small files which can simulate the usage of programs such as Squid, INN, or Maildir format email.

printk


  • https://en.wikipedia.org/wiki/Printk printk - a function that prints messages and is used in the C Programming Language exclusively for the Linux Kernel. It accepts a string parameter called the format string, which specifies a method for rendering an arbitrary number of varied data type parameter(s) into a string. The string is then printed to the kernel log.It provides a printf-like abstraction and its parsing of the format string and arguments behave exactly the same way. It acts as a debugging tool for kernel programmers who need this function for logging messages from the kernel.



KProbes / dprobes




  • https://en.wikipedia.org/wiki/Dprobes - a Linux kernel analysis framework created in 2004, which features the ability to insert software probes dynamically into running code. It is based on kprobes.


debugfs

  • https://en.wikipedia.org/wiki/Debugfs - a special file system available in the Linux kernel since version 2.6.10-rc3. It was written by Greg Kroah-Hartman. debugfs is a simple-to-use RAM-based file system specially designed for debugging purposes. It exists as a simple way for kernel developers to make information available to user space. Unlike /proc, which is only meant for information about a process, or sysfs, which has strict one-value-per-file rules, debugfs has no rules at all. Developers can put any information they want there.

Kernel_marker

  • https://en.wikipedia.org/wiki/Kernel_marker - were a static kernel instrumentation support mechanism for Linux kernel source code, allowing special tools such as LTTng or SystemTap to trace information exposed by these probe points. A patch-set implementing them was merged into version 2.6.24, which was released on January 24, 2008. To address issues regarding kernel markers, Mathieu Desnoyers, their original author, implemented a simpler and more type-safe version of static probe points named Tracepoints.

ptrace

  • https://en.wikipedia.org/wiki/Ptrace - system call found in several Unix and Unix-like operating systems. By using ptrace (the name is an abbreviation of "process trace") one process can control another, enabling the controller to inspect and manipulate the internal state of its target. ptrace is used by debuggers and other code-analysis tools, mostly as aids to software development.


echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

To permanently allow it edit /etc/sysctl.d/10-ptrace.conf and change the line: 

kernel.yama.ptrace_scope = 1

To read: 

kernel.yama.ptrace_scope = 0

strace

ltrace

ftrace

  • Ftrace - an internal tracer designed to help out developers and designers of systems to find what is going on inside the kernel. It can be used for debugging or analyzing latencies and performance issues that take place outside of user-space. Although ftrace is typically considered the function tracer, it is really a frame work of several assorted tracing utilities. There's latency tracing to examine what occurs between interrupts disabled and enabled, as well as for preemption and from a time a task is woken to the task is actually scheduled in.





perf-tools

  • iosnoop: trace disk I/O with details including latency. Examples.
  • iolatency: summarize disk I/O latency as a histogram. Examples.
  • execsnoop: trace process exec() with command line argument details. Examples.
  • opensnoop: trace open() syscalls showing filenames.
  • killsnoop: trace kill() signals showing process and signal details. Examples.
  • fs/cachestat: basic cache hit/miss statistics for the Linux page cache. Examples.
  • net/tcpretrans: show TCP retransmits, with address and other details. Examples.
  • system/tpoint: trace a given tracepoint. Examples.
  • kernel/funccount: count kernel function calls, matching a string with wildcards. Examples.
  • kernel/functrace: trace kernel function calls, matching a string with wildcards. Examples.
  • kernel/funcslower: trace kernel functions slower than a threshold. Examples.
  • kernel/funcgraph: trace a graph of kernel function calls, showing children and times. Examples.
  • kernel/kprobe: dynamically trace a kernel function call or its return, with variables. Examples.
  • user/uprobe: dynamically trace a user-level function call or its return, with variables. Examples.
  • tools/reset-ftrace: reset ftrace state if needed. Examples.

Using perf_events:

  • misc/perf-stat-hist: power-of aggregations for tracepoint variables. Examples.
  • syscount: count syscalls by syscall or process. Examples.
  • disk/bitesize: histogram summary of disk I/O size. Examples.


Using eBPF: As a preview of things to come, see the bcc tracing Tools section. These use bcc, a front end for using eBPF. bcc+eBPF will allow some of these tools to be rewritten and improved, and additional tools to be created.



  • opensnoop For Linux - "Here's another tool I didn't think would be possible: what files are being opened on my Linux system? This is tracing all processes system-wide, and using a more efficient tracer than strace."


trace-cmd

  • trace-cmd(1) - Linux man page - interacts with the Ftrace tracer that is built inside the Linux kernel. It interfaces with the Ftrace specific files found in the debugfs file system under the tracing directory. A COMMAND must be specified to tell trace-cmd what to do.


trace-cmd list -f
  # get a list of all the functions you can trace

trace-cmd record -p function -l do_page_fault

trace-cmd record -p function -P 25314
  # record for PID 25314

trace-cmd report
  # full trace

trace-cmd record -p function_graph -P 25314


KernelShark

  • KernelShark - a front end reader of trace-cmd(1) output. "trace-cmd record" and "trace-cmd extract" create a trace.dat (trace-cmd.dat(5)) file. kernelshark can read this file and produce a graph and list view of its data.


LTT / LTTng

  • https://en.wikipedia.org/wiki/Linux_Trace_Toolkit - a set of tools that is designed to log program execution details from a patched Linux kernel and then perform various analyses on them, using console-based and graphical tools. LTT has been mostly superseded by its successor LTTng (Linux Trace Toolkit Next Generation).
trace 15 foo
  # trace for 15 seconds

traceview foo
  # view results


  • LTTng - Linux Trace Toolkit Next Generation, is a system software package for correlated tracing of the Linux kernel, applications and libraries. The project was originated by Mathieu Desnoyers with an initial release in 2005. Its predecessor is the Linux Trace Toolkit. LTTng uses the Tracepoint instrumentation of the Linux kernel, as well as various other information sources such as kprobes, and the Perf performance monitoring counters. Designed for minimal performance impact and having a near-zero impact when not tracing, it is useful for debugging a wide range of bugs that are otherwise extremely challenging.

LTTng consists of kernel modules (for Linux kernel tracing) and dynamically linked libraries (for application and library tracing). It is controlled by a session daemon, which receives commands from a command line interface, lttng. The Babeltrace project allows translating traces into a human-readable log, and provides a trace reading library, libbabeltrace.

SystemTap

  • SystemTap - provides free software (GPL) infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data. SystemTap provides a simple command line interface and scripting language for writing instrumentation for a live running kernel plus user-space applications. We are publishing samples, as well as enlarging the internal "tapset" script library to aid reuse and abstraction. Among other tracing/probing tools, SystemTap is the tool of choice for complex tasks that may require live analysis, programmable on-line response, and whole-system symbolic access. SystemTap can also handle simple tracing jobs.



Extended BPF / eBPF

  • https://en.wikipedia.org/wiki/Berkeley_Packet_Filter Berkeley Packet Filter - provides a raw interface to data link layers, permitting raw link-layer packets to be sent and received. It is available on most Unix-like operating systems. In addition, if the driver for the network interface supports promiscuous mode, it allows the interface to be put into that mode so that all packets on the network can be received, even those destined to other hosts.BPF supports filtering packets, allowing a userspace process to supply a filter program that specifies which packets it wants to receive. For example, a tcpdump process may only want to receive packets that initiate a TCP connection. BPF only returns packets that pass the filter that the process supplies. This avoids copying unwanted packets from the operating system kernel to the process, greatly improving performance.BPF is sometimes used to refer just to the filtering mechanism, rather than to the entire interface. Some systems, such as Linux and Tru64 UNIX, provide a raw interface to the data link layer other than the BPF raw interface but use the BPF filtering mechanisms for that raw interface.

Since version 3.18, the Linux kernel includes an extended BPF virtual machine, termed extended BPF (eBPF). It can be used for non-networking purposes, such as for attaching eBPF programs to various tracepoints. Since kernel version 3.19, eBPF filters can be attached to sockets, and, since kernel version 4.1, to traffic control classifiers for the ingress and egress networking data path.



BCC

  • BCC - a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of eBPF (Extended Berkeley Packet Filters), a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above. [21]


bpftrace


ply

  • ply - A dynamic tracer for Linux that lets you:Extract arbitrary data, i.e register values, function arguments, stack/heap data, stack traces.Perform in-kernel aggregations on arbitrary data.ply follows the Little Language approach of yore, compiling ply scripts into Linux BPF programs that are attached to kprobes and tracepoints in the kernel. The scripts have a C-like syntax, heavily inspired by dtrace(1) and by extension awk(1).

bpftime

  • bpftime - a full-featured, high-performance eBPF runtime designed to operate in userspace. It offers fast Uprobe and Syscall hook capabilities: Userspace uprobe can be 10x faster than kernel uprobe! and can programmatically hook all syscalls of a process safely and efficiently.

ktrace

  • https://en.wikipedia.org/wiki/ktrace - a utility included with certain versions of BSD Unix and Mac OS X that traces kernel interaction with a program and dumps it to disk for the purposes of debugging and analysis. Traced kernel operations include system calls, namei translations, signal processing, and I/O. ktrace is somewhat similar to Linux's strace, except for being much faster – with strace, every system call executed by the traced program requires context switch to the tracing program and back, while the tracing with ktrace is actually performed by the kernel, so no additional context switches are required.

Trace files generated by ktrace (named ktrace.out by default) can be viewed in human-readable form by using the kdump utility.

Since Mac OS X 10.5 Leopard, ktrace has been replaced by DTrace.

DTrace

  • https://en.wikipedia.org/wiki/DTrace - a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License (CDDL) and has been ported to several other Unix-like systems. In December 2012, Oracle announced the general availability of DTrace for Oracle Linux.


Crashes


  • https://en.wikipedia.org/wiki/Core_dump - crash dump, memory dump, or system dump consists of the recorded state of the working memory of a computer program at a specific time, generally when the program has crashed or otherwise terminated abnormally. In practice, other key pieces of program state are usually dumped at the same time, including the processor registers, which may include the program counter and stack pointer, memory management information, and other processor and operating system flags and information. A snapshot dump (or snap dump) is a memory dump requested by the computer operator or by the running program, after which the program is able to continue. Core dumps are often used to assist in diagnosing and debugging errors in computer programs.


  • https://wiki.archlinux.org/index.php/Core_dump - a file containing a process's address space (memory) when the process terminates unexpectedly. Core dumps may be produced on-demand (such as by a debugger), or automatically upon termination. Core dumps are triggered by the kernel in response to program crashes, and may be passed to a helper program (such as systemd-coredump) for further processing. A core dump is not typically used by an average user, but may be passed on to developers upon request where it can be invaluable as a post-mortem snapshot of the program's state at the time of the crash, especially if the fault is hard to reliably reproduce.


  • systemd-coredump - a system service that can acquire core dumps from the kernel and handle them in various ways. The systemd-coredump executable does the actual work. It is invoked twice: once as the handler by the kernel, and the second time in the systemd-coredump@.service to actually write the data to the journal.



kdump

  • https://en.wikipedia.org/wiki/kdump_(Linux) - a feature of the Linux kernel that creates crash dumps in the event of a kernel crash. When triggered, kdump exports a memory image (also known as vmcore) that can be analyzed for the purposes of debugging and determining the cause of a crash. The dumped image of main memory, exported as an Executable and Linkable Format (ELF) object, can be accessed either directly through /proc/vmcore during the handling of a kernel crash, or it can be automatically saved to a locally accessible file system, to a raw device, or to a remote system accessible over network.

Apport

  • https://wiki.ubuntu.com/Apport - a system which: intercepts crashes right when they happen the first time, gathers potentially useful information about the crash and the OS environment, can be automatically invoked for unhandled exceptions in other programming languages (e. g. in Ubuntu this is done for Python), can be automatically invoked for other problems that can be automatically detected (e. g. Ubuntu automatically detects and reports package installation/upgrade failures from update-manager), presents a UI that informs the user about the crash and instructs them on how to proceed, and is able to file non-crash bug reports about software, so that developers still get information about package versions, OS version etc.

Debugging

gdb




  • https://github.com/cyrus-and/gdb-dashboard - Modular visual interface for GDB in Python.This comes as a standalone single-file .gdbinit which, among the other things, enables a configurable dashboard showing the most relevant information during the program execution. Its main goal is to reduce the number of GDB commands issued to inspect the current program status allowing the programmer to focus on the control flow instead.




  • https://kgdb.wiki.kernel.org/index.php/Main_Page - a debugger for the Linux kernel and the kernels of NetBSD and FreeBSD. It requires two machines that are connected via a serial connection. The serial connection may either be an RS-232 interface using a null modem cable, or via the UDP/IP networking protocol (KGDB over Ethernet, KGDBoE). The target machine (the one being debugged) runs the patched kernel and the other (host) machine runs gdb. The GDB remote protocol is used between the two machines.


  • cgdb - a lightweight curses (terminal-based) interface to the GNU Debugger (GDB). In addition to the standard gdb console, cgdb provides a split screen view that displays the source code as it executes. The keyboard interface is modeled after vim, so vim users should feel at home using cgdb.





  • GEF - GDB Enhanced Features, a kick-ass set of commands for X86, ARM, MIPS, PowerPC and SPARC to make GDB cool again for exploit dev. It is aimed to be used mostly by exploit developers and reverse-engineers, to provide additional features to GDB using the Python API to assist during the process of dynamic analysis and exploit development.It has full support for both Python2 and Python3 indifferently (as more and more distros start pushing gdb compiled with Python3 support).

Valgrind

  • Valgrind - an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.
    • https://en.wikipedia.org/wiki/Valgrind - a programming tool for memory debugging, memory leak detection, and profiling. Valgrind was originally designed to be a free memory debugging tool for Linux on x86, but has since evolved to become a generic framework for creating dynamic analysis tools such as checkers and profilers.



  • Callgrind / KCachegrind - profiling tool and the profile data visualization Both are licensed under GPL V2. Callgrind uses runtime instrumentation via the Valgrind framework for its cache simulation and call-graph generation. This way, even shared libraries and dynamically opened plugins can be profiled. The data files generated by Callgrind can be loaded into KCachegrind for browsing the performance results. But there is also a command line tool in the package to get ASCII reports from data files without the need to use KCachegrind. The format of Callgrind output is documented here. With conversion scripts, KCachegrind is able to visualize output of other profilers like OProfile, a system-wide profiler for Linux using statistical sampling with hardware performance counters. There also exist converters for profiling output of Python, PHP and PERL.


heapusage

  • https://github.com/d99kris/heapusage - a light-weight tool for finding memory leaks in Linux and macOS applications. It provides a small but important subset of Valgrind's memcheck functionality, and can be a useful alternative to it for debugging memory leaks in certain scenarios such as: Large complex applications which cannot be run at Valgrind slowdown speed Embedded systems with CPU architectures not supported by Valgrind

Sysprof

  • Sysprof - Statistical, system-wide Profiler for Linux

rr

  • rr - aspires to be your primary C/C++ debugging tool for Linux, replacing — well, enhancing — gdb. You record a failure once, then debug the recording, deterministically, as many times as you want. The same execution is replayed every time. rr also provides efficient reverse execution under gdb. Set breakpoints and data watchpoints and quickly reverse-execute to where they were hit.


QIRA


radare2

  • radare - started as a forensics tool, a scriptable commandline hexadecimal editor able to open disk files, but later support for analyzing binaries, disassembling code, debugging programs, attaching to remote gdb servers, .. r2 is a rewrite from scratch of radare in order to provide a set of libraries and tools to work with binary files.





voltron

  • https://github.com/snare/voltron - an extensible debugger UI toolkit written in Python. It aims to improve the user experience of various debuggers (LLDB, GDB, VDB and WinDbg) by enabling the attachment of utility views that can retrieve and display data from the debugger host. By running these views in other TTYs, you can build a customised debugger user interface to suit your needs. [27]


ProcDump-for-Linux

  • https://github.com/Microsoft/ProcDump-for-Linux - a Linux reimagining of the classic ProcDump tool from the Sysinternals suite of tools for Windows. ProcDump provides a convenient way for Linux developers to create core dumps of their application based on performance triggers. [28]


coala

  • coala - provides a unified interface for linting and fixing code with a single configuration file, regardless of the programming languages used. You can use coala from within your favorite editor, integrate it with your CI, get the results as JSON, or customize it to your needs with its flexible configuration syntax. [29]



Senseye

  • https://github.com/letoram/senseye - Dynamic Visual Debugging / Reverse Engineering Toolsuite. a set of data providers, parsers that work with the Arcan display servers IPC subsystem, and as a set of extension script for the Durden desktop environment/window management scheme.

Ghidra

  • Ghidra - a software reverse engineering (SRE) framework developed by NSA's Research Directorate for NSA's cybersecurity mission. It helps analyze malicious code and malware like viruses, and can give cybersecurity professionals a better understanding of potential vulnerabilities in their networks and systems.NSA will be making Ghidra available to the public as an open source release in time for its first public demonstration at the 2019 RSA Conference this March. For more NSA releases, check out CODE.NSA.GOV for open source, and NSA’s Technology Transfer Program for other technology. [30]

dbgtools

dbg-macro

drgn

Sourcetrail

GNU Binutils

The main ones are:

  • ld - the GNU linker.
  • as - the GNU assembler.
  • gold - a new, faster, ELF only linker.

But they also include:

  • addr2line - Converts addresses into filenames and line numbers.
  • ar - A utility for creating, modifying and extracting from archives.
  • c++filt - Filter to demangle encoded C++ symbols.
  • dlltool - Creates files for building and using DLLs.
  • elfedit - Allows alteration of ELF format files.
  • gprof - Displays profiling information.
  • nlmconv - Converts object code into an NLM.
  • nm - Lists symbols from object files.
  • objcopy - Copies and translates object files.
  • objdump - Displays information from object files.
  • ranlib - Generates an index to the contents of an archive.
  • readelf - Displays information from any ELF format object file.
  • size - Lists the section sizes of an object or archive file.
  • strings - Lists printable strings from files.
  • strip - Discards symbols.
  • windmc - A Windows compatible message compiler.
  • windres - A compiler for Windows resource files.



  • objdump - displays information about one or more object files. The options control what particular information to display. This information is mostly useful to programmers who are working on the compilation tools, as opposed to programmers who just want their program to compile and work.objfile… are the object files to be examined. When you specify archives, objdump shows information on each of the member object files.

edb-debugger

  • x64dbg - a cross platform AArch32/x86/x86-64 debugger. It was inspired by Ollydbg, but aims to function on AArch32, x86, and x86-64 as well as multiple OS's. Linux is the only officially supported platform at the moment, but FreeBSD, OpenBSD, OSX and Windows ports are underway with varying degrees of functionality.

YARA

  • https://github.com/VirusTotal/yara - a tool aimed at (but not limited to) helping malware researchers to identify and classify malware samples. With YARA you can create descriptions of malware families (or whatever you want to describe) based on textual or binary patterns. Each description, a.k.a. rule, consists of a set of strings and a boolean expression which determine its logic.

RAD Debugger

  • https://github.com/EpicGames/raddebugger - a native, user-mode, multi-process, graphical debugger. It currently only supports local-machine Windows x64 debugging with PDBs, with plans to expand and port in the future. In the future we'll expand to also support native Linux debugging and DWARF debug info.

Performance

  • make-linux-fast-again.com - "noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off mitigations=off"


General

perf




Gprof




turbostat


hegemon

hotspot

  • https://github.com/KDAB/hotspot - This project is a KDAB R&D effort to create a standalone GUI for performance data. As the first goal, we want to provide a UI like KCachegrind around Linux perf. Looking ahead, we intend to support various other performance data formats under this umbrella.

tracy

  • https://github.com/wolfpld/tracy - A real time, nanosecond resolution, remote telemetry, hybrid frame and sampling profiler for games and other applications.Tracy supports profiling CPU (C, C++11, Lua), GPU (OpenGL, Vulkan, OpenCL, Direct3D 12), memory, locks, context switches, per-frame screenshots and more.

OProfile

magic-trace

heaptrack

memory-profiler

Coz

  • https://github.com/plasma-umass/coz - a new kind of profiler that unlocks optimization opportunities missed by traditional profilers. Coz employs a novel technique we call causal profiling that measures optimization potential. This measurement matches developers' assumptions about profilers: that optimizing highly-ranked code will have the greatest impact on performance. Causal profiling measures optimization potential for serial, parallel, and asynchronous programs without instrumentation of special handling for library calls and concurrency primitives. Instead, a causal profiler uses performance experiments to predict the effect of optimizations. This allows the profiler to establish causality: "optimizing function X will have effect Y," exactly the measurement developers had assumed they were getting all along.

CPU

CoreFreq

  • https://github.com/cyring/CoreFreq - a CPU monitoring software designed for 64-bits Processors w/ architectures Intel Atom, Core2, Nehalem, SandyBridge and superior, AMD Family 0F


Power

tlp

  • linrunner.de: TLP – Linux Advanced Power Management - brings you the benefits of advanced power management for Linux without the need to understand every technical detail. TLP comes with a default configuration already optimized for battery life, so you may just install and forget it. Nevertheless TLP is highly customizable to fulfil your specific requirements. TLP packages are available for Arch, Debian, Fedora, Gentoo, openSUSE, Ubuntu and further. You're welcome if you want to package TLP for other distributions. Please take a look at the developer documentation.


PowerTOP

  • PowerTOP - a Linux tool to diagnose issues with power consumption and power management. In addition to being a diagnostic tool, PowerTOP also has an interactive mode where the user can experiment various power management settings for cases where the Linux distribution has not enabled these settings. The second generation of PowerTOP (version 2.0 and later) is a complete redesign of the core code, and offers more comprehensive diagnostic capabilities, a more intuitive interactive mode and reporting modes that are helpful for QA teams and their automated tools. PowerTOP will report which components in the system are most likely to blame for a higher-than-needed power consumption, from software applications to active components in the system. Detailed screens are available for CPU C and P states, Device activity as well as Software activity.

Benchmarks


  • OpenBenchmarking.org - an open, collaborative testing platform designed by Phoronix Media and the developers behind the Phoronix Test Suite, the most comprehensive benchmarking platform for Linux and other operating systems. OpenBenchmarking.org makes the Phoronix Test Suite an even more extensible platform for conducting automated tests with complete integration into Phoronix Test Suite 3.0-Iveland as well as within Phoromatic, an online test remote management system designed for managing test farms within enterprise environments.


  • http://www.userbenchmark.com/page/about - run by a small company of hardcore geeks. We are dedicated to providing our visitors with top notch hardware research on tens of thousands of PC components. If you have any feedback please contact us using the email link at the bottom of this page.


  • Geekbench - measures your device's CPU and Compute performance. Use the Geekbench Browser to organize and share your results with others around the world.


  • Iozone Filesystem Benchmark - a filesystem benchmark tool. The benchmark generates and measures a variety of file operations. Iozone has been ported to many machines and runs under many operating systems.



  • https://github.com/n-st/nench - VPS benchmark script — based on the popular bench.sh, plus CPU and ioping tests, and dual-stack IPv4 and v6 speedtests by default




  • https://github.com/andrewrk/poop - Performance Optimizer Observation PlatformStop flushing your performance down the drain.OverviewThis command line tool uses Linux's perf_event_open functionality to compare the performance of multiple commands with a colorful terminal user interface.

to sort





  • Flame - graphs are a visualization of profiled software, allowing the most frequent code-paths to be identified quickly and accurately. They can be generated using my open source programs on github.com/brendangregg/FlameGraph, which create interactive SVGs. See the Updates section for other implementations.


  • jtool - a command meant to meet and exceed the functionality to XCode's otool(1), picking up along the way additional Mach-O commands such as atos(1), dyldinfo(1), nm(1), segedit(1), pagestuff(1), strings(1), and even codesign(1) and the informal ldid. jtool also provides novel features, such in-binary search functionality, symbol injection and a disassembler functionality with (limited but constantly improving) emulation capabilities. It also provides color output. Most importantly, it can be run on a variety of platforms - OS X, iOS, and even Linux. It is ENTIRELY FREE for use of any type, and the latest version




  • Forkstat - a program that logs process fork(), exec() and exit() activity. It is useful for monitoring system behaviour and to track down rogue processes that are spawning off processes and potentially abusing the system.










  • Dwarf Home - a debugging file format used by many compilers and debuggers to support source level debugging. It addresses the requirements of a number of procedural languages, such as C, C++, and Fortran, and is designed to be extensible to other languages. DWARF is architecture independent and applicable to any processor or operating system. It is widely used on Unix, Linux and other operating systems, as well as in stand-alone environments.


  • https://github.com/google/sanitizers - This project is the home for Sanitizers: AddressSanitizer, MemorySanitizer, ThreadSanitizer, LeakSanitizer. The actual code resides in the LLVM repository. Here we keep extended documentation, bugs and some helper code.


  • https://github.com/ColinIanKing/stress-ng -stress test a computer system in various selectable ways. It was designed to exercise various physical subsystems of a computer as well as the various operating system kernel interfaces.


Decompiling

Retrieved from "https://wiki.thingsandstuff.org/index.php?title=Debugging&oldid=64438"