Logging
Jump to navigation
Jump to search
General
- Linux log files location and how do I view logs files? - Jul 17, 2006
- 20 Linux Log Files that are Located under /var/log Directory - Aug 1, 2011
logrotate
- logrotate is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.
- Using logrotate to rotate and archive log - Aug 05, 2010
- HowTo: The Ultimate Logrotate Command Tutorial with 10 Examples - Jul 14, 2010
Systems
syslog
- syslog
- syslogd
- /etc/syslog.conf
- How To Set Up A Debian Linux Syslog Server
- https://tools.ietf.org/html/rfc5424 - The Syslog Protocol
- logger - a shell command interface to the syslog(3) system log module
syslog-ng
- syslog-ng is a high-performance syslog server with advanced log processing services and direct database access. syslog-ng clients collect the log messages from the various applications, files, and other sources then send important log messages to the remote syslog-ng server which sorts and stores them.
rsyslog
- Rsyslog is an enhanced syslogd supporting, among others, MySQL, PostgreSQL, failover log destinations, syslog/tcp, fine grain output format control, high precision timestamps, queued operations, the ability to filter on any message part and encryption protected syslog relay chains. It is quite compatible to stock sysklogd and can be used as a drop-in replacement.
journald
systemd's journal service
- http://www.freedesktop.org/software/systemd/man/systemd-journald.service.html
- http://www.freedesktop.org/software/systemd/man/journald.conf.html
- https://wiki.archlinux.org/index.php/Systemd#Journal
- https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs
- lnav - The Log File Navigator, An advanced log file viewer for the small-scale. Watch and analyze your log files from a terminal. No server. No setup. Still featureful. [2]
Aggregation
swatch
- swatch - swatchdog.pl started out as swatch, the "simple watchdog" for activity monitoring log files produced by UNIX's syslog facility. It has since been evolving into a utility that can monitor just about any type of log. The name has been changed to satisfy a request made by the old Swiss watch company.
MultiTail
- MultiTail - allows you to monitor logfiles and command output in multiple windows in a terminal, colorize, filter and merge.
Logwatch
- Logwatch is a customizable log analysis system. Logwatch parses through your system's logs and creates a report analyzing areas that you specify. Logwatch is easy to use and will work right out of the package on most systems.
Logcheck
- Logcheck is a simple utility which is designed to allow a system administrator to view the logfiles which are produced upon hosts under their control. It does this by mailing summaries of the logfiles to them, after first filtering out "normal" entries.
logstash
- logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.
Dendrite
- Dendrite scrapes your existing logs and re-emits a unified log stream in modern, sensible, structured formats like JSON and StatsD over common protocols such as TCP, UDP, file streams
l2met
- https://github.com/ryandotsmith/l2met - Convert a formatted log stream into metrics. Heroku.
Graylog2
- Graylog2 enables you to unleash the power that lays inside your logs. Use it to run analytics, alerting, monitoring and powerful searches over your whole log base. Need to debug a failing request? Just run a quick filter search to find it and see what errors it produced. Want to see all messages a certain API consumer is consuming in real time? Create streams for every consumer and have them always only one click away.
GoAccess
- GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems. It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly.
- lnav - An advanced log file viewer for the small-scale. Watch and analyze your log files from a terminal. No server. No setup. Still featureful.
Flume
- Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
Fluentd
- Fluentd is an open-source tool to collect events and logs. 150+ plugins instantly enables you to store the massive data for Log Search, Big Data Analytics, and Archiving (MongoDB, S3, Hadoop).
Kibana
- Kibana is an open source (Apache Licensed), browser based analytics and search interface to Logstash and other timestamped data sets stored in ElasticSearch. With those in place Kibana is a snap to setup and start using (seriously). Kibana strives to be easy to get started with, while also being flexible and powerful
log.io
- log.io - Real-time log monitoring in your browser. Powered by node.js + socket.io
http://www.vanheusden.com/multitail/
glTail.rb
- glTail.rb - Real-time visualization of server log traffic, events and statistics with Ruby, SSH and OpenGL.
logstalgia
- The Log File Navigator - An advanced log file viewer for the small-scale. Watch and analyze your log files from a terminal. No server. No setup. Still featureful.
slit
- https://github.com/tigrawap/slit - a modern PAGER for viewing logs, get more than most in less time
loki
- https://github.com/grafana/loki - Like Prometheus, but for logs.
Access Watch
- Access Watch - Open Source Web Traffic Processor. A flexible stream processor for developers to analyze and control web traffic.
Telegraf
- https://github.com/influxdata/telegraf - an agent for collecting, processing, aggregating, and writing metrics.Design goals are to have a minimal memory footprint with a plugin system so that developers in the community can easily add support for collecting metrics.Telegraf is plugin-driven and has the concept of 4 distinct plugin types: Input Plugins collect metrics from the system, services, or 3rd party APIs. Processor Plugins transform, decorate, and/or filter metrics. Aggregator Plugins create aggregate metrics (e.g. mean, min, max, quantiles, etc.). Output Plugins write metrics to various destinations.
to sort
RRDtool
- RRDtool is the OpenSource industry standard, high performance data logging and graphing system for time series data. RRDtool can be easily integrated in shell scripts, perl, python, ruby, lua or tcl applications.