Log Analysis and Monitoring Tools

Category: DevOps and System Tools
Type: Linux Commands
Generated on: 2025-07-10 03:20:36
For: System Administration, Development & Technical Interviews

---

Log Analysis & Monitoring Cheatsheet (Linux - DevOps/Sysadmin)

This cheatsheet provides a practical guide to log analysis and monitoring tools in Linux, focusing on commands commonly used in DevOps and system administration.

1. tail - Display the End of a File (Real-time Monitoring)

Command Overview: tail displays the last part of a file. Primarily used for monitoring log files in real-time, especially when debugging applications or monitoring system activity.

Basic Syntax:

tail [OPTION]... [FILE]...

Practical Examples:

• Display the last 10 lines of /var/log/syslog:

  tail /var/log/syslog

  ... (last 10 lines of syslog) ...

• Follow /var/log/apache2/access.log in real-time (continuously update the display):

  tail -f /var/log/apache2/access.log

  ... (continuously updated access log entries) ...

Common Options:

• -n NUM: Output the last NUM lines, instead of the last 10. Example: tail -n 20 /var/log/nginx/error.log
• -f: Follow (output appended data as the file grows). Crucial for real-time monitoring. Example: tail -f /var/log/application.log
• -F: Like -f, but reopens a file even if it is renamed or rotated. More robust for log rotation scenarios. Example: tail -F /var/log/application.log
• --retry: Keep trying to open a file even if it is initially inaccessible. Useful if a log file is created later.
• +NUM: Output beginning with line NUM, instead of the last 10. Example: tail +100 /var/log/kern.log (starts at line 100)

Advanced Usage:

• Monitor multiple log files simultaneously:

  tail -f /var/log/syslog /var/log/apache2/error.log

  Output will be interleaved, but tail prefixes each line with the filename.

• Combine tail with grep to filter log entries:

  tail -f /var/log/nginx/access.log | grep "404"

  This displays only access log entries containing “404” (Not Found errors).

Tips & Tricks:

• Use Ctrl+C to stop tail -f.
• For very large log files, consider using less or multitail for better navigation and performance.

Troubleshooting:

• tail: cannot open '/path/to/file' for reading: Permission denied: You don’t have read permissions on the file. Use sudo tail /path/to/file or change file permissions.
• tail: '/path/to/file' has become inaccessible: No such file or directory: The file has been deleted or moved. If using -F, tail will try to reopen it.

Related Commands:

• head: Displays the beginning of a file.
• less: A more versatile pager for viewing files, including log files.
• grep: Filters lines matching a pattern.
• multitail: Display multiple log files in separate windows within a single terminal.

2. grep - Search Text Using Regular Expressions

Command Overview: grep searches input files for lines containing a match to a given pattern (regular expression). Essential for finding specific events, errors, or patterns in log files.

Basic Syntax:

grep [OPTION]... PATTERN [FILE]...

Practical Examples:

• Find all lines in /var/log/syslog containing the word “error”:

  grep "error" /var/log/syslog

  ... (lines from syslog containing "error") ...

• Find all lines in /var/log/nginx/error.log containing the IP address 192.168.1.100:

  grep "192.168.1.100" /var/log/nginx/error.log

  ... (lines from error.log containing the IP address) ...

Common Options:

• -i: Ignore case distinctions (case-insensitive search). Example: grep -i "Error" /var/log/syslog
• -v: Invert the match (select non-matching lines). Example: grep -v "INFO" /var/log/application.log (show all lines that are NOT “INFO” level)
• -c: Count the number of matching lines. Example: grep -c "Exception" /var/log/java.log
• -n: Precede each matching line with its line number. Example: grep -n "warning" /var/log/syslog
• -r or -R: Recursively search directories. Example: grep -r "password" /etc/ (careful with this!)
• -w: Match whole words only. Example: grep -w "user" /var/log/auth.log (matches “user” but not “username”)
• -E: Interpret PATTERN as an extended regular expression (ERE). Example: grep -E "error|warning" /var/log/syslog (finds lines containing either “error” or “warning”)
• -o: Print only the matching part of the line. Example: grep -o "[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}" /var/log/apache2/access.log (extract IP addresses)
• -A NUM: Print NUM lines after the matching line. Example: grep -A 2 "Exception" /var/log/java.log (show the exception and the next 2 lines)
• -B NUM: Print NUM lines before the matching line. Example: grep -B 1 "Error" /var/log/syslog (show the line before the error)
• -C NUM: Print NUM lines before and after the matching line (context). Example: grep -C 3 "Critical" /var/log/application.log (show the critical error and 3 lines before and after)

Advanced Usage:

• Use regular expressions for more complex pattern matching:

  grep "^[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}" /var/log/application.log  # Find lines starting with a date in YYYY-MM-DD format

• Pipe output from other commands to grep:

  ps aux | grep "java"  # Find processes running Java

• Combine grep with awk to extract specific fields:

  grep "Failed password" /var/log/auth.log | awk '{print $10}' # Extract the username from failed password attempts

• Use grep -f to read patterns from a file:

  # Create a file (patterns.txt) with a list of patterns to search for, one per line
  grep -f patterns.txt /var/log/application.log

Tips & Tricks:

• Use single quotes to enclose patterns containing spaces or special characters: grep 'Failed password for invalid user' /var/log/auth.log
• Escape special characters in regular expressions using a backslash: grep ".*\.example\.com" /var/log/nginx/access.log

Troubleshooting:

• grep: command not found: grep is not installed. Install it using your distribution’s package manager (e.g., sudo apt install grep on Debian/Ubuntu, sudo yum install grep on CentOS/RHEL).
• No output: The pattern was not found in the specified file(s), or the pattern is incorrect. Double-check the pattern and the file path.
• Incorrect regular expression syntax: Refer to man grep for details on regular expression syntax. Use -E for extended regular expressions.

Related Commands:

• awk: A powerful text processing tool.
• sed: A stream editor for transforming text.
• find: Find files based on various criteria.
• zgrep, bzgrep, xzgrep: grep variants for compressed files (gzip, bzip2, xz).

3. awk - Text Processing and Data Extraction

Command Overview: awk is a powerful text processing tool that allows you to extract, transform, and report on data within text files. It operates on a line-by-line basis and is particularly useful for parsing structured log files.

Basic Syntax:

awk [OPTION]... 'program' [FILE]...

Where program is typically in the form: pattern { action }

Practical Examples:

• Print the first field of each line in /var/log/apache2/access.log:

  awk '{print $1}' /var/log/apache2/access.log

  This will output the IP address for each request.

• Print the first and seventh fields (IP address and request URL) of each line:

  awk '{print $1, $7}' /var/log/apache2/access.log

Common Options:

• -F FS: Use FS as the input field separator. Example: awk -F ',' '{print $1, $2}' data.csv (use comma as the delimiter)
• -v var=value: Assign a variable before execution. Example: awk -v threshold=100 '{if ($2 > threshold) print $1}' data.txt
• -f program_file: Read the awk program from a file. Useful for complex awk scripts.

Advanced Usage:

• Filter lines based on conditions:

  awk '$9 >= 400 {print $1, $7, $9}' /var/log/apache2/access.log  # Print IP, URL, and status code for requests with status code >= 400

• Calculate statistics:

  awk '{sum += $9} END {print "Total:", sum}' /var/log/apache2/access.log  # Calculate the sum of the status codes.  END block executes after all lines are processed.

• Use BEGIN and END blocks:

  awk 'BEGIN {print "Starting..."} {print $1} END {print "Finished."}' /var/log/syslog

• Use built-in variables:

  • NF: Number of fields in the current record.
  • NR: Number of the current record.
  • FILENAME: Name of the current input file.
  • FS: Input field separator (default is space).
  • OFS: Output field separator (default is space).

  awk '{print "Line " NR ": " $0}' /var/log/syslog # Print line number and the entire line

• Custom field separators:

  awk -F ":" '{print $1, $3}' /etc/passwd  # Print username and user ID from /etc/passwd (colon-separated)

Tips & Tricks:

• Use printf for formatted output: awk '{printf "%-15s %s\n", $1, $7}' /var/log/apache2/access.log (left-justify IP address)
• Combine awk with grep for complex filtering and extraction.

Troubleshooting:

• Syntax errors: awk can be sensitive to syntax. Double-check your program for typos, missing braces, or incorrect variable names.
• Incorrect field separator: If the fields are not being parsed correctly, ensure that you have specified the correct field separator using -F.
• Unexpected output: Verify that the field numbers you are referencing are correct for the structure of your log file.

Related Commands:

• sed: Stream editor for transforming text.
• grep: Search text using regular expressions.
• cut: Remove sections from each line of files.

4. sed - Stream Editor for Text Transformation

Command Overview: sed is a powerful stream editor that can perform text transformations on input streams or files. It’s particularly useful for automating repetitive editing tasks on log files or configuration files.

Basic Syntax:

sed [OPTION]... 'command' [FILE]...

Practical Examples:

• Replace all occurrences of “error” with “ERROR” in /var/log/syslog and print to the console:

  sed 's/error/ERROR/g' /var/log/syslog

• Delete all lines containing the word “debug” from /var/log/application.log and print to the console:

  sed '/debug/d' /var/log/application.log

Common Options:

• -i: Edit the file in place (modify the original file). USE WITH CAUTION! Consider backing up the file first. Example: sed -i 's/old/new/g' config.txt
• -i.bak: Edit the file in place, creating a backup with the .bak extension. Safer than -i. Example: sed -i.bak 's/old/new/g' config.txt
• -n: Suppress automatic printing of pattern space. Used with the p command to print only specific lines. Example: sed -n '/error/p' /var/log/syslog (print only lines containing “error”)
• -e command: Execute multiple sed commands. Example: sed -e 's/error/ERROR/g' -e '/debug/d' /var/log/application.log

Advanced Usage:

• Replace only the first occurrence of a pattern on each line:

  sed 's/first/FIRST/' /var/log/application.log

• Replace occurrences only on specific lines:

  sed '5,10s/old/new/g' /var/log/config.txt  # Replace 'old' with 'new' on lines 5 through 10
  sed '10s/old/new/g' /var/log/config.txt # Replace 'old' with 'new' on line 10 only
  sed '/pattern/s/old/new/g' /var/log/config.txt  # Replace 'old' with 'new' on lines matching 'pattern'

• Delete lines based on a range:

  sed '5,10d' /var/log/config.txt # Delete lines 5 through 10

• Insert text before or after a matching line:

  sed '/pattern/i\This is a new line before' /var/log/config.txt  # Insert a line before lines matching 'pattern'
  sed '/pattern/a\This is a new line after' /var/log/config.txt  # Insert a line after lines matching 'pattern'

• Use regular expressions for more complex substitutions:

  sed 's/\([0-9]\{3\}\)\.\([0-9]\{3\}\)\.\([0-9]\{3\}\)\.\([0-9]\{3\}\)/\1-\2-\3-\4/g' /var/log/apache2/access.log  # Replace dots in IP addresses with hyphens

Tips & Tricks:

• Always test your sed commands without the -i option first to preview the changes.
• Use the -i.bak option to create a backup before modifying the file in place.
• For complex transformations, consider using a separate sed script file.

Troubleshooting:

• sed: -e expression #1, char 1: unknown command: ...: Syntax error in the sed command. Check the command carefully.
• File not modified: If you are using -i and the file is not being modified, you may not have write permissions on the file.
• Unexpected results: Carefully review your regular expressions and sed commands to ensure they are doing what you intend.

Related Commands:

• awk: Text processing and data extraction.
• grep: Search text using regular expressions.
• tr: Translate or delete characters.

5. journalctl - View Systemd Logs

Command Overview: journalctl is the command-line utility for querying and displaying logs collected by systemd, the system and service manager. It’s the primary tool for accessing system-level logs in modern Linux distributions.

Basic Syntax:

journalctl [OPTION]... [MATCHES]...

Practical Examples:

• View all system logs:

  journalctl

• View logs for the current boot:

  journalctl -b

• View logs for the previous boot:

  journalctl -b -1

• View logs for a specific unit (service):

  journalctl -u nginx.service

Common Options:

• -b: Show logs for the specified boot. -b 0 for current boot, -b -1 for previous boot, etc.
• -u UNIT: Show logs for the specified systemd unit (service, socket, etc.).
• -f: Follow the log in real-time, similar to tail -f.
• -k: Show kernel messages only.
• -p PRIORITY: Filter by priority level. PRIORITY can be emerg, alert, crit, err, warning, notice, info, or debug. Example: journalctl -p err (show only error messages)
• --since=DATE and --until=DATE: Filter by time range. Dates can be relative (e.g., “yesterday”, “1 hour ago”) or absolute (e.g., “2023-10-27 10:00:00”). Examples:
  • journalctl --since="yesterday"
  • journalctl --since="2023-10-26" --until="2023-10-27"
  • journalctl --since="1 hour ago"
• -o FORMAT: Specify output format. Common formats include short, verbose, export, json, and cat. Example: journalctl -o verbose
• --disk-usage: Show current disk space used by journal logs.

Advanced Usage:

• Combine multiple filters:

  journalctl -u nginx.service -p err --since="1 hour ago"

  This shows error messages from the nginx service within the last hour.

• Filter by specific fields (MATCHES):

  journalctl SYSLOG_IDENTIFIER=nginx
  journalctl _PID=1234

• Persist journal logs across reboots (by default, logs are stored in RAM):

  sudo mkdir -p /var/log/journal
  sudo systemctl restart systemd-journald

Tips & Tricks:

• Use Tab completion to help with unit names and other options.
• Press Shift+G in journalctl to go to the end of the log. Use / to search.
• Use Ctrl+C to exit journalctl -f.

Troubleshooting:

• journalctl: command not found: Systemd is not installed or the journal is not enabled.
• No output: No logs match the specified filters or time range.
• Permissions issues: You may need sudo to view certain logs, especially system-level logs.

Related Commands:

• systemctl: Control the systemd system and service manager.
• logger: A command-line interface to the syslog system.
• /var/log/: Traditional log directory (though many services now use systemd journal).

6. dmesg - Display Kernel Ring Buffer

Command Overview: dmesg displays the kernel ring buffer, which contains messages from the kernel, including hardware detection, device driver initialization, and error messages. Useful for troubleshooting hardware issues or kernel-related problems.

Basic Syntax:

dmesg [OPTION]...

Practical Examples:

• Display the entire kernel ring buffer:

  dmesg

• Clear the kernel ring buffer (requires root privileges):

  sudo dmesg -c

  WARNING: This will clear the buffer and make it harder to diagnose past issues. Use with caution.

Common Options:

• -H: Human-readable output (colorized and paginated).
• -w or --follow: Follow the kernel ring buffer in real-time, similar to tail -f.
• -l LEVEL: Filter by log level. Levels are emerg, alert, crit, err, warn, notice, info, and debug. Example: dmesg -l err,warn (show errors and warnings)
• -x: Decode facility and level names.
• --time-format=FORMAT: Specify the timestamp format. ctime is the default. iso provides ISO 8601 timestamps. Example: dmesg --time-format=iso

Advanced Usage:

• Filter dmesg output using grep:

  dmesg | grep "error"

  This displays only kernel messages containing the word “error”.

• Display dmesg output with ISO 8601 timestamps and filter for USB-related messages:

  dmesg --time-format=iso | grep "usb"

Tips & Tricks:

• Use dmesg -w to monitor for new hardware events or errors in real-time.
• Examine dmesg output after plugging in a new USB device or experiencing hardware issues.

Troubleshooting:

• dmesg: read kernel buffer failed: Operation not permitted: You need root privileges to run dmesg. Use sudo dmesg.
• No output: No messages in the kernel ring buffer, or the messages have already scrolled off the screen.

Related Commands:

• journalctl: View systemd logs (often contains kernel messages as well).
• lspci, lsusb: List PCI and USB devices, respectively.
• /proc/kmsg: Directly access the kernel ring buffer (less common).

7. zgrep, bzgrep, xzgrep - Grep on Compressed Files

Command Overview: These commands are variants of grep designed to search directly within compressed files without needing to decompress them first. zgrep works with gzip-compressed files (.gz), bzgrep with bzip2-compressed files (.bz2), and xzgrep with xz-compressed files (.xz).

Basic Syntax:

zgrep [OPTION]... PATTERN [FILE]...
bzgrep [OPTION]... PATTERN [FILE]...
xzgrep [OPTION]... PATTERN [FILE]...

Practical Examples:

• Search for “error” in a gzip-compressed log file:

  zgrep "error" /var/log/apache2/access.log.gz

• Search for “warning” in a bzip2-compressed log file:

  bzgrep "warning" /var/log/application.log.bz2

• Search for “exception” in an xz-compressed log file:

  xzgrep "exception" /var/log/java.log.xz

Common Options:

These commands generally support the same options as grep, including:

• -i: Ignore case.
• -v: Invert match.
• -c: Count matching lines.
• -n: Show line numbers.
• -E: Extended regular expressions.
• -A NUM, -B NUM, -C NUM: Context lines.

Advanced Usage:

• Use regular expressions with zgrep, bzgrep, and xzgrep:

  zgrep -E "404|500" /var/log/nginx/access.log.gz  # Search for 404 or 500 errors

• Pipe output from other commands to zgrep, bzgrep, or xzgrep:

  ls -l /var/log/*.gz | zgrep "access.log" # List gzip files containing "access.log" in their names

Tips & Tricks:

• These commands are significantly faster than manually decompressing the file and then using grep.
• Ensure the correct *grep variant is used for the corresponding compression format.

Troubleshooting:

• zgrep: command not found, bzgrep: command not found, xzgrep: command not found: These commands may not be installed by default. Install them using your distribution’s package manager (e.g., sudo apt install gzip, sudo apt install bzip2, sudo apt install xz-utils on Debian/Ubuntu, and ensure the grep package is installed too).
• “Not a gzip/bzip2/xz file”: The specified file is not in the expected compression format.
• Incorrect output: Double-check the pattern and ensure it is compatible with the compression format.

Related Commands:

• grep: Search text using regular expressions (on uncompressed files).
• gzip, bzip2, xz: Compression utilities.
• zcat, bzcat, xzcat: Display the contents of compressed files without decompressing them to disk.

This cheatsheet covers essential log analysis and monitoring commands. Remember to always consult the man pages (man command) for the most up-to-date and comprehensive information. Practice and experimentation are key to mastering these powerful tools.