Security Hardening and Monitoring

Category: Advanced Linux Administration
Type: Linux Commands
Generated on: 2025-07-10 03:17:41
For: System Administration, Development & Technical Interviews

Security Hardening & Monitoring - Linux Commands Cheatsheet

This cheat sheet provides a comprehensive guide to Linux commands used for security hardening and monitoring. It’s designed for both system administrators and developers, offering practical examples and advanced techniques.

1. Command Overview

These commands are essential for securing your Linux systems, from initial hardening to continuous monitoring for potential threats and vulnerabilities. They allow you to control access, audit system activity, detect intrusions, and maintain a secure environment.

2. Basic Syntax

command [options] [arguments]

3. Practical Examples

iptables / nftables (Firewall Management)

Command Overview: iptables (legacy) and nftables (modern) are used to configure the Linux kernel’s built-in firewall. They define rules for allowing or blocking network traffic based on various criteria (source/destination IP, port, protocol, etc.). nftables is generally preferred due to its improved syntax and performance.
Basic Syntax (nftables):
Terminal window
```
nft [options] command object
```

Basic Syntax (iptables):

iptables [-t table] command [chain] rule-specification [options]

Practical Examples (nftables):

# List all nftables rules
nft list rules

# Create a new filter table
nft add table inet filter

# Create an input chain
nft add chain inet filter input { type filter hook input priority 0 \; policy drop \; }

# Allow SSH traffic
nft add rule inet filter input tcp dport 22 ct state new accept

# Allow established and related connections
nft add rule inet filter input ct state {established, related} accept

# Drop all other incoming traffic
nft add rule inet filter input drop

# Save the configuration (Debian/Ubuntu)
nft list rules > /etc/nftables.conf
systemctl restart nftables.service

# Load the configuration on boot
systemctl enable nftables.service

Practical Examples (iptables):

# List all iptables rules
iptables -L

# Allow SSH traffic (port 22)
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP traffic (port 80)
iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# Allow HTTPS traffic (port 443)
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Allow established and related connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Drop all other incoming traffic
iptables -P INPUT DROP

# Save iptables rules (Debian/Ubuntu)
iptables-save > /etc/iptables/rules.v4

# Load iptables rules on boot (Debian/Ubuntu)
# Create /etc/network/if-pre-up.d/iptablesload:
# #!/bin/sh
# iptables-restore < /etc/iptables/rules.v4
# chmod +x /etc/network/if-pre-up.d/iptablesload

# Save iptables rules (CentOS/RHEL)
iptables-save > /etc/sysconfig/iptables

# Load iptables rules on boot (CentOS/RHEL)
systemctl enable iptables.service
systemctl start iptables.service

Sample Output (iptables -L):

Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DROP       all  --  anywhere             anywhere

Common Options (iptables):
- -A chain: Append a new rule to the specified chain.
- -I chain [rulenum]: Insert a new rule at the specified position in the chain.
- -D chain rulenum: Delete a rule from the chain.
- -D chain rule-specification: Delete a rule matching the specification.
- -L chain: List rules in the specified chain. -L -v for verbose output. -L -n for numeric output.
- -F chain: Flush (delete all rules) in the specified chain.
- -P chain target: Set the default policy for the chain.
- -p protocol: Specify the protocol (e.g., tcp, udp, icmp).
- --dport port: Specify the destination port.
- --sport port: Specify the source port.
- -s source: Specify the source IP address or network.
- -d destination: Specify the destination IP address or network.
- -j target: Specify the target action (e.g., ACCEPT, DROP, REJECT).
- -m module: Use a specific module (e.g., conntrack, state).
Common Options (nftables):
- list rules: Lists all the rules
- add table: Creates a new table
- add chain: Creates a new chain
- add rule: Adds a new rule

Advanced Usage (iptables):

# Rate limit incoming SSH connections to prevent brute-force attacks
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set --name SSH --rsource
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 --name SSH --rsource -j DROP

# Log dropped packets
iptables -A INPUT -j LOG --log-prefix "IPTABLES DROP: " --log-level 4

Advanced Usage (nftables):

# Create a set to store IP addresses to block
nft add set inet filter blacklist { type ipv4_addr \; flags interval,timeout \; }

# Add an IP address to the blacklist with a timeout of 1 hour
nft add element inet filter blacklist { 192.168.1.1 timeout 3600s }

# Block traffic from IPs in the blacklist
nft add rule inet filter input ip saddr @blacklist drop

Tips & Tricks:
- Start with a restrictive policy (e.g., DROP all incoming traffic) and then selectively allow necessary traffic.
- Use iptables-save or nft list rules to save your configuration and ensure it’s loaded on boot.
- Test your rules carefully before applying them to a production system.
- Use connection tracking (-m conntrack) to efficiently handle related connections.
- Use logging (-j LOG) to monitor dropped packets and identify potential attacks.
Troubleshooting:
- If you lock yourself out of your system, use a console connection or out-of-band management to revert the changes.
- Check the system logs for error messages related to iptables/nftables.
- Use iptables -L -v -n or nft list rules to verify that your rules are configured correctly.
Related Commands: tcpdump, nmap, netstat, ss

auditd (Auditing)
- Command Overview: auditd is the Linux audit daemon. It collects detailed information about system events, such as file access, command execution, and user logins. This information is invaluable for security auditing, compliance, and forensic analysis.
- Basic Syntax:
  Terminal window
```
auditctl [options]
ausearch [options]
aureport [options]
```
- Practical Examples:
  Terminal window
```
# Start the audit daemon
systemctl start auditd

# Check the status of the audit daemon
systemctl status auditd

# Configure audit rules (e.g., monitor all access to /etc/passwd)
auditctl -w /etc/passwd -p war -k passwd_changes

# List current audit rules
auditctl -l

# Search audit logs for events related to passwd_changes
ausearch -k passwd_changes

# Generate a report of all audit events
aureport -au

# Generate a report of all login events
aureport -l

# Generate a report of all file access events
aureport -f

# Stop the audit daemon
systemctl stop auditd
```
  Sample Output (auditctl -l):
```
LIST_RULES: exit,always watch=/etc/passwd perm=war key=passwd_changes
```
- Common Options:
  - auditctl -w path -p permissions -k key: Add a watch rule for a specific file or directory.
    - path: The file or directory to watch.
    - permissions: The permissions to monitor (r=read, w=write, x=execute, a=attribute change).
    - key: A unique identifier for the rule.
  - auditctl -l: List current audit rules.
  - auditctl -d rule_number: Delete an audit rule.
  - ausearch -k key: Search audit logs for events with a specific key.
  - ausearch -m message_type: Search for specific message types (e.g., USER_LOGIN, SYSCALL).
  - ausearch -ts start_time -te end_time: Search within a specific time range.
  - aureport -au: Generate a user activity report.
  - aureport -l: Generate a login report.
  - aureport -f: Generate a file access report.
- Advanced Usage:
  Terminal window
```
# Monitor all system calls related to user management (e.g., useradd, userdel)
auditctl -a always,exit -F arch=b64 -S adduser,deluser,usermod -k user_management

# Configure auditd to rotate logs daily and keep 30 days of logs
# Edit /etc/audit/auditd.conf:
# freq = daily
# num_logs = 30
# max_log_file = 10
# systemctl restart auditd
```
- Tips & Tricks:
  - Start with a minimal set of audit rules to avoid overwhelming the system with log data.
  - Use descriptive keys to easily identify and search for specific events.
  - Regularly review audit logs to identify suspicious activity.
  - Consider using a log management system to centralize and analyze audit logs.
  - Configure auditd.conf appropriately for log rotation and retention.
- Troubleshooting:
  - If auditd is not running, check the system logs for error messages.
  - Verify that the audit rules are configured correctly using auditctl -l.
  - Ensure that the audit logs are being written to the correct location (default: /var/log/audit/audit.log).
  - If you’re not seeing the expected events in the audit logs, double-check the permissions and parameters of your audit rules.
- Related Commands: strace, lsof, systemd-journald
ps / top / htop (Process Monitoring)
- Command Overview: These commands provide information about running processes on the system. ps is a static snapshot, while top and htop provide real-time monitoring. They are essential for identifying resource-intensive processes, detecting rogue processes, and troubleshooting performance issues.
- Basic Syntax:
  Terminal window
```
ps [options]
top [options]
htop [options]
```
- Practical Examples:
  Terminal window
```
# List all running processes
ps aux

# List processes owned by a specific user
ps -u username

# Find the process ID (PID) of a process by name
ps -ef | grep process_name

# Display real-time process monitoring with top
top

# Display real-time process monitoring with htop (requires installation)
htop
```
  Sample Output (ps aux | head -n 5):
```
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0 168284  6540 ?        Ss   Oct26   0:01 /sbin/init
root         2  0.0  0.0      0     0 ?        S    Oct26   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        I<   Oct26   0:00 [rcu_gp]
root         4  0.0  0.0      0     0 ?        I<   Oct26   0:00 [rcu_par_gp]
```
- Common Options:
  - ps aux: List all processes with detailed information (user, PID, CPU usage, memory usage, etc.).
  - ps -ef: List all processes with full command line information.
  - ps -u username: List processes owned by a specific user.
  - top: Display real-time process monitoring.
  - htop: An interactive process viewer with more features than top (requires installation: sudo apt install htop or sudo yum install htop).
- Advanced Usage:
  Terminal window
```
# Find the process ID (PID) of a process and kill it
PID=$(ps -ef | grep process_name | grep -v grep | awk '{print $2}')
kill -9 $PID

# Monitor processes consuming the most CPU
top -o %CPU

# Monitor processes consuming the most memory
top -o %MEM

# Monitor processes started by a specific user
top -u username
```
- Tips & Tricks:
  - Use grep to filter the output of ps to find specific processes.
  - Use kill to terminate a process. Use kill -9 as a last resort (forces immediate termination).
  - htop provides a more user-friendly interface and more features than top.
  - Pay attention to the CPU and memory usage of processes to identify potential bottlenecks or resource leaks.
  - Monitor processes started by unknown users or processes that are consuming excessive resources.
- Troubleshooting:
  - If a process is unresponsive, try sending a SIGTERM signal (kill -15 PID) before resorting to SIGKILL (kill -9 PID).
  - Check the system logs for error messages related to the process.
  - Use strace to trace the system calls made by a process.
- Related Commands: kill, pmap, lsof, vmstat, iostat
netstat / ss (Network Monitoring)
- Command Overview: netstat (legacy) and ss (modern) are used to display network connections, listening ports, routing tables, and network interface statistics. ss is generally faster and provides more detailed information.
- Basic Syntax:
  Terminal window
```
netstat [options]
ss [options]
```
- Practical Examples:
  Terminal window
```
# List all listening ports (netstat)
netstat -tulnp

# List all listening ports (ss)
ss -tulnp

# List all established TCP connections (netstat)
netstat -t

# List all established TCP connections (ss)
ss -t

# Show network interface statistics (netstat)
netstat -i

# Show network interface statistics (ss)
ss -i

# Show routing table (netstat)
netstat -r

# Show routing table (ss)
ip route show
```
  Sample Output (netstat -tulnp | head -n 5):
```
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      950/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1321/master
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1548/apache2
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1548/apache2
```
- Common Options (netstat):
  - -t: Display TCP connections.
  - -u: Display UDP connections.
  - -l: Display listening sockets.
  - -n: Display numerical addresses (don’t resolve hostnames).
  - -p: Display the process ID (PID) and program name associated with the socket.
  - -r: Display the routing table.
  - -i: Display network interface statistics.
- Common Options (ss):
  - -t: Display TCP connections.
  - -u: Display UDP connections.
  - -l: Display listening sockets.
  - -n: Display numerical addresses (don’t resolve hostnames).
  - -p: Display the process ID (PID) associated with the socket.
  - -i: Display network interface statistics.
  - -s: Display summary statistics.
- Advanced Usage (netstat):
  Terminal window
```
# Find the process listening on a specific port
netstat -tulnp | grep :80

# Monitor network traffic in real-time using watch
watch -n 1 netstat -tulnp
```
- Advanced Usage (ss):
  Terminal window
```
# Find the process listening on a specific port
ss -tulnp | grep :80

# Monitor network traffic in real-time using watch
watch -n 1 ss -tulnp

# Show TCP connections in state ESTABLISHED
ss -t state established
```
- Tips & Tricks:
  - Use netstat -tulnp or ss -tulnp to identify which processes are listening on specific ports.
  - Use netstat -r or ip route show to examine the routing table.
  - Use netstat -i or ss -i to monitor network interface statistics (e.g., packets sent/received, errors).
  - Monitor for unusual network activity, such as connections to unknown IP addresses or ports.
- Troubleshooting:
  - If a service is not listening on the expected port, check the service configuration and logs.
  - If you’re unable to connect to a remote host, check the routing table and firewall rules.
  - Use tcpdump to capture and analyze network traffic.
- Related Commands: tcpdump, nmap, iptables, nftables, ip
lsof (List Open Files)
- Command Overview: lsof lists all open files and the processes that are using them. In Linux, “everything is a file,” so this includes network sockets, pipes, devices, and regular files. It’s useful for identifying which process is using a particular file or socket, troubleshooting resource conflicts, and detecting suspicious activity.
- Basic Syntax:
  Terminal window
```
lsof [options] [file...]
```
- Practical Examples:
  Terminal window
```
# List all open files
lsof

# List open files for a specific process ID (PID)
lsof -p PID

# List open files for a specific user
lsof -u username

# List open files for a specific file
lsof /path/to/file

# List open files for a specific network port
lsof -i :port_number

# List open files for a specific internet address
lsof -i@host:port
```
  Sample Output (lsof | head -n 5):
```
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd     1   root cwd    DIR  253,0     4096    2 /
systemd     1   root rtd    DIR  253,0     4096    2 /
systemd     1   root txt    REG  253,0  1682840 8129 /usr/lib/systemd/systemd
systemd     1   root mem    REG  253,0  1671776 2442 /usr/lib/locale/locale-archive
```
- Common Options:
  - -p PID: List open files for a specific process ID.
  - -u username: List open files for a specific user.
  - /path/to/file: List processes using the specified file.
  - -i [protocol][@hostname|address][:port]: List open files related to network connections.
    - -i TCP:80 - TCP connections on port 80
    - -i UDP - All UDP connections
  - -c command: List open files used by processes whose command names begin with the string command.
  - +D directory: Recursively list all files under a directory. Useful for finding processes holding open files in a directory.
- Advanced Usage:
  Terminal window
```
# Find the process listening on port 80
lsof -i :80

# Find all files opened by the Apache web server
lsof -c apache2

# Find all files opened in /tmp directory
lsof +D /tmp

# Kill all processes using a specific file (use with caution!)
lsof /path/to/file | awk 'NR!=1 {print $2}' | xargs kill -9
```
- Tips & Tricks:
  - Use lsof -i to identify which process is listening on a specific port.
  - Use lsof /path/to/file to determine which process is preventing you from deleting or modifying a file.
  - Use lsof +D /directory to find processes holding open files in a directory.
- Troubleshooting:
  - If you’re unable to delete a file, use lsof to find the process that is using it and terminate the process.
  - If a service is not working correctly, use lsof to check if it has the necessary files open.
- Related Commands: ps, netstat, ss, fuser
find (File Searching)
- Command Overview: The find command is a powerful tool for locating files and directories based on various criteria, including name, size, modification time, permissions, and ownership. It’s essential for security audits, identifying suspicious files, and performing file management tasks.
- Basic Syntax:
  Terminal window
```
find [path...] [expression]
```
- Practical Examples:
  Terminal window
```
# Find all files in the current directory
find .

# Find all files with a specific name
find . -name "filename.txt"

# Find all files with a specific extension
find . -name "*.log"

# Find all files larger than 10MB
find . -size +10M

# Find all files modified in the last 24 hours
find . -mtime -1

# Find all files owned by a specific user
find . -user username

# Find all files with specific permissions (e.g., world-writable)
find . -perm 777
```
  Sample Output (find . -name "*.log"):
```
./access.log
./error.log
./debug.log
```
- Common Options:
  - -name pattern: Find files with a specific name matching the pattern.
  - -type type: Find files of a specific type (f=file, d=directory, l=symbolic link).
  - -size [+|-]size: Find files larger (+) or smaller (-) than the specified size. Use suffixes like k (kilobytes), M (megabytes), G (gigabytes).
  - -mtime [+|-]n: Find files modified more than (+) or less than (-) n days ago.
  - -atime [+|-]n: Find files accessed more than (+) or less than (-) n days ago.
  - -ctime [+|-]n: Find files whose status was changed more than (+) or less than (-) n days ago.
  - -user username: Find files owned by a specific user.
  - -group groupname: Find files owned by a specific group.
  - -perm mode: Find files with specific permissions (e.g., 777, 644). -perm -mode finds files with at least these permissions. -perm /mode finds files with any of these permissions set.
  - -exec command {} \;: Execute a command on each file found. {} is replaced with the filename. \; terminates the command.
  - -print: Print the filename (default action).
  - -delete: Delete the files found (USE WITH EXTREME CAUTION!).
- Advanced Usage:
  Terminal window
```
# Find all world-writable files and directories and print their permissions
find . -perm 777 -print -exec ls -l {} \;

# Find all files modified in the last 7 days and back them up
find . -mtime -7 -exec cp {} /backup/ \;

# Find all empty files and delete them (USE WITH EXTREME CAUTION!)
find . -type f -empty -delete

# Find all files larger than 100MB and compress them
find . -size +100M -exec gzip {} \;
```
- Tips & Tricks:
  - Use -type f to search only for files, and -type d to search only for directories.
  - Use -size, -mtime, -atime, and -ctime to narrow down your search based on file attributes.
  - Use -exec to perform actions on the files found.
  - Be extremely careful when using -delete to avoid accidentally deleting important files. Test your command without -delete first.
  - Use quotes around patterns containing wildcards (e.g., find . -name "*.log") to prevent shell expansion.
- Troubleshooting:
  - If find is not finding the files you expect, double-check the search criteria and the path.
  - If you get a “Permission denied” error, run find as root or with sudo.
  - If you’re using -exec, make sure the command is valid and that you have the necessary permissions to execute it.
- Related Commands: locate, grep, ls
chmod / chown (Permissions and Ownership)
- Command Overview: chmod (change mode) is used to modify the permissions of files and directories. chown (change owner) is used to change the ownership (user and group) of files and directories. These commands are fundamental for controlling access to your system’s resources.
- Basic Syntax:
  Terminal window
```
chmod [options] mode file...
chown [options] user[:group] file...
```
- Practical Examples:
  Terminal window
```
# Change the permissions of a file to read/write for the owner, read-only for the group, and read-only for others (644)
chmod 644 filename.txt

# Change the permissions of a directory to read/write/execute for the owner, read/execute for the group, and read/execute for others (755)
chmod 755 directory_name

# Add execute permissions for the owner to a file
chmod +x filename.sh

# Remove write permissions for the group from a file
chmod g-w filename.txt

# Change the owner of a file to a specific user
chown username filename.txt

# Change the owner and group of a file
chown username:groupname filename.txt

# Change the owner and group of a directory and all its contents recursively
chown -R username:groupname directory_name
```
  Sample Output (no output, but the permissions change):
  Terminal window
```
# Before:
ls -l filename.txt
-rw-r--r-- 1 user user 1024 Oct 27 10:00 filename.txt

chmod 755 filename.txt

# After:
ls -l filename.txt
-rwxr-xr-x 1 user user 1024 Oct 27 10:00 filename.txt
```
- Common Options:
  - chmod mode file: Change the permissions of the file using octal notation (e.g., 644, 755).
  - chmod +permissions file: Add permissions (e.g., +x for execute).
  - chmod -permissions file: Remove permissions (e.g., -w for write).
  - chmod u+permissions file: Change permissions for the user (owner).
  - chmod g+permissions file: Change permissions for the group.
  - chmod o+permissions file: Change permissions for others.
  - chmod a+permissions file: Change permissions for all (user, group, others).
  - chmod -R directory: Recursively change the permissions of a directory and all its contents.
  - chown user file: Change the owner of the file.
  - chown group file: Change the group of the file.
  - chown user:group file: Change both the owner and group of the file.
  - chown -R directory: Recursively change the owner and group of a directory and all its contents.
- Advanced Usage:
  Terminal window
```
# Use symbolic mode to give the owner read/write/execute permissions, the group read/execute permissions, and others no permissions.
chmod u=rwx,g=rx,o= filename.txt

# Give the group the same permissions as the owner
chmod --reference=owner_file group_file

# Find all files with mode 644 and change them to 600
find . -perm 644 -exec chmod 600 {} \;

# Change the owner and group of all files in a directory to a specific user and group
chown -R username:groupname /path/to/directory
```
- Tips & Tricks:
  - Use octal notation or symbolic mode to specify permissions.
  - Use -R to recursively change permissions or ownership of directories.
  - Be careful when using chmod 777 as it makes files world-writable.
  - Ensure that the user and group you are changing ownership to exist on the system.
  - Use ls -l to verify the permissions and ownership of files after making changes.
- Troubleshooting:
  - If you get a “Permission denied” error, run chmod or chown as root or with sudo.
  - If you accidentally change the permissions of a critical system file, restore the original permissions from a backup or consult the documentation for the file.
  - If you are unable to change the owner or group of a file, ensure that you have the necessary permissions (typically, you must be root).
- Related Commands: ls, chgrp, umask
su / sudo (Privilege Escalation)
- Command Overview: su (switch user) allows you to become another user, typically the root user. sudo (superuser do) allows you to execute a single command with root privileges. These commands are crucial for performing administrative tasks that require elevated permissions.
- Basic Syntax:
  Terminal window
```
su [options] [user]
sudo [options] command
```
- Practical Examples:
  Terminal window
```
# Become the root user (su)
su

# Become a specific user (su)
su username

# Execute a command as root (sudo)
sudo apt update

# Execute a command as a specific user (sudo)
sudo -u username command
```
  Sample Output (su):
  Terminal window
```
# Normal user:
whoami
user

su
Password:
whoami
root
```
- Common Options:
  - su: Switch to the root user.
  - su -: Switch to the root user with a login shell (loads the root user’s environment).
  - su username: Switch to a specific user.
  - sudo command: Execute a command as root.
  - sudo -u username command: Execute a command as a specific user.
  - sudo -i: Start a new shell as root (similar to su -).
  - sudo -l: List the commands that the current user is allowed to run with sudo.
- Advanced Usage:
  Terminal window
```
# Grant a user specific sudo privileges by editing /etc/sudoers (using visudo)
# Example:
# username ALL=(ALL:ALL) ALL  # User can run any command as any user
# username ALL=(root) /usr/bin/apt update, /usr/bin/apt upgrade # User can only run apt update and upgrade as root

# Execute a sequence of commands as root using sudo -i
sudo -i
apt update
apt upgrade
exit
```
- Tips & Tricks:
  - Use sudo instead of su whenever possible to minimize the risk of accidental damage.
  - Configure sudoers carefully to grant users only the necessary privileges.
  - Use `