Advanced Shell Scripting and Automation

Category: Advanced Linux Administration
Type: Linux Commands
Generated on: 2025-07-10 03:12:55
For: System Administration, Development & Technical Interviews

Advanced Shell Scripting and Automation Cheatsheet (Linux)

This cheatsheet covers advanced shell scripting and automation techniques for Linux system administration and development.

1. Command Overview

Command	Description	Use Cases
`find`	Locate files based on various criteria (name, size, modification time, permissions, etc.). Essential for finding specific files or directories for processing.	Locating log files, finding files older than a certain date, searching for files with specific permissions.
`xargs`	Build and execute command lines from standard input. Useful for processing large lists of files or data that would exceed command line limits.	Processing the output of `find`, running commands on a large number of files.
`awk`	A powerful text processing tool for pattern matching, field extraction, and data manipulation. Ideal for parsing log files, generating reports, and transforming data.	Extracting specific data from log files, calculating statistics from data files, reformatting data.
`sed`	A stream editor for performing text transformations (substitution, deletion, insertion) on files or standard input. Useful for automating text editing tasks.	Replacing text in files, deleting lines from files, inserting text into files.
`grep`	Search for patterns in files or standard input. Fundamental for finding specific strings or regular expressions within text.	Finding specific errors in log files, searching for configuration options in configuration files.
`rsync`	A fast and versatile file synchronization tool. Used for backups, mirroring, and transferring files between systems.	Backing up files, synchronizing data between servers, deploying code to multiple servers.
`screen/tmux`	Terminal multiplexers that allow you to run multiple terminal sessions within a single window, detach and reattach sessions, and share sessions with others. Essential for long-running processes and remote work.	Running long-running processes without being disconnected, managing multiple terminal windows within a single session, collaborating with others on a remote server.
`systemd`	System and service manager. Used for managing system services, starting/stopping/restarting services, and configuring service dependencies.	Managing web servers, databases, and other system services, ensuring services automatically start on boot.
`journalctl`	Query and display logs collected by systemd. Essential for troubleshooting system issues and monitoring service status.	Analyzing system logs, identifying error messages, monitoring service performance.
`ssh`	Secure Shell. Used for secure remote access to systems, executing commands remotely, and transferring files securely.	Connecting to remote servers, running commands on remote servers, transferring files securely between systems.
`scp`	Secure Copy. Used for securely copying files between systems using SSH.	Transferring files to/from remote servers, backing up files to a remote server.
`ssh-keygen`	Generate, manage, and convert authentication keys for SSH. Key-based authentication is more secure than password-based authentication.	Creating SSH keys for passwordless login, managing SSH keys.
`set -euxo pipefail`	Set options to make shell scripts more robust and easier to debug.	Ensuring that scripts exit immediately on errors, displaying commands before execution, and handling pipeline errors.

2. Basic Syntax

find: find [path] [options] [expression]
xargs: [command] | xargs [options] [command]
awk: awk '[pattern] { action }' [file]
sed: sed '[address]command' [file]
grep: grep [options] pattern [file]
rsync: rsync [options] source destination
systemctl: systemctl [command] [unit]
journalctl: journalctl [options]
ssh: ssh [options] user@host
scp: scp [options] source destination
ssh-keygen: ssh-keygen [options]
set: set [options]

3. Practical Examples

find

# Find all files named "error.log" in the /var/log directory
find /var/log -name "error.log"

# Find all files larger than 10MB in the current directory
find . -size +10M

# Find all files modified in the last 7 days
find . -mtime -7

# Find all directories with permissions 777
find . -type d -perm 777

# Find all files ending in .txt and execute a command on them
find . -name "*.txt" -exec chmod 644 {} \;

xargs

# Find all .txt files and delete them (USE WITH CAUTION!)
find . -name "*.txt" -print0 | xargs -0 rm -f

# Find all .log files and gzip them
find /var/log -name "*.log" -print0 | xargs -0 gzip

# Limit the number of parallel processes to 4
find . -name "*.jpg" -print0 | xargs -0 -P 4 convert -resize 50% {} resized/{}

awk

# Print the first field of each line in a file
awk '{print $1}' data.txt

# Print lines where the second field is greater than 100
awk '$2 > 100 {print}' data.txt

# Calculate the sum of the third field in a file
awk '{sum += $3} END {print "Sum:", sum}' data.txt

# Extract IP addresses from a log file
awk '/Failed password/{print $11}' /var/log/auth.log

sed

# Replace "old" with "new" in a file
sed 's/old/new/g' file.txt

# Delete the first line of a file
sed '1d' file.txt

# Delete lines containing "error"
sed '/error/d' file.txt

# Insert a line after the first line
sed '1a This is a new line' file.txt

# Replace every occurrence of 'foo' with 'bar' in-place
sed -i 's/foo/bar/g' file.txt

grep

# Search for "error" in a file
grep "error" file.txt

# Search for "error" ignoring case
grep -i "error" file.txt

# Search for lines that do NOT contain "error"
grep -v "error" file.txt

# Search recursively for "error" in all files in the current directory
grep -r "error" .

# Search for "error" and show 5 lines before and after the match
grep -C 5 "error" file.txt

rsync

# Backup a directory to another location
rsync -avz /path/to/source /path/to/destination

# Synchronize a directory with a remote server
rsync -avz user@host:/path/to/source /path/to/destination

# Delete extraneous files from the destination directory (mirroring)
rsync -avz --delete /path/to/source /path/to/destination

# Exclude certain files or directories
rsync -avz --exclude 'node_modules' /path/to/source /path/to/destination

# Resume an interrupted transfer
rsync -avz --partial --progress /path/to/source /path/to/destination

screen/tmux

# Start a new screen session
screen

# Start a new named screen session
screen -S mysession

# Detach from a screen session (Ctrl+a, d)

# Reattach to a screen session
screen -r

# Reattach to a named screen session
screen -r mysession

# List running screen sessions
screen -ls

# Kill a screen session
screen -X -S mysession quit

# tmux new session
tmux new -s my_session

# tmux attach to a session
tmux attach -t my_session

# tmux detach from a session (Ctrl+b, d)

# tmux list sessions
tmux ls

# tmux kill session
tmux kill-session -t my_session

systemd

# Start a service
sudo systemctl start apache2

# Stop a service
sudo systemctl stop apache2

# Restart a service
sudo systemctl restart apache2

# Check the status of a service
sudo systemctl status apache2

# Enable a service to start on boot
sudo systemctl enable apache2

# Disable a service from starting on boot
sudo systemctl disable apache2

# Reload systemd configuration
sudo systemctl daemon-reload

journalctl

# Show all system logs
journalctl

# Show logs for a specific service
journalctl -u apache2

# Show logs for the current boot
journalctl -b

# Show logs from the last hour
journalctl --since "1 hour ago"

# Follow logs in real-time
journalctl -f

# Show logs with a specific priority (error, warning, etc.)
journalctl -p err

ssh

# Connect to a remote server
ssh user@host

# Connect to a remote server using a specific port
ssh -p 2222 user@host

# Execute a command on a remote server
ssh user@host "ls -l /home/user"

# Forward a local port to a remote port
ssh -L 8080:localhost:80 user@host

# Forward a remote port to a local port
ssh -R 8080:localhost:80 user@host

# Use a specific identity file (private key)
ssh -i ~/.ssh/my_private_key user@host

scp

# Copy a file to a remote server
scp file.txt user@host:/path/to/destination

# Copy a file from a remote server
scp user@host:/path/to/file.txt /path/to/destination

# Copy a directory to a remote server recursively
scp -r /path/to/directory user@host:/path/to/destination

# Use a specific port
scp -P 2222 file.txt user@host:/path/to/destination

# Use a specific identity file (private key)
scp -i ~/.ssh/my_private_key file.txt user@host:/path/to/destination

ssh-keygen

# Generate a new SSH key pair
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

# Change the passphrase of an existing SSH key
ssh-keygen -p -f ~/.ssh/id_rsa

# Convert a key to a different format (e.g., for PuTTY)
ssh-keygen -p -m PEM -f ~/.ssh/id_rsa

# Display the public key
ssh-keygen -y -f ~/.ssh/id_rsa

set -euxo pipefail

#!/bin/bash
set -euxo pipefail

# e: Exit immediately if a command exits with a non-zero status.
# u: Treat unset variables as an error.
# x: Print commands before execution (debugging).
# o pipefail: If a command in a pipeline fails, the entire pipeline fails.

echo "Starting script"
ls -l /nonexistent_directory # This will cause the script to exit
echo "This will not be printed"

4. Common Options

find:
- -name: Search by filename.
- -type: Search by file type (e.g., f for file, d for directory).
- -size: Search by file size (e.g., +10M for greater than 10MB).
- -mtime: Search by modification time (e.g., -7 for modified in the last 7 days).
- -exec: Execute a command on found files.
- -print0: Print filenames separated by null characters (for use with xargs -0).
xargs:
- -0: Expect input separated by null characters.
- -n: Maximum number of arguments per command.
- -P: Number of parallel processes to run.
- -I: Replace occurrences of a placeholder string with the input.
awk:
- -F: Specify the field separator.
- -v: Assign a value to a variable.
sed:
- s/old/new/g: Substitute “old” with “new” globally.
- -i: Edit the file in place (USE WITH CAUTION!).
- -n: Suppress default output.
- -e: Execute multiple commands.
grep:
- -i: Ignore case.
- -v: Invert the match (show lines that do NOT match).
- -r: Recursive search.
- -C: Show context (lines before and after the match).
- -n: Show line numbers.
- -w: Match whole words only.
rsync:
- -a: Archive mode (preserves permissions, ownership, etc.).
- -v: Verbose output.
- -z: Compress data during transfer.
- --delete: Delete extraneous files from the destination.
- --exclude: Exclude files or directories.
- --partial: Keep partially transferred files.
- --progress: Show progress during transfer.
systemctl:
- start: Start a service.
- stop: Stop a service.
- restart: Restart a service.
- status: Check the status of a service.
- enable: Enable a service to start on boot.
- disable: Disable a service from starting on boot.
- daemon-reload: Reload systemd configuration.
journalctl:
- -u: Specify a service unit.
- -b: Show logs for the current boot.
- --since: Show logs since a specific time.
- -f: Follow logs in real-time.
- -p: Specify a priority level.
ssh:
- -p: Specify a port.
- -i: Specify an identity file (private key).
- -L: Forward a local port to a remote port.
- -R: Forward a remote port to a local port.
- -v: Verbose output (for debugging).
scp:
- -P: Specify a port.
- -r: Recursive copy for directories.
- -i: Specify an identity file (private key).
ssh-keygen:
- -t: Specify the key type (e.g., rsa, ed25519).
- -b: Specify the key size (e.g., 4096).
- -C: Add a comment to the key (usually your email address).
- -p: Change the passphrase.
- -f: Specify the key file.
- -y: Display the public key.
- -m: Specify the key format.

5. Advanced Usage

Combining find and xargs with awk and sed:

# Find all .log files, extract the date and time from each line, and print the unique dates.
find /var/log -name "*.log" -print0 | xargs -0 awk '{print $1, $2}' | sort -u

# Find all configuration files, back them up, and replace a specific string.
find /etc -name "*.conf" -print0 | xargs -0 -I {} bash -c 'cp {} {}.bak && sed -i "s/old_value/new_value/g" {}'

Using awk for complex data processing:

# Calculate the average of a column in a CSV file.
awk -F, '{sum += $3; count++} END {if (count > 0) print "Average:", sum / count}' data.csv

# Convert a delimited file to a different format.
awk 'BEGIN {FS=","; OFS=":"} {print $1, $2, $3}' input.txt > output.txt

Using sed for advanced text manipulation:

# Add a comment to the beginning of each line in a file.
sed 's/^/# /' file.txt

# Remove blank lines from a file.
sed '/^$/d' file.txt

# Surround a specific word with quotes.
sed 's/\<word\>/\"&\"/g' file.txt

Automating tasks with cron and systemd timers:

# Cron example (run a script every day at 3 AM):
# Edit crontab: crontab -e
0 3 * * * /path/to/your/script.sh

# Systemd timer example (create a timer and a service unit):
# /etc/systemd/system/my-script.service:
[Unit]
Description=My Script Service

[Service]
ExecStart=/path/to/your/script.sh

# /etc/systemd/system/my-script.timer:
[Unit]
Description=Run My Script Daily

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

# Enable and start the timer:
sudo systemctl enable my-script.timer
sudo systemctl start my-script.timer

Using rsync for incremental backups:

# Create an incremental backup with hard links to unchanged files:
rsync -avz --link-dest=/path/to/previous/backup /path/to/source /path/to/new/backup

Leveraging shell functions for code reusability:

# Define a function to check if a service is running.
is_service_running() {
  if systemctl is-active --quiet "$1"; then
    echo "Service $1 is running"
    return 0
  else
    echo "Service $1 is not running"
    return 1
  fi
}

# Use the function.
is_service_running apache2

6. Tips & Tricks

Use tab completion: Press Tab to auto-complete commands, filenames, and options.
Use history search: Press Ctrl+R to search through your command history.
Use aliases: Create aliases for frequently used commands in your .bashrc or .zshrc file.
Terminal window
```
alias la='ls -la'
alias grep='grep --color=auto'
```
Use shell variables: Store values in variables for reusability.
Terminal window
```
LOG_DIR=/var/log
grep "error" $LOG_DIR/syslog
```
Use set -x for debugging: Add set -x to your script to print each command before it’s executed. Remove it for production.
Use comments: Add comments to your scripts to explain what each section does.
Test your scripts: Test your scripts in a non-production environment before deploying them to production.
Use version control: Store your scripts in a version control system (e.g., Git) to track changes and collaborate with others.
Use sudo -E: When using sudo to run commands, use -E to preserve the user’s environment variables.
Always quote variables: Use double quotes around variables to prevent word splitting and globbing. Example: echo "$MY_VARIABLE"
Use printf for precise output formatting: printf "%-20s %10d\n" "Filename" 12345
Use process substitution: Compare the output of two commands using diff <(command1) <(command2)

7. Troubleshooting

“Command not found”: The command is not in your PATH environment variable. Either specify the full path to the command or add the directory containing the command to your PATH.
“Permission denied”: You don’t have the necessary permissions to execute the command or access the file. Use chmod to change file permissions or sudo to run the command with elevated privileges.
xargs exceeding argument limits: Use -n option to limit the number of arguments passed to each command execution.
sed -i corrupting files: Always back up your files before using sed -i. Consider using temporary files and mv for safer in-place editing.
rsync not syncing correctly: Double-check your source and destination paths and ensure that the correct options are used. Pay attention to trailing slashes, which can affect behavior.
Script exiting prematurely: Check for errors in your script and use set -e to ensure that the script exits immediately on errors. Review output from set -x if enabled.
Incorrect awk output: Check the field separator (-F) and ensure that the correct fields are being accessed.

8. Related Commands

cut: Extract specific columns from a file.
sort: Sort lines in a file.
uniq: Remove duplicate lines from a file.
wc: Count lines, words, and characters in a file.
diff: Compare two files.
patch: Apply changes to a file.
tar: Archive and compress files.
gzip/gunzip: Compress and decompress files.
bzip2/bunzip2: Compress and decompress files.
xz/unxz: Compress and decompress files.
curl/wget: Download files from the internet.
jq: Process JSON data.
yq: Process YAML data.
watch: Execute a command periodically and display the output.

This cheatsheet provides a comprehensive overview of advanced shell scripting and automation techniques. Remember to practice and experiment with these commands to gain a deeper understanding and become proficient in using them for your daily tasks. Always back up your data before running potentially destructive commands.