System Performance Tuning

Category: Advanced Linux Administration
Type: Linux Commands
Generated on: 2025-07-10 03:13:43
For: System Administration, Development & Technical Interviews

System Performance Tuning Cheatsheet (Linux)

This cheatsheet provides a practical guide to Linux commands used for system performance tuning, targeting both system administrators and developers. It covers command overviews, syntax, practical examples, common options, advanced usage, tips & tricks, troubleshooting, and related commands.

1. Command Overview

Command	Description	Use Case
`top`	Displays real-time system resource usage, including CPU, memory, and process information.	Identifying CPU-intensive processes, memory leaks, and overall system load.
`htop`	An interactive process viewer, similar to `top` but with more features and a user-friendly interface. Requires installation (e.g., `apt install htop`, `yum install htop`).	More easily identifying resource-hogging processes and managing them (e.g., killing processes).
`vmstat`	Reports virtual memory statistics, including CPU usage, memory usage, paging, and disk I/O.	Identifying memory bottlenecks, excessive paging, and disk I/O issues.
`iostat`	Reports disk I/O statistics, including read/write speeds, transactions per second, and disk utilization.	Identifying disk I/O bottlenecks and determining if a disk is overloaded.
`free`	Displays the amount of free and used memory in the system, including RAM and swap space.	Determining if the system is running out of memory or if swap space is being heavily used.
`sar`	Collects, reports, and saves system activity information, including CPU usage, memory usage, disk I/O, network activity, and more. Requires installation and configuration (e.g., `apt install sysstat`, `yum install sysstat`).	Historical performance analysis, identifying trends, and troubleshooting performance issues over time.
`perf`	A powerful performance analysis tool for profiling applications and the kernel.	Identifying performance bottlenecks in code, analyzing CPU usage, and identifying hot spots.
`tcpdump`	A packet analyzer that captures network traffic.	Diagnosing network performance issues, analyzing network traffic patterns, and troubleshooting network connectivity problems.
`netstat`	Displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. Often replaced by `ss`.	Identifying network bottlenecks, troubleshooting network connectivity problems, and monitoring network traffic.
`ss`	`ss` (socket statistics) is used to dump socket statistics. It allows showing more TCP and state info than `netstat` and other tools. Faster and more powerful than `netstat`.	Identifying network connections, troubleshooting network connectivity problems, and monitoring network traffic. Preferred over `netstat`.
`lsof`	Lists open files. Everything in Linux is a file, so this can be used to find what processes are using specific resources (files, sockets, etc.).	Finding which process is holding a file open, identifying processes using network sockets, and troubleshooting resource contention issues.
`strace`	Traces system calls made by a process.	Debugging application errors, understanding how an application interacts with the kernel, and identifying performance bottlenecks.
`pmap`	Shows the memory map of a process.	Identifying memory leaks, understanding how memory is allocated to a process, and debugging memory-related issues.
`sysctl`	Used to configure kernel parameters at runtime. Changes can be temporary or permanent.	Optimizing kernel performance, tuning network settings, and adjusting security parameters.
`ulimit`	Controls the resource limits for a user or process.	Preventing processes from consuming excessive resources, limiting file sizes, and improving system stability.
`nice`/`renice`	`nice` sets the priority of a process when it’s started. `renice` changes the priority of a running process.	Prioritizing important processes and preventing less important processes from hogging resources.
`cpulimit`	Limits the CPU usage of a process. Requires installation.	Preventing processes from consuming excessive CPU resources and ensuring that other processes have enough CPU time.
`ionice`	Sets the I/O scheduling class and priority for a process.	Prioritizing I/O operations for important processes and preventing less important processes from slowing down the system.
`dstat`	Versatile resource monitoring tool that combines `vmstat`, `iostat`, `netstat`, and `ifstat`. Requires installation.	Providing a comprehensive overview of system performance in a single command.
`pidstat`	Reports statistics for processes managed by the Linux kernel’s task scheduler. Part of the `sysstat` package.	Identifying which processes are consuming the most CPU time, I/O, and memory.

2. Basic Syntax

top:
Terminal window
```
top [options]
```
htop:
Terminal window
```
htop [options]
```
vmstat:
Terminal window
```
vmstat [delay] [count]
```

iostat:

iostat [options] [delay] [count] [device...]

free:
Terminal window
```
free [options]
```
sar:
Terminal window
```
sar [options] [delay] [count]
```
perf:
Terminal window
```
perf [options] command
```
tcpdump:
Terminal window
```
tcpdump [options] [expression]
```
netstat:
Terminal window
```
netstat [options]
```
ss:
Terminal window
```
ss [options]
```
lsof:
Terminal window
```
lsof [options]
```
strace:
Terminal window
```
strace [options] command
```
pmap:
Terminal window
```
pmap [pid]
```
sysctl:
Terminal window
```
sysctl [options] variable=value
```
ulimit:
Terminal window
```
ulimit [options] [value]
```
nice:
Terminal window
```
nice -n <priority> command
```
renice:
Terminal window
```
renice -n <priority> -p <pid>
```
cpulimit:
Terminal window
```
cpulimit -l <percentage> -p <pid>
```
ionice:
Terminal window
```
ionice -c <class> -n <priority> command
```
dstat:
Terminal window
```
dstat [options]
```
pidstat:
Terminal window
```
pidstat [options] [delay] [count]
```

3. Practical Examples

top: Display real-time system resource usage.

top

top - 16:05:23 up 10 days, 22:12,  1 user,  load average: 0.00, 0.01, 0.05
Tasks: 185 total,   1 running, 184 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.1 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8175608 total,   385104 free,  6666204 used,  1124300 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used.  1188380 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0  166500   5584   3456 S   0.0  0.1   0:04.45 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root       0  -20       0      0      0 I   0.0  0.0   0:00.00 rcu_gp
    4 root       0  -20       0      0      0 I   0.0  0.0   0:00.00 rcu_par_gp
    6 root       0  -20       0      0      0 I   0.0  0.0   0:00.00 kworker/0:0H
    8 root      20   0       0      0      0 I   0.0  0.0   0:00.00 kdevtmpfs
    9 root       0  -20       0      0      0 I   0.0  0.0   0:00.00 netns
   10 root      20   0       0      0      0 I   0.0  0.0   0:00.00 rcu_sched
   11 root      20   0       0      0      0 I   0.0  0.0   0:00.00 rcu_bh
   12 root      20   0       0      0      0 I   0.0  0.0   0:00.00 kworker/1:0H
...

htop: Interactive process viewer.
Terminal window
```
htop
```
(Requires htop to be installed. Interface is interactive within the terminal.)

vmstat 1 5: Report virtual memory statistics every 1 second, 5 times.

vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 384852   1492 1124868    0    0     0     0   12   10  0  0 100  0  0
 0  0      0 384852   1492 1124868    0    0     0     0   12   10  0  0 100  0  0
 0  0      0 384852   1492 1124868    0    0     0     0   12   10  0  0 100  0  0
 0  0      0 384852   1492 1124868    0    0     0     0   12   10  0  0 100  0  0
 0  0      0 384852   1492 1124868    0    0     0     0   12   10  0  0 100  0  0

iostat -x 1 5: Report extended disk I/O statistics every 1 second, 5 times.

iostat -x 1 5

Linux 5.4.0-122-generic (ubuntu20)   07/03/2024   _x86_64_  (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       0.18    0.00    0.10    0.00    0.00   99.72

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await  aqusz  %util
sda              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop0            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop1            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop2            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop3            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop4            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop5            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop6            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop7            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       0.00    0.00    0.00    0.00    0.00  100.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await  aqusz  %util
sda              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop0            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop1            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop2            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop3            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop4            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop5            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop6            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
loop7            0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00   0.00
... (repeated 3 more times)

free -m: Display memory usage in megabytes.

free -m

              total        used        free      shared  buff/cache   available
Mem:           7984        6509         375         355        1099         942
Swap:          2047           0        2047

sar -u 1 5: Report CPU utilization every 1 second, 5 times.

sar -u 1 5

Linux 5.4.0-122-generic (ubuntu20)   07/03/2024   _x86_64_  (1 CPU)

16:08:46        CPU     %user     %nice   %system   %iowait    %steal     %idle
16:08:47        all      0.00      0.00      0.00      0.00      0.00    100.00
16:08:48        all      0.00      0.00      0.00      0.00      0.00    100.00
16:08:49        all      0.00      0.00      0.00      0.00      0.00    100.00
16:08:50        all      0.00      0.00      0.00      0.00      0.00    100.00
16:08:51        all      0.00      0.00      0.00      0.00      0.00    100.00
Average:        all      0.00      0.00      0.00      0.00      0.00    100.00

perf top: Profile the system and display hot spots.
Terminal window
```
sudo perf top
```
(Requires root privileges and perf to be installed. Output is interactive and depends on system activity.)
tcpdump -i eth0 -n -s 0 port 80: Capture HTTP traffic on interface eth0, display IP addresses and port numbers, and capture full packets.
Terminal window
```
sudo tcpdump -i eth0 -n -s 0 port 80
```
(Requires root privileges. Captures network traffic until interrupted with Ctrl+C.)
netstat -tulnp: Display listening TCP and UDP ports with process IDs and names. (Deprecated, use ss instead.)
Terminal window
```
netstat -tulnp
```
(May require root privileges.)

ss -tulnp: Display listening TCP and UDP ports with process IDs and names.

ss -tulnp

State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port  Process
LISTEN  0       4096             127.0.0.1:53           0.0.0.0:*      users:(("systemd-resolve",pid=780,fd=13))
LISTEN  0       128                    *:22                   *:*      users:(("sshd",pid=862,fd=3))
LISTEN  0       5              127.0.0.53:53           0.0.0.0:*      users:(("systemd-resolve",pid=780,fd=12))

lsof -i :80: List processes listening on port 80.

lsof -i :80

COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nginx   1234  nginx    6u  IPv4  12345      0t0  TCP *:http (LISTEN)
nginx   1235  nginx    6u  IPv4  12345      0t0  TCP *:http (LISTEN)

strace -p 1234: Trace system calls made by process with PID 1234.
Terminal window
```
sudo strace -p 1234
```
(Requires root privileges. Output is verbose and depends on process activity.)

pmap 1234: Show the memory map of process with PID 1234.

pmap 1234

1234:   /usr/bin/python3
0000555555554000      4K r-x-- /usr/bin/python3
0000555555555000      4K r---- /usr/bin/python3
0000555555556000     16K rw--- /usr/bin/python3
000055555555a000    132K rw---   [ anon ]
00007ffff7d98000   1400K r-x-- /lib/x86_64-linux-gnu/libc-2.31.so
00007ffff7ee6000   2048K ----- /lib/x86_64-linux-gnu/libc-2.31.so
00007ffff80e6000     40K r---- /lib/x86_64-linux-gnu/libc-2.31.so
00007ffff80f0000     12K rw--- /lib/x86_64-linux-gnu/libc-2.31.so
00007ffff80f3000     20K rw---   [ anon ]
00007ffff8110000     16K r-x-- /lib/x86_64-linux-gnu/libpthread-2.31.so
00007ffff8114000   2088K ----- /lib/x86_64-linux-gnu/libpthread-2.31.so
00007ffff8314000      4K r---- /lib/x86_64-linux-gnu/libpthread-2.31.so
00007ffff8315000      8K rw--- /lib/x86_64-linux-gnu/libpthread-2.31.so
00007ffff8317000     16K rw---   [ anon ]
00007ffff8320000    100K r-x-- /lib/x86_64-linux-gnu/libz.so.1.2.11
00007ffff8339000   2048K ----- /lib/x86_64-linux-gnu/libz.so.1.2.11
00007ffff8539000      4K r---- /lib/x86_64-linux-gnu/libz.so.1.2.11
00007ffff853a000      8K rw--- /lib/x86_64-linux-gnu/libz.so.1.2.11
00007ffff8540000     44K r-x-- /lib/x86_64-linux-gnu/libexpat.so.1.6.12
00007ffff854b000   2048K ----- /lib/x86_64-linux-gnu/libexpat.so.1.6.12
00007ffff874b000      4K r---- /lib/x86_64-linux-gnu/libexpat.so.1.6.12
00007ffff874c000      8K rw--- /lib/x86_64-linux-gnu/libexpat.so.1.6.12
00007ffff8750000    104K r-x-- /lib/x86_64-linux-gnu/liblzma.so.5.2.4
00007ffff876a000   2048K ----- /lib/x86_64-linux-gnu/liblzma.so.5.2.4
00007ffff896a000      4K r---- /lib/x86_64-linux-gnu/liblzma.so.5.2.4
00007ffff896b000      8K rw--- /lib/x86_64-linux-gnu/liblzma.so.5.2.4
00007ffff8970000     20K r-x-- /lib/x86_64-linux-gnu/libutil.so.1
00007ffff8975000   2048K ----- /lib/x86_64-linux-gnu/libutil.so.1
00007ffff8b75000      4K r---- /lib/x86_64-linux-gnu/libutil.so.1
00007ffff8b76000      8K rw--- /lib/x86_64-linux-gnu/libutil.so.1
00007ffff8b80000     20K r-x-- /lib/x86_64-linux-gnu/libuuid.so.1.3.0
00007ffff8b85000   2048K ----- /lib/x86_64-linux-gnu/libuuid.so.1.3.0
00007ffff8d85000      4K r---- /lib/x86_64-linux-gnu/libuuid.so.1.3.0
00007ffff8d86000      8K rw--- /lib/x86_64-linux-gnu/libuuid.so.1.3.0
00007ffff8d90000    128K r-x-- /lib/x86_64-linux-gnu/libcrypto.so.1.1
00007ffff8db0000   2048K ----- /lib/x86_64-linux-gnu/libcrypto.so.1.1
00007ffff8fb0000    172K r---- /lib/x86_64-linux-gnu/libcrypto.so.1.1
00007ffff8feb000     24K rw--- /lib/x86_64-linux-gnu/libcrypto.so.1.1
00007ffff8ff1000     24K rw---   [ anon ]
00007ffff9000000     72K r-x-- /lib/x86_64-linux-gnu/libssl.so.1.1
00007ffff9012000   2048K ----- /lib/x86_64-linux-gnu/libssl.so.1.1
00007ffff9212000     20K r---- /lib/x86_64-linux-gnu/libssl.so.1.1
00007ffff9217000      8K rw--- /lib/x86_64-linux-gnu/libssl.so.1.1
00007ffff9220000     44K r-x-- /lib/x86_64-linux-gnu/libffi.so.7.1.0
00007ffff922b000   2048K ----- /lib/x86_64-linux-gnu/libffi.so.7.1.0
00007ffff942b000      4K r---- /lib/x86_64-linux-gnu/libffi.so.7.1.0
00007ffff942c000      8K rw--- /lib/x86_64-linux-gnu/libffi.so.7.1.0
00007ffff9430000     88K r-x-- /lib/x86_64-linux-gnu/libtinfo.so.6.2
00007ffff9446000   2048K ----- /lib/x86_64-linux-gnu/libtinfo.so.6.2
00007ffff9646000     12K r---- /lib/x86_64-linux-gnu/libtinfo.so.6.2
00007ffff9649000      8K rw--- /lib/x86_64-linux-gnu/libtinfo.so.6.2
00007ffff9650000   1300K r-x-- /lib/x86_64-linux-gnu/libpython3.8.so.1.0
00007ffff9794000   2048K ----- /lib/x86_64-linux-gnu/libpython3.8.so.1.0
00007ffff9994000     16K r---- /lib/x86_64-linux-gnu/libpython3.8.so.1.0
00007ffff9998000     20K rw--- /lib/x86_64-linux-gnu/libpython3.8.so.1.0
00007ffff999d000     40K rw---   [ anon ]
00007ffff9a00000      8K r---- /usr/lib/locale/locale-archive
00007ffff9a10000    140K r-x-- /lib64/ld-linux-x86-64.so.2
00007ffff9a33000     12K rw---   [ anon ]
00007ffff9a3d000      4K r---- /lib64/ld-linux-x86-64.so.2
00007ffff9a3e000      4K rw--- /lib64/