Performance analysis is the discipline of understanding what your system is
actually doing versus what you think it should be doing. When a server feels
slow, when a process is eating resources, when users complain — you need
a systematic approach to isolate the bottleneck. Is it CPU? Memory? Disk I/O?
Network? Or something else entirely?
This page covers the essential tools for real-time
performance analysis — top, htop,
vmstat, iostat, free, and
ss — with a systematic workflow for isolating problems.
The Four Bottleneck Areas
Area
Primary Tools
Key Metrics
CPU
top, htop, mpstat, vmstat
%us, %sy, %wa, load average
Memory
free, vmstat, top
available, swap used, page faults
Disk I/O
iostat, iotop, df, du
%util, await, r/s, w/s
Network
ss, iftop, nethogs, ip
bandwidth, connections, errors
Examples
1
top — The First Tool You Reach For
# Basic top
top
# Key interactive commands inside top:# 1 -- show per-CPU stats# M -- sort by memory usage# P -- sort by CPU usage (default)# T -- sort by cumulative time# k -- kill a process (enter PID)# r -- renice a process# f -- field management (add/remove columns)# W -- save current config# q -- quit# Run top non-interactively (1 iteration, useful in scripts)
top -bn1
# Watch a specific process
top -p 12345
top -p $(pgrep nginx | tr '\n' ',')
Annotated top header:
top - 10:22:15 up 5 days, 2:14, 2 users, load average: 0.52, 0.48, 0.45
Tasks: 185 total, 1 running, 184 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.2 us, 1.1 sy, 0.0 ni, 95.2 id, 0.3 wa, 0.0 hi, 0.2 si
MiB Mem : 15887.3 total, 8234.1 free, 4521.6 used, 3131.6 buff/cache
MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 10987.4 avail Mem
Fields: us=user sy=system ni=nice id=idle wa=iowait hi=hw-irq si=sw-irq
Key: high wa% = disk bottleneck | low id% = CPU bottleneck | swap used = memory pressure
Read the top header first. Before looking at individual
processes, read the summary lines. High wa (iowait) means
disk is the bottleneck. Low id (idle) means CPU is saturated.
Swap in use means memory pressure. These three numbers tell you where
to look next.
2
htop — top With a Better Interface
# Install htop
sudo dnf install -y htop # RHEL
sudo apt install -y htop # Debian# Run htop
htop
# Key htop features over top:# Visual CPU/memory bars at the top# Color-coded process tree# Mouse support# F3 -- search for a process by name# F4 -- filter processes# F5 -- tree view (show parent/child)# F6 -- sort by column# F9 -- kill selected process# Space -- tag a process (for bulk operations)# htop showing only processes from one user
htop -u craig
# htop for a specific PID and its children
htop -p 12345
💡 F5 tree view is invaluable.
htop's process tree shows parent-child relationships clearly —
you can see which web worker spawned which child, which shell spawned
which script. When a mystery process is consuming resources, the tree
tells you where it came from.
3
vmstat — System-Wide Activity at a Glance
# vmstat with 2-second interval, 10 samples
vmstat 2 10
# vmstat with timestamps
vmstat -t 2 10
# Show disk statistics
vmstat -d 2 5
# Show slab memory info
vmstat -m
vmstat 2 5 output:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 8234112 102400 3207168 0 0 2 18 245 512 3 1 95 1 0
0 0 0 8233856 102400 3207168 0 0 0 24 198 445 2 1 97 0 0
2 0 0 8231424 102400 3207168 0 0 0 156 312 678 8 2 89 1 0
0 1 0 8230912 102400 3207168 0 0 0 892 287 601 3 2 88 7 0
0 0 0 8232640 102400 3207168 0 0 0 18 201 452 2 1 97 0 0
Column
Meaning
Watch for
r
Processes waiting for CPU
Consistently > number of CPUs = CPU bottleneck
b
Processes in uninterruptible sleep
Non-zero = waiting on I/O
swpd
Swap in use (KB)
Non-zero and growing = memory pressure
si/so
Swap in/out per second
Non-zero = actively swapping = serious problem
bi/bo
Blocks read/written per second
High bo with high wa = disk write bottleneck
wa
CPU time waiting for I/O
Consistently >10% = disk bottleneck
cs
Context switches per second
Very high = too many threads/processes
4
free — Memory Usage
# Show memory in human-readable format
free -h
# Show in MB
free -m
# Watch memory every 2 seconds
watch -n2 free -h
# More detailed memory breakdown
cat /proc/meminfo
cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree"
free -h output annotated:
total used free shared buff/cache available
Mem: 15Gi 4.4Gi 8.0Gi 156Mi 3.1Gi 10Gi
Swap: 4.0Gi 0B 4.0Gi
Key column: "available" not "free"
available = free + buff/cache that can be reclaimed
This is the real answer to "how much memory do I have left?"
Swap: 0B used = healthy. Any swap usage = memory pressure.
⚠️ "free" does not mean available.
Linux uses free memory for disk cache — a server with very little
"free" memory is not necessarily in trouble. The available
column is the correct metric — it shows how much memory can be
given to a new application after reclaiming cache. If available
is low and swap is in use, you have a real memory problem.
💡 %util and await together tell the story.
High %util (80%+) with high await (20ms+) = disk is saturated and
requests are queuing. High %util with low await = disk is busy but
keeping up. Low %util with high await = disk may be failing or
misconfigured.
6
iotop — Which Process Is Hammering the Disk
# Install iotop
sudo dnf install -y iotop
sudo apt install -y iotop
# Run iotop (requires root)
sudo iotop
# Show only processes doing I/O (cleaner view)
sudo iotop -o
# Non-interactive -- good for scripts
sudo iotop -b -n 3 -o # 3 iterations, only active I/O# Find top I/O consumers right now
sudo iotop -b -n 1 -o | head -20
iotop -o output:
Total DISK READ: 45.2 M/s | Total DISK WRITE: 128.5 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
1234 be/4 mysql 0.00 B/s 85.2 M/s 0.00 % 42.15 % mysqld
5678 be/4 www-data 0.00 B/s 38.1 M/s 0.00 % 18.33 % php-fpm
9012 be/4 root 45.2 M/s 0.00 B/s 0.00 % 12.44 % rsync
iotop is your disk equivalent of top.
When iostat shows the disk is saturated but you don't know which
process is responsible, iotop names the culprit immediately.
The combination of iostat (is the disk a bottleneck?) and iotop
(which process is causing it?) solves most disk performance problems.
7
Network Performance — ss and iftop
# ss -- socket statistics (modern netstat replacement)
ss -tulpn # listening ports
ss -s # summary statistics
ss -tn state established # established TCP connections# Count connections by state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
# Count connections per remote IP
ss -tan state established | awk 'NR>1 {print $5}' | \
cut -d: -f1 | sort | uniq -c | sort -rn | head -10
# iftop -- live bandwidth by connection
sudo dnf install -y iftop
sudo iftop # all interfaces
sudo iftop -i ens3 # specific interface# nethogs -- bandwidth per process
sudo dnf install -y nethogs
sudo nethogs
sudo nethogs ens3
💡 ss -s for a quick network health check.
The summary shows total sockets, TCP states, and UDP counts at a glance.
A very high number of TIME_WAIT connections often indicates a busy
web server that could benefit from keep-alive tuning.
8
Performance Triage Script — First 60 Seconds
When you get the call "the server is slow" — run this:
#!/bin/bash
# perf-triage.sh -- first 60 seconds on a slow server
echo "========================================"
echo " Performance Triage: $(hostname)"
echo " $(date)"
echo "========================================"
echo ""
echo "--- UPTIME AND LOAD ---"
uptime
echo ""
echo "--- CPU SUMMARY (5 seconds) ---"
vmstat 1 5
echo ""
echo "--- MEMORY ---"
free -h
echo ""
cat /proc/meminfo | grep -E "MemAvailable|SwapFree|SwapTotal"
echo ""
echo "--- TOP CPU CONSUMERS ---"
ps aux --sort=-%cpu | head -10
echo ""
echo "--- TOP MEMORY CONSUMERS ---"
ps aux --sort=-%mem | head -10
echo ""
echo "--- DISK I/O (5 seconds) ---"
iostat -x 1 5 | tail -20
echo ""
echo "--- DISK SPACE ---"
df -h | grep -v tmpfs
echo ""
echo "--- NETWORK CONNECTIONS ---"
ss -s
echo ""
ss -tan state established | wc -l | xargs echo "Established TCP connections:"
echo ""
echo "--- RECENT ERRORS ---"
journalctl -b -p err --since "1 hour ago" --no-pager | tail -15
echo "========================================"