📊 Performance Analysis

Performance Series: Part 1 — Performance Analysis  |  Part 2 — sar & sysstat  |  Part 3 — Load Average Explained

Know What Your System Is Doing

Performance analysis is the discipline of understanding what your system is actually doing versus what you think it should be doing. When a server feels slow, when a process is eating resources, when users complain — you need a systematic approach to isolate the bottleneck. Is it CPU? Memory? Disk I/O? Network? Or something else entirely?

This page covers the essential tools for real-time performance analysis — top, htop, vmstat, iostat, free, and ss — with a systematic workflow for isolating problems.

The Four Bottleneck Areas

AreaPrimary ToolsKey Metrics
CPUtop, htop, mpstat, vmstat%us, %sy, %wa, load average
Memoryfree, vmstat, topavailable, swap used, page faults
Disk I/Oiostat, iotop, df, du%util, await, r/s, w/s
Networkss, iftop, nethogs, ipbandwidth, connections, errors

Examples

1
top — The First Tool You Reach For
# Basic top
top

# Key interactive commands inside top:
# 1       -- show per-CPU stats
# M       -- sort by memory usage
# P       -- sort by CPU usage (default)
# T       -- sort by cumulative time
# k       -- kill a process (enter PID)
# r       -- renice a process
# f       -- field management (add/remove columns)
# W       -- save current config
# q       -- quit

# Run top non-interactively (1 iteration, useful in scripts)
top -bn1

# Watch a specific process
top -p 12345
top -p $(pgrep nginx | tr '\n' ',')
Annotated top header:
top - 10:22:15 up 5 days, 2:14, 2 users, load average: 0.52, 0.48, 0.45 Tasks: 185 total, 1 running, 184 sleeping, 0 stopped, 0 zombie %Cpu(s): 3.2 us, 1.1 sy, 0.0 ni, 95.2 id, 0.3 wa, 0.0 hi, 0.2 si MiB Mem : 15887.3 total, 8234.1 free, 4521.6 used, 3131.6 buff/cache MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 10987.4 avail Mem Fields: us=user sy=system ni=nice id=idle wa=iowait hi=hw-irq si=sw-irq Key: high wa% = disk bottleneck | low id% = CPU bottleneck | swap used = memory pressure
Read the top header first. Before looking at individual processes, read the summary lines. High wa (iowait) means disk is the bottleneck. Low id (idle) means CPU is saturated. Swap in use means memory pressure. These three numbers tell you where to look next.
2
htop — top With a Better Interface
# Install htop
sudo dnf install -y htop    # RHEL
sudo apt install -y htop    # Debian

# Run htop
htop

# Key htop features over top:
# Visual CPU/memory bars at the top
# Color-coded process tree
# Mouse support
# F3 -- search for a process by name
# F4 -- filter processes
# F5 -- tree view (show parent/child)
# F6 -- sort by column
# F9 -- kill selected process
# Space -- tag a process (for bulk operations)

# htop showing only processes from one user
htop -u craig

# htop for a specific PID and its children
htop -p 12345
💡 F5 tree view is invaluable. htop's process tree shows parent-child relationships clearly — you can see which web worker spawned which child, which shell spawned which script. When a mystery process is consuming resources, the tree tells you where it came from.
3
vmstat — System-Wide Activity at a Glance
# vmstat with 2-second interval, 10 samples
vmstat 2 10

# vmstat with timestamps
vmstat -t 2 10

# Show disk statistics
vmstat -d 2 5

# Show slab memory info
vmstat -m
vmstat 2 5 output:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 8234112 102400 3207168 0 0 2 18 245 512 3 1 95 1 0 0 0 0 8233856 102400 3207168 0 0 0 24 198 445 2 1 97 0 0 2 0 0 8231424 102400 3207168 0 0 0 156 312 678 8 2 89 1 0 0 1 0 8230912 102400 3207168 0 0 0 892 287 601 3 2 88 7 0 0 0 0 8232640 102400 3207168 0 0 0 18 201 452 2 1 97 0 0
ColumnMeaningWatch for
rProcesses waiting for CPUConsistently > number of CPUs = CPU bottleneck
bProcesses in uninterruptible sleepNon-zero = waiting on I/O
swpdSwap in use (KB)Non-zero and growing = memory pressure
si/soSwap in/out per secondNon-zero = actively swapping = serious problem
bi/boBlocks read/written per secondHigh bo with high wa = disk write bottleneck
waCPU time waiting for I/OConsistently >10% = disk bottleneck
csContext switches per secondVery high = too many threads/processes
4
free — Memory Usage
# Show memory in human-readable format
free -h

# Show in MB
free -m

# Watch memory every 2 seconds
watch -n2 free -h

# More detailed memory breakdown
cat /proc/meminfo
cat /proc/meminfo | grep -E "MemTotal|MemFree|MemAvailable|Buffers|Cached|SwapTotal|SwapFree"
free -h output annotated:
total used free shared buff/cache available Mem: 15Gi 4.4Gi 8.0Gi 156Mi 3.1Gi 10Gi Swap: 4.0Gi 0B 4.0Gi Key column: "available" not "free" available = free + buff/cache that can be reclaimed This is the real answer to "how much memory do I have left?" Swap: 0B used = healthy. Any swap usage = memory pressure.
⚠️ "free" does not mean available. Linux uses free memory for disk cache — a server with very little "free" memory is not necessarily in trouble. The available column is the correct metric — it shows how much memory can be given to a new application after reclaiming cache. If available is low and swap is in use, you have a real memory problem.
5
iostat — Disk I/O Performance
# Install sysstat (provides iostat, sar, mpstat)
sudo dnf install -y sysstat    # RHEL
sudo apt install -y sysstat    # Debian

# Basic iostat -- 2 second intervals, 5 samples
iostat -x 2 5

# Show specific device
iostat -x sda 2 5

# Human readable sizes
iostat -xh 2 5
iostat -x output (key columns):
Device r/s w/s rkB/s wkB/s await r_await w_await %util sda 12.5 45.2 512.0 1840.0 8.3 4.2 9.8 62.4 sdb 0.1 0.2 4.0 8.0 1.2 1.0 1.4 0.1
ColumnMeaningWatch for
r/s, w/sReads/writes per secondContext-dependent
rkB/s, wkB/sKB read/written per secondCompare to disk spec
awaitAverage I/O wait time (ms)>20ms = slow disk or overloaded
%utilPercentage of time disk was busy>80% consistently = disk bottleneck
💡 %util and await together tell the story. High %util (80%+) with high await (20ms+) = disk is saturated and requests are queuing. High %util with low await = disk is busy but keeping up. Low %util with high await = disk may be failing or misconfigured.
6
iotop — Which Process Is Hammering the Disk
# Install iotop
sudo dnf install -y iotop
sudo apt install -y iotop

# Run iotop (requires root)
sudo iotop

# Show only processes doing I/O (cleaner view)
sudo iotop -o

# Non-interactive -- good for scripts
sudo iotop -b -n 3 -o    # 3 iterations, only active I/O

# Find top I/O consumers right now
sudo iotop -b -n 1 -o | head -20
iotop -o output:
Total DISK READ: 45.2 M/s | Total DISK WRITE: 128.5 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 1234 be/4 mysql 0.00 B/s 85.2 M/s 0.00 % 42.15 % mysqld 5678 be/4 www-data 0.00 B/s 38.1 M/s 0.00 % 18.33 % php-fpm 9012 be/4 root 45.2 M/s 0.00 B/s 0.00 % 12.44 % rsync
iotop is your disk equivalent of top. When iostat shows the disk is saturated but you don't know which process is responsible, iotop names the culprit immediately. The combination of iostat (is the disk a bottleneck?) and iotop (which process is causing it?) solves most disk performance problems.
7
Network Performance — ss and iftop
# ss -- socket statistics (modern netstat replacement)
ss -tulpn                   # listening ports
ss -s                       # summary statistics
ss -tn state established    # established TCP connections

# Count connections by state
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn

# Count connections per remote IP
ss -tan state established | awk 'NR>1 {print $5}' | \
    cut -d: -f1 | sort | uniq -c | sort -rn | head -10

# iftop -- live bandwidth by connection
sudo dnf install -y iftop
sudo iftop             # all interfaces
sudo iftop -i ens3     # specific interface

# nethogs -- bandwidth per process
sudo dnf install -y nethogs
sudo nethogs
sudo nethogs ens3
💡 ss -s for a quick network health check. The summary shows total sockets, TCP states, and UDP counts at a glance. A very high number of TIME_WAIT connections often indicates a busy web server that could benefit from keep-alive tuning.
8
Performance Triage Script — First 60 Seconds

When you get the call "the server is slow" — run this:

#!/bin/bash
# perf-triage.sh -- first 60 seconds on a slow server

echo "========================================"
echo " Performance Triage: $(hostname)"
echo " $(date)"
echo "========================================"

echo ""
echo "--- UPTIME AND LOAD ---"
uptime

echo ""
echo "--- CPU SUMMARY (5 seconds) ---"
vmstat 1 5

echo ""
echo "--- MEMORY ---"
free -h
echo ""
cat /proc/meminfo | grep -E "MemAvailable|SwapFree|SwapTotal"

echo ""
echo "--- TOP CPU CONSUMERS ---"
ps aux --sort=-%cpu | head -10

echo ""
echo "--- TOP MEMORY CONSUMERS ---"
ps aux --sort=-%mem | head -10

echo ""
echo "--- DISK I/O (5 seconds) ---"
iostat -x 1 5 | tail -20

echo ""
echo "--- DISK SPACE ---"
df -h | grep -v tmpfs

echo ""
echo "--- NETWORK CONNECTIONS ---"
ss -s
echo ""
ss -tan state established | wc -l | xargs echo "Established TCP connections:"

echo ""
echo "--- RECENT ERRORS ---"
journalctl -b -p err --since "1 hour ago" --no-pager | tail -15

echo "========================================"

Quick Reference

ToolBest ForKey Command
topOverall CPU and process overviewtop (then press 1 for per-CPU)
htopInteractive process managementhtop -u user or F5 for tree
vmstatCPU, memory, swap, I/O summaryvmstat 2 10
freeMemory usagefree -h (watch the available column)
iostatDisk I/O per deviceiostat -x 2 5
iotopWhich process is doing I/Osudo iotop -o
ssNetwork connections and portsss -tulpn or ss -s
iftopNetwork bandwidth by connectionsudo iftop -i ens3
nethogsNetwork bandwidth by processsudo nethogs
ps auxAll processes with CPU/memps aux --sort=-%cpu

← Back to Performance Index ↑ Back to EXPANDED