You have been using grep to find things. Now you need to change them,
extract specific fields, or summarize data across thousands of lines. That is where
sed and awk live. They have been on every Unix and Linux
system since before most of us started, and they will be there long after the tool
of the week is forgotten.
This primer covers the patterns you will actually use on the job: substitution, field extraction, conditional processing, and pipeline integration. It is not exhaustive — it is the 20% that handles 80% of the real work.
Both tools process text line by line. The question of which to reach for comes down to what you need to do with each line:
sed is a stream editor. Give it a pattern and a transformation; it applies that transformation to every matching line.
awk is a data processing language. It splits each line into fields and lets you write logic against those fields.
sed reads input one line at a time, applies your commands to each line, and writes
the result to stdout. The original file is untouched unless you use -i.
The core command structure is:
sed 'ADDRESS COMMAND' file
# ADDRESS — which lines to act on (optional — default is all lines)
# COMMAND — what to do (substitute, delete, print, etc.)This is the command you will use 90% of the time:
sed 's/PATTERN/REPLACEMENT/' file # Replace first match on each line
sed 's/PATTERN/REPLACEMENT/g' file # Replace ALL matches on each line (global)
sed 's/PATTERN/REPLACEMENT/I' file # Case-insensitive match
sed 's/PATTERN/REPLACEMENT/2' file # Replace only the 2nd occurrence# Replace 'http' with 'https' everywhere in a config file
sed 's/http:/https:/g' nginx.conf
# Replace a hostname — useful after a server rename
sed 's/old-server.example.com/new-server.example.com/g' /etc/hosts
# Preview the change — don't write yet
sed 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
# Edit in-place — THIS CHANGES THE FILE
sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
# Edit in-place WITH a backup (safer)
sed -i.bak 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
# Creates sshd_config.bak before modifying-i first and verify the output looks right. Then add -i
or -i.bak. The .bak suffix creates a backup — habit
worth keeping on production files.
# Act on a specific line number
sed '5s/foo/bar/' file # Only substitute on line 5
# Act on a range of lines
sed '10,20s/foo/bar/g' file # Lines 10 through 20 only
# Act on lines matching a pattern
sed '/^#/d' file # Delete all comment lines (starting with #)
sed '/error/s/warning/ERROR/g' file # On lines containing 'error', replace 'warning' with 'ERROR'
# From a pattern to end of file
sed '/START/,$d' file # Delete from line matching START to end of file
# Between two patterns
sed '/BEGIN/,/END/d' file # Delete everything between BEGIN and END linesDeletes comment lines and blank lines, leaving only the active configuration. Useful for reviewing what a config file actually does without wading through documentation comments.
# Delete lines matching a pattern
sed '/^$/d' file # Delete blank lines
sed '/^#/d' file # Delete comment lines
sed '/deprecated/d' file # Delete lines containing 'deprecated'
# Delete a specific line number
sed '3d' file # Delete line 3
# Append a line after a match
sed '/^Port 22/a Port 2222' sshd_config # Add Port 2222 after Port 22 line
# Insert a line before a match
sed '/^PermitRootLogin/i # Managed by Ansible' sshd_config
# Replace an entire line
sed '/^MaxAuthTries/c MaxAuthTries 3' sshd_configa (append), i (insert), and
c (change) commands are less commonly known but very useful for
config file management — especially when you need to add a setting after a
known anchor line rather than at a fixed line number.
# Run multiple sed commands with -e
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file
# Or separate commands with semicolons
sed 's/foo/bar/g; s/baz/qux/g' file
# Practical: clean up a CSV export for import
# Remove Windows line endings, strip BOM, fix quotes
cat export.csv | sed 's/\r//g; s/^\xef\xbb\xbf//' > clean.csv
# Extract lines between two markers (config block)
sed -n '/^\[database\]/,/^\[/p' app.conf | sed '$d'
# Replace with captured group — backreference with \1
sed 's/\(192\.168\.[0-9]*\)\.[0-9]*/\1.0\/24/g' firewall.rules
# Replaces specific IPs with their /24 subnet-n flag suppresses
automatic output. Combined with the p command, only lines you
explicitly print are shown. This is the sed equivalent of grep for extracting
blocks of text between markers.
| Command | What it does |
|---|---|
s/old/new/ | Substitute first occurrence per line |
s/old/new/g | Substitute all occurrences per line |
d | Delete the line |
p | Print the line (use with -n) |
a text | Append text after the line |
i text | Insert text before the line |
c text | Replace the entire line with text |
-i | Edit file in-place |
-i.bak | Edit in-place, create .bak backup first |
-n | Suppress automatic output |
-e | Multiple commands |
N,Ms/old/new/ | Substitute only on lines N through M |
/pattern/d | Delete lines matching pattern |
awk reads input line by line, automatically splits each line into fields, and executes your program against each line. The key concepts:
awk 'PATTERN { ACTION }' file
# $0 = the entire line
# $1 = first field, $2 = second field, $N = Nth field
# NF = number of fields on the current line
# NR = current line (record) number
# FS = field separator (default: whitespace)
# OFS = output field separatorawk programs have three optional sections:
awk '
BEGIN { # runs once before any input — setup, print headers }
/pattern/ { # runs on each line matching the pattern }
END { # runs once after all input — print totals, summaries }
' file# Print specific fields from command output
ps aux | awk '{print $1, $2, $11}' # user, PID, command
# From /etc/passwd — print username and shell
awk -F: '{print $1, $7}' /etc/passwd
# Print the last field (regardless of how many fields there are)
awk '{print $NF}' file
# Print all but the first field
awk '{$1=""; print $0}' file
# Print specific fields from df output
df -h | awk 'NR>1 {print $5, $6}' # Skip header line, print use% and mountpoint
# Print username and UID for accounts with UID >= 1000
awk -F: '$3 >= 1000 {print $1, $3}' /etc/passwd# Print lines where the 3rd field is greater than 100
awk '$3 > 100 {print}' file
# Print lines matching a pattern (like grep, but you can also process fields)
awk '/error/ {print NR, $0}' /var/log/syslog # Line number + line
# If/else logic
awk '{if ($5 > 80) print "HIGH:", $0; else print "OK:", $0}' file
# Multiple conditions
awk '$3 > 100 && $4 == "FAILED" {print $1, $2}' logfile
# Pattern range — print lines between two patterns
awk '/START/,/END/ {print}' file
# Skip the header line, process the rest
awk 'NR > 1 {print $1, $3}' report.txt
# Print lines where field 2 matches a regex
awk '$2 ~ /^192\.168\./ {print}' connections.log~ for regex match
($2 ~ /pattern/) and == for exact string match
($2 == "value"). Use !~ for "does not match regex"
and != for "not equal."
# Sum a column of numbers
awk '{sum += $1} END {print "Total:", sum}' numbers.txt
# Count lines matching a pattern
awk '/Failed password/ {count++} END {print count, "failed attempts"}' /var/log/auth.log
# Average of a field
awk '{sum += $3; count++} END {print "Average:", sum/count}' data.txt
# Count occurrences of each unique value in field 1
awk '{count[$1]++} END {for (k in count) print count[k], k}' file | sort -rn
# Sum of disk usage by filesystem type (from df)
df | awk 'NR>1 {used[$1] += $3} END {for (fs in used) print used[fs], fs}'
# Top IPs from access log — count and rank
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
# Equivalent in pure awk:
awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' /var/log/nginx/access.log \
| sort -rn | head -20# Print a formatted report with a header
df -h | awk '
BEGIN { printf "%-20s %5s %8s\n", "Filesystem", "Use%", "Mounted" }
NR > 1 { printf "%-20s %5s %8s\n", $1, $5, $6 }
'
# Convert colon-delimited to tab-delimited
awk -F: 'OFS="\t" {print $1, $3, $7}' /etc/passwd
# Add a field to each line
ps aux | awk 'NR>1 {print $0, "| host=" HOSTNAME}'
# Print every Nth line (e.g., every 10th)
awk 'NR % 10 == 0 {print NR, $0}' largefile.log
# Print lines between line numbers
awk 'NR>=50 && NR<=60 {print}' file
# Reverse the fields on each line
awk '{for(i=NF; i>0; i--) printf "%s%s", $i, (i>1?" ":"\n")}' fileprintf in awk works like C printf — use
%-20s for left-justified string in 20 characters, %5d
for right-justified integer in 5 characters. This is how you get clean columns
in awk output without post-processing with column.
| Pattern/Command | What it does |
|---|---|
{print $1, $3} | Print fields 1 and 3 |
{print $NF} | Print the last field |
NR>1 | Skip the first line (header) |
-F: | Set field separator to colon |
$3 > 100 | Numeric comparison on field 3 |
$1 == "root" | Exact string match on field 1 |
$0 ~ /pattern/ | Regex match on entire line |
{count[$1]++} | Count occurrences of each unique $1 |
{sum += $3} | Running sum of field 3 |
BEGIN {} | Execute before any input |
END {} | Execute after all input |
printf "%s\n", $1 | Formatted print |
OFS="\t" | Set output field separator |
The real power comes from combining these tools with each other and with the rest of the Unix toolchain. Here are the patterns that show up repeatedly in real SysAdmin work:
# --- LOG ANALYSIS ---
# Count failed SSH attempts by IP, top 10
grep "Failed password" /var/log/auth.log \
| awk '{print $11}' \
| sort | uniq -c | sort -rn | head -10
# Summarize HTTP status codes from nginx log
awk '{print $9}' /var/log/nginx/access.log \
| sort | uniq -c | sort -rn
# Find the hour with the most errors
grep "error" /var/log/syslog \
| awk '{print $3}' \
| cut -d: -f1 \
| sort | uniq -c | sort -rn
# --- CONFIG FILE MANAGEMENT ---
# Extract active (non-comment, non-blank) config lines
sed '/^[[:space:]]*#/d; /^[[:space:]]*$/d' /etc/ssh/sshd_config
# Find all unique values for a specific config directive
grep -i "^LogLevel" /etc/ssh/sshd_config | awk '{print $2}'
# Bulk rename pattern in multiple config files
sed -i 's/old-hostname/new-hostname/g' /etc/nginx/sites-enabled/*.conf
# --- FIELD PROCESSING ---
# Extract and reformat data from /etc/passwd
awk -F: '$3 >= 1000 && $3 < 65534 {printf "%-15s %5d %s\n", $1, $3, $6}' /etc/passwd
# Convert ps output into a clean report
ps aux | awk 'NR==1 || $3>0.5 {printf "%-10s %5s %5s %s\n", $1, $2, $3, $11}'# sed and awk excel at well-defined, repeatable transformations.
# Know when to reach for a different tool:
# STILL use sed/awk when:
# - The pattern is simple and the output format is stable
# - You need in-place file editing (sed -i)
# - The task is a one-liner that needs to run on any system
# - Performance matters — awk processes millions of lines fast
# Reach for jc + jq when:
# - The command output has inconsistent column widths
# - You need nested data structures or array operations
# - The script needs to be maintainable by others
ps aux | jc --ps | jq '.[] | select(.cpu_percent > 10) | {pid, command}'
# Reach for AI assist (sgpt/fabric) when:
# - You have mixed, unstructured log output to summarize
# - You need to explain what a complex awk program does
# - You want to generate a sed/awk one-liner from a description
grep "Failed\|error\|warn" /var/log/syslog | tail -100 \
| sgpt "Summarize these log entries. Group by error type."
# Have AI explain a complex awk program:
echo 'awk '"'"'NR>1{a[$1]+=$3; b[$1]++} END{for(k in a) printf "%s %.2f\n",k,a[k]/b[k]}'"'" \
| sgpt "Explain what this awk program does in plain English"-i first and verify
output before editing files in-place. This applies to human-written sed too.
| Task | Tool | Command |
|---|---|---|
| Find and replace in file | sed | sed -i.bak 's/old/new/g' file |
| Delete comment lines | sed | sed '/^#/d' file |
| Delete blank lines | sed | sed '/^$/d' file |
| Extract lines between markers | sed | sed -n '/START/,/END/p' file |
| Insert line after match | sed | sed '/pattern/a new line' file |
| Print specific fields | awk | awk '{print $1, $3}' file |
| Set field separator | awk | awk -F: '{print $1}' /etc/passwd |
| Skip header line | awk | awk 'NR>1 {print}' file |
| Sum a column | awk | awk '{sum+=$3} END{print sum}' file |
| Count by unique value | awk | awk '{c[$1]++} END{for(k in c) print c[k],k}' file |
| Filter by numeric field | awk | awk '$3 > 100 {print}' file |
| Formatted column output | awk | awk '{printf "%-15s %5s\n", $1, $2}' file |
| Top IPs from access log | awk+sort | awk '{print $1}' access.log | sort | uniq -c | sort -rn | head |
| Active config lines only | sed | sed '/^[[:space:]]*#/d; /^[[:space:]]*$/d' file |