LearnLinux › LearnLinux › CMDS

awk Command Reference Guide

Text processing and pattern-action language for data manipulation

Overview

Purpose

The awk command is a powerful text processing language that scans files line-by-line, splits each line into fields, and executes actions based on pattern matching. It's ideal for extracting, transforming, and summarizing data from structured text files.

Basic Syntax

awk [options] 'pattern { action }' file

Key Strengths

awk excels at field manipulation, pattern matching, arithmetic operations, and generating formatted reports. It's particularly powerful because it combines pattern matching with a complete programming language, making it far more capable than simple text filtering tools.

Basic Concepts and Structure

Field Variables

$0 = entire line, $1 = first field, $2 = second field, etc. Fields are separated by whitespace by default.

Built-in Variables

  • NF = Number of Fields in current line
  • NR = Number of Records (lines) processed so far
  • FS = Field Separator (default is space/tab)
  • RS = Record Separator (default is newline)
  • OFS = Output Field Separator
  • ORS = Output Record Separator

Pattern-Action Structure

BEGIN block executes before processing any files, main block processes each line matching the pattern, END block executes after all input is processed.

Common Options

Option Description
-F Specify field separator (e.g., -F: for colon)
-v var=value Set a variable before processing
-f script.awk Read awk program from a file instead of command line

10 Detailed Examples

1
Print Specific Fields from Structured Data
Command
awk '{print $1, $3}' /etc/passwd
root 0
daemon 1
bin 2
sys 3
sync 4

What This Does:

Extracts the first field (username) and third field (UID) from /etc/passwd. The fields are separated by colons, but awk splits on whitespace by default here. This demonstrates basic field extraction.

Practical Application:

Use this to quickly extract specific columns from any structured text file. Combine with grep to filter specific lines before processing.

2
Change Field Separator (Colon-Delimited Data)
Command
awk -F: '{print $1, $3}' /etc/passwd
root 0
daemon 1
bin 2
sync 5
games 12

What This Does:

The -F: option sets the field separator to colon, properly parsing /etc/passwd fields. Now $1 is the username and $3 is the UID (accounting for all colon-separated fields).

Field Separator Rules:

  • Single character: -F:
  • Regular expression: -F'[ ,]' for space or comma
  • Tab character: -F'\t'
3
Filter Lines with Pattern Matching
Command
awk '$3 > 1000 {print $1, $3}' /etc/passwd
user1 1001
user2 1002
user3 1003

What This Does:

Only prints lines where the third field ($3) is greater than 1000. This filters users with UID over 1000 (typically non-system users). The pattern precedes the action block.

Pattern Types:

  • $1 == "root" - Exact string match
  • /pattern/ - Regular expression match
  • NR > 5 - Line number greater than 5
  • $2 ~ /[0-9]+/ - Field contains digits
4
Calculate Sum of Numeric Fields
Input File (sales.txt)
Product A 100
Product B 250
Product C 175
Product D 300
Command
awk '{total += $3} END {print "Total Sales:", total}' sales.txt
Total Sales: 825

What This Does:

Accumulates the third field (sales amounts) across all lines and prints the sum in the END block. The END block executes after all input has been processed.

Breaking Down the Command:

  • {total += $3} - Main pattern block: add third field to total
  • END {...} - Executes once after all lines processed
  • print - Output the final total
5
Count Lines Matching Pattern
Command
awk '/error/ {count++} END {print "Errors found:", count}' logfile.txt
Errors found: 23

What This Does:

Counts lines containing "error" (case-sensitive). The pattern /error/ matches lines with this string, and the counter increments for each match.

Pattern Matching Options:

  • Case-insensitive: tolower($0) ~ /error/
  • Multiple patterns: /error|warning|failed/
  • Negation: !/debug/ - lines NOT matching
6
Format Output with BEGIN Block
Command
awk 'BEGIN {print "Username\tUID\tShell"} -F: {print $1"\t"$3"\t"$7}' /etc/passwd | head -5
Username UID Shell
root 0 /bin/bash
daemon 1 /usr/sbin/nologin
bin 2 /usr/sbin/nologin
sys 3 /usr/sbin/nologin

What This Does:

The BEGIN block prints a header line before processing any input. This creates a formatted table with headers. The main block then prints data with tab separators.

Block Structure:

  • BEGIN {} - Executes once before any input
  • {} - Main pattern block for each line
  • END {} - Executes once after all input
7
Conditional Logic with Multiple Patterns
Input File (status.txt)
server1 online
server2 offline
server3 online
server4 error
Command
awk '{if ($2 == "online") online++; else if ($2 == "offline") offline++; else errors++} END {print "Online:", online, "Offline:", offline, "Errors:", errors}' status.txt
Online: 2 Offline: 1 Errors: 1

What This Does:

Uses if-else logic to categorize each line into different counters based on the second field's value. Demonstrates conditional logic within awk.

Advanced Conditional:

You can use all standard operators: ==, !=, <, >, <=, >=, &&, ||, and ternary operator: $2=="online" ? "UP" : "DOWN"

8
String Functions and Transformations
Command
awk '{print toupper($1), length($2), substr($3, 1, 3)}' data.txt
ITEM1 5 123
ITEM2 6 456
ITEM3 4 789

What This Does:

Demonstrates built-in string functions: toupper() converts to uppercase, length() returns string length, and substr() extracts a substring.

Useful String Functions:

  • length(s) - Length of string s
  • substr(s, i, n) - Extract n chars from string s starting at position i
  • index(s, t) - Position of string t in string s
  • split(s, a, fs) - Split string s into array a using separator fs
  • gsub(r, s) - Global substitution (replace all)
  • sub(r, s) - Replace first occurrence
9
Array Processing and Aggregation
Input File (access.log snippet)
192.168.1.1 GET
192.168.1.2 POST
192.168.1.1 GET
192.168.1.3 DELETE
192.168.1.1 POST
Command
awk '{count[$1]++} END {for (ip in count) print ip, count[ip]}' access.log
192.168.1.1 3
192.168.1.2 1
192.168.1.3 1

What This Does:

Uses an associative array to count occurrences of each unique IP address. The array index becomes the IP, and the value is the count. The END block iterates through all array elements.

Array Features:

  • Associative arrays (index can be any string)
  • for (key in array) - Iterate through all elements
  • delete array[key] - Remove element
  • key in array - Check if key exists
10
Complex Report Generation
Input File (employees.txt)
John Sales 50000
Jane IT 75000
Bob Sales 55000
Alice IT 80000
Charlie Sales 52000
Command
awk 'BEGIN {printf "%-15s %-10s %-10s\n", "Department", "Count", "Avg Salary"} {dept[$2]++; salary[$2]+=$3} END {for (d in dept) printf "%-15s %-10d $%-9.2f\n", d, dept[d], salary[d]/dept[d]}' employees.txt
Department Count Avg Salary
Sales 3 $52333.33
IT 2 $77500.00

What This Does:

A sophisticated example that generates a formatted report. Uses two arrays to track department counts and total salaries, then calculates and displays averages per department.

Advanced Features Demonstrated:

  • printf for formatted output (%-15s = left-aligned 15 chars)
  • Multiple associative arrays tracking different data
  • Complex calculations (salary/dept)
  • Formatted numeric output with decimal places (%-9.2f)

Real-World Application:

This pattern is used in production systems for log analysis, billing reports, capacity planning, and system metrics aggregation.

Advanced Techniques and Patterns

Regular Expressions in awk

Use ~ for match and !~ for non-match. Example: $1 ~ /^[A-Z]/ matches first field starting with uppercase letter.

Field Modification and Reconstruction

Modify fields and reconstruct lines: {$2="new"; print} rebuilds the line with changed $2. Use OFS to control output field separator.

Multi-File Processing

Process multiple files with FILENAME variable: awk '{print FILENAME":"$0}' file1 file2 shows which file each line came from.

Performance Considerations

awk is very efficient for most text processing tasks. For multi-gigabyte files, consider using gawk with -mf large file support or splitting input with split command first.

When to Use awk vs Other Tools

Task awk sed grep
Extract specific columns ✓ Best choice Limited Not suitable
Simple pattern filtering Good Good ✓ Best choice
Line substitution Good ✓ Best choice Not suitable
Calculate aggregates ✓ Best choice Not suitable Not suitable
Complex logic ✓ Best choice Limited Not suitable

Troubleshooting Common Issues

Problem: Field separator not working

Ensure you use -F (uppercase) not -f. The -f option specifies a file. For complex separators, use regex: -F'[ ,:]+' for multiple separators.

Problem: Variables not initialized

awk initializes variables to 0 or empty string automatically. However, for clarity in complex scripts, initialize in BEGIN: BEGIN {count=0; total=0}

Problem: Unexpected number of fields

Use print NF to debug field counts. Unexpected whitespace or tabs may affect field splitting. Consider -F'[[:space:]]+' for robust whitespace handling.

Practical System Administration Examples

Parse Apache access logs

awk '{print $1}' access.log | sort | uniq -c | sort -rn - Count requests per IP (use with other tools)

Generate password summary

awk -F: '{print $1, $5}' /etc/passwd - List all usernames with their GECOS info

Calculate disk space from df output

df -h | awk 'NR>1 {total+=$2} END {print "Total capacity:", total}' - Sum disk capacities

Monitor log file growth

awk '{if (/ERROR/) errors++; if (/WARNING/) warns++} END {print "Errors:", errors, "Warnings:", warns}' app.log - Count error types