50 Complex AWK Examples

• 1. Extracting Specific Columns with a Condition:

  awk '$3 > 100 {print $1, $5}' file.txt

  Prints the first and fifth columns for lines where the third column's value is greater than 100.

• 2. Counting Occurrences of a Value:

  awk '{count[$1]++} END {for (word in count) print word, count[word]}' file.txt

  Counts and prints the frequency of each unique value in the first column.

• 3. Calculating a Sum of a Column:

  awk '{sum += $4} END {print "Total:", sum}' file.txt

  Calculates and prints the sum of all values in the fourth column.

• 4. Filtering and Formatting Data:

  awk '/^Error/ {printf "Line %d: %s\n", NR, $0}' logfile.log

  Finds all lines starting with "Error" and prints the line number followed by the entire line.

• 5. Using Multiple Field Separators:

  awk -F '[:/]' '{print $1, $3}' /etc/passwd

  Uses both colon and forward slash as field separators to print the username and user ID from the password file.

• 6. Processing Data Across Multiple Files:

  awk 'FNR==1 && NR>1 {print "\n--- New File ---"} {print}' file1.txt file2.txt

  Adds a separator line between the contents of file1.txt and file2.2.txt.

• 7. Replacing a String and Saving to a New File:

  awk '{gsub(/old_string/, "new_string"); print}' file.txt > new_file.txt

  Globally substitutes "old_string" with "new_string" on every line and saves the result to new_file.txt.

• 8. Finding the Maximum Value in a Column:

  awk '{if ($2 > max) max=$2} END {print "Max value:", max}' file.txt

  Finds and prints the maximum value found in the second column.

• 9. Conditional Calculations:

  awk '{if ($3 == "complete") {sum += $4}} END {print "Sum of completed tasks:", sum}' tasks.txt

  Sums the fourth column only for lines where the third column's value is "complete."

• 10. Generating Reports with Headers and Footers:

  awk 'BEGIN {print "--- Report ---"} {print $1, $2} END {print "--- End of Report ---"}' data.txt

  Prints a header before and a footer after all lines are processed.

• 11. Finding Averages of Multiple Columns:

  awk '{sum1+=$2; sum2+=$3; count++} END {print "Avg Col 2:", sum1/count; print "Avg Col 3:", sum2/count}' file.txt

  Calculates and prints the average of both the second and third columns.

• 12. Processing Paired Lines (Next Record):

  awk 'NR%2==1 {line1=$0; next} {print line1, $0}' file.txt

  Processes two lines at a time, printing the odd-numbered line followed by the even-numbered line on the same line. next skips the rest of the script for the current line.

• 13. Splitting a Line into an Array:

  awk '{split($0, arr, ","); for (i in arr) print arr[i]}' csv_file.csv

  Splits each line of a comma-separated file into an array and prints each element on a new line.

• 14. Dynamically Setting Field Separators:

  awk '{FS = ($1 == "user") ? "," : " "; print $2}' mixed_file.txt

  Changes the field separator to a comma if the first field is "user" and to a space otherwise.

• 15. Using a while loop within a pattern block:

  awk '{i=1; while(i<=NF) {if ($i ~ /test/) print "Found on line", NR; i++}}' file.txt

  Iterates through each field of every line to find a specific pattern.

• 16. Creating a Tab-Separated File:

  awk 'BEGIN {OFS="\t"} {print $1, $3, $5}' file.txt > tab_file.tsv

  Sets the output field separator to a tab, creating a tab-separated output file.

• 17. Printing the Last Column:

  awk '{print $NF}' file.txt

  NF is a built-in variable that holds the number of fields in the current record, so $NF is always the last field.

• 18. Deleting Records Based on a Condition:

  awk '!/pattern/ {print}' file.txt

  Prints any line that does not contain the specified pattern, effectively deleting lines that do.

• 19. Reading a Multiline Record:

  awk 'BEGIN {RS=""} /pattern/ {print}' multiline.txt

  Sets the record separator (RS) to an empty string, treating paragraphs as single records, and then prints paragraphs that contain the pattern.

• 20. Calculating Cumulative Sum:

  awk '{sum+=$1; print $0, "->", sum}' file.txt

  Prints each line and its cumulative sum of the first column.

• 21. Finding Unique Values and Counting:

  awk '!seen[$1]++' file.txt

  A concise way to print only the unique values of the first column. The seen[$1]++ part creates an associative array to track seen values.

• 22. Using External Variables:

  awk -v var="string" '$1 == var {print}' file.txt

  Passes an external shell variable "string" into the awk script using the -v flag.

• 23. Formatting with sprintf:

  awk '{printf "Name: %-10s ID: %05d\n", $1, $2}' file.txt

  Formats the output to a fixed width, aligning the name to the left and padding the ID with leading zeros.

• 24. Simulating a grep -v and head combination:

  awk 'NR <= 10 && !/pattern/ {print}' file.txt

  Prints the first 10 lines of the file that do not contain the specified pattern.

• 25. Using awk as a pre-processor for other commands:

  awk '{print $2}' file.txt | sort | uniq -c

  Pipes the output of awk (the second column) to sort and then to uniq -c to count the occurrences of unique values.

• 26. Conditional Substitution:

  awk '{if ($3 == "fail") sub(/fail/, "FAILED"); print}' report.log

  Substitutes "fail" with "FAILED" only if the third field is exactly "fail".

• 27. Generating Sequence Numbers:

  awk '{print NR, $0}' file.txt

  Prepends each line with its record number.

• 28. Printing the First and Last Line:

  awk 'NR==1 || NR==10 {print}' file.txt

  Prints only the first and tenth lines of the file.

• 29. Reversing the Order of Fields:

  awk '{for (i=NF; i>=1; i--) printf "%s ", $i; printf "\n"}' file.txt

  Prints all fields of a line in reverse order.

• 30. Summing a Column with a Header:

  awk 'NR>1 {sum+=$3} END {print "Total sales:", sum}' sales.csv

  Skips the first line (header) and then sums the third column.

• 31. Creating a "Pivot Table" or Summary Report:

  awk '{sum[$1]+=$2} END {for (item in sum) print item, sum[item]}' sales.txt

  Aggregates sales by product, summing the second column for each unique value in the first.

• 32. Finding Averages by Category:

  awk '{count[$1]++; sum[$1]+=$2} END {for (item in sum) print item, sum[item]/count[item]}' scores.txt

  Calculates the average score for each category listed in the first column.

• 33. Checking for Duplicates:

  awk '{if (seen[$1]++) print "Duplicate found:", $1}' file.txt

  Prints a message for every line where the value in the first column has been seen before.

• 34. Filtering by String Length:

  awk 'length($1) > 5 {print}' file.txt

  Prints lines where the first field has a length greater than 5 characters.

• 35. Using awk to generate html tables:

  awk 'BEGIN {print "<table>"} {print "<tr><td>" $1 "</td><td>" $2 "</td></tr>"} END {print "</table>"}' data.txt

  Wraps each line of a file in HTML table row and cell tags.

• 36. Complex Field Extraction:

  awk -F '/' 'NF > 3 {print $NF}' file.txt

  For lines with more than three fields (using / as a separator), this prints the last field.

• 37. Simulating tail:

  awk '{if (NR > (FNR-10)) print}' file.txt

  Prints the last 10 lines of the file.

• 38. Grouping records with a blank line separator:

  awk 'BEGIN {RS="\n\n"; ORS="\n\n"} /pattern/ {print}' file.txt

  Processes the file in paragraph mode (separated by blank lines) and prints paragraphs containing the pattern.

• 39. Replacing all occurrences in a line:

  awk '{while (sub(/pattern/, "replacement"))}' file.txt

  Repeatedly substitutes "pattern" with "replacement" on each line until no more occurrences are found. This is useful for overlapping patterns.

• 40. Filtering based on a list from another file:

  awk 'NR==FNR {a[$1]; next} ($1 in a)' list.txt main_file.txt

  A two-file awk script. It first reads list.txt to create an associative array of names, then reads main_file.txt and prints lines where the first field is in the array.

• 41. Using awk with sort for numerical sorting:

  awk '{print $1, $2, $3 | "sort -k3,3n"}' file.txt

  Pipes the output of awk to the sort command, sorting the lines numerically based on the third field.

• 42. Counting Words and Characters:

  awk '{words+=NF; chars+=length($0)} END {print "Words:", words, "Chars:", chars}' file.txt

  Counts the total number of words and characters in a file.

• 43. Parsing JSON-like data:

  awk -F '[:,]' '/"name"/ {print $2} /"age"/ {print $2}' data.json

  Uses colon and comma as separators to extract values for "name" and "age" from a simple JSON structure.

• 44. Finding the Longest Line:

  awk '{if (length > max) {max = length; line = $0}} END {print "Longest line (" max "):", line}' file.txt

  Finds and prints the longest line in the file.

• 45. Implementing a simple state machine:

  awk '/^--- start ---/ {state=1; next} /^--- end ---/ {state=0} state==1 {print}' file.txt

  Prints lines that fall between the start and end markers.

• 46. Removing a specific column:

  awk '{$2=""; sub(/  /, " ")}1' file.txt

  Sets the second field to empty, then uses sub to remove the resulting double space, effectively deleting the column.

• 47. Calculating Standard Deviation:

  awk '{sum+=$1; sumsq+=$1^2} END {mean=sum/NR; stdev=sqrt(sumsq/NR - mean^2); print "Mean:", mean, "Stdev:", stdev}' numbers.txt

  Calculates the mean and standard deviation of a column of numbers.

• 48. Printing the First Two Fields and the Rest of the Line:

  awk '{first_two = $1" "$2; $1=$2=""; print first_two, $0}' file.txt

  Combines the first two fields, then prints them followed by the rest of the line, which has been reset.

• 49. Dynamically changing the output separator:

  awk 'NR==1 {OFS=","} NR==2 {OFS=" "} {print}' file.txt

  Sets the output field separator to a comma for the first line and a space for the second.

• 50. Finding the Top N Values:

  awk '{print $1, $2 | "sort -k2,2nr | head -n 5"}' data.txt

  Pipes the first two fields to sort to sort numerically in reverse (nr) by the second field, and then to head to get the top 5 results.