LearnLinux › LearnLinux › CMDS

uniq Command Examples

Filter and Report Duplicate Lines

About uniq

The uniq command filters adjacent matching lines from input, effectively removing or reporting duplicates. It's a fundamental tool for data deduplication and analysis. Important: uniq only compares adjacent lines, so input typically needs to be sorted first (using sort) to catch all duplicates.

Key Features:

⚠️ Critical: uniq only detects adjacent duplicate lines. Always sort your data first: sort file.txt | uniq

Command Syntax

uniq [OPTIONS] [INPUT [OUTPUT]]

Essential Options

Option Description Example
-c Count occurrences (prefix lines) sort file.txt | uniq -c
-d Only show duplicate lines sort file.txt | uniq -d
-u Only show unique lines (non-duplicated) sort file.txt | uniq -u
-i Ignore case sort -f file.txt | uniq -i
-f N Skip first N fields sort file.txt | uniq -f 2
-s N Skip first N characters sort file.txt | uniq -s 5
-w N Compare only first N characters sort file.txt | uniq -w 10
-D Show all duplicate lines (not just one) sort file.txt | uniq -D

Detailed Examples

Example 1: Basic Duplicate Removal

cat << EOF | uniq apple apple banana cherry cherry cherry date EOF
apple banana cherry date
Basic uniq removes adjacent duplicate lines:
  • Two consecutive "apple" lines become one
  • Three consecutive "cherry" lines become one
  • Single "banana" and "date" lines unchanged
  • Only works on adjacent duplicates!
⚠️ Adjacent Only:
# This won't work as expected! echo -e "apple\\nbanana\\napple" | uniq # Output: apple, banana, apple (both apples shown!) # Correct way: echo -e "apple\\nbanana\\napple" | sort | uniq # Output: apple, banana

Example 2: Count Occurrences

cat << EOF | sort | uniq -c apple banana apple cherry banana apple cherry cherry EOF
3 apple 2 banana 3 cherry
The -c option counts occurrences and prefixes each line:
  • Shows frequency of each unique line
  • Count is right-aligned with leading spaces
  • Essential for frequency analysis
  • Combine with sort -rn to find most common items
💡 Power Pattern: Find most frequent items:
sort file.txt | uniq -c | sort -rn | head -10

Example 3: Show Only Duplicates

cat << EOF | sort | uniq -d apple banana apple cherry date apple EOF
apple
The -d option shows only lines that appear more than once:
  • Shows each duplicate line exactly once
  • Perfect for finding what data is duplicated
  • Useful for data quality checks
  • banana, cherry, date don't appear (only once each)

Example 4: Show Only Unique Lines

cat << EOF | sort | uniq -u apple banana apple cherry date EOF
banana cherry date
The -u option shows only lines that appear exactly once:
  • Opposite of -d
  • Shows truly unique items (non-duplicated)
  • apple doesn't appear (it's duplicated)
  • Useful for filtering out repeated entries

Example 5: Case-Insensitive Comparison

cat << EOF | sort -f | uniq -i Apple APPLE apple Banana banana EOF
Apple Banana
The -i option ignores case when comparing:
  • Treats Apple, APPLE, apple as identical
  • Must also sort case-insensitively (sort -f)
  • Keeps first occurrence of each case variation
  • Essential for text with mixed case

Example 6: Count with Frequency Sorting

cat << EOF | sort | uniq -c | sort -rn ERROR WARNING ERROR INFO ERROR WARNING DEBUG INFO ERROR EOF
4 ERROR 2 INFO 2 WARNING 1 DEBUG
Powerful pattern for frequency analysis:
  • Step 1: sort - Group identical lines
  • Step 2: uniq -c - Count occurrences
  • Step 3: sort -rn - Sort by count (descending)
  • Shows most frequent items first

🎯 Real-World Example:

# Find most common error messages grep ERROR app.log | sort | uniq -c | sort -rn | head -10 # Most common HTTP status codes awk '{print $9}' access.log | sort | uniq -c | sort -rn

Example 7: Skip Fields When Comparing

cat << EOF | sort -k2 | uniq -f 1 001 apple red 002 apple green 003 banana yellow 004 banana yellow EOF
002 apple green 003 banana yellow
The -f N option skips first N fields:
  • -f 1 - Skip first field, compare from second field onward
  • Useful for data with IDs or timestamps
  • 001 and 002 both have "apple" → treated as duplicate
  • 003 and 004 both have "banana yellow" → treated as duplicate

Example 8: Skip Characters When Comparing

cat << EOF | uniq -s 4 2024-01-15 Server started 2024-01-15 Database connected 2024-01-16 Server restarted EOF
2024-01-15 Server started 2024-01-16 Server restarted
The -s N option skips first N characters:
  • Ignores first 4 characters when comparing
  • Useful for timestamped data
  • Lines 1 and 2 both start with "-01-15" after skipping 4 chars
  • Line 3 is different ("-01-16")

Example 9: Compare Only First N Characters

cat << EOF | sort | uniq -w 3 apple pie apple tart banana split banana cream EOF
apple pie banana split
The -w N option compares only first N characters:
  • Only looks at first 3 characters: "app", "app", "ban", "ban"
  • Both "apple pie" and "apple tart" start with "app"
  • Both "banana" lines start with "ban"
  • Useful for prefix-based deduplication

Example 10: Comprehensive Log Analysis Pipeline

#!/bin/bash # Comprehensive log analysis using uniq LOG_FILE="application.log" echo "=== Top 10 Error Messages ===" grep "ERROR" "$LOG_FILE" | \ sed 's/^.*ERROR: //' | \ sort | uniq -c | sort -rn | head -10 echo "" echo "=== Unique Warning Types ===" grep "WARNING" "$LOG_FILE" | \ awk '{print $NF}' | \ sort | uniq echo "" echo "=== Duplicate Requests (potential replay attacks) ===" awk '{print $1, $7}' "$LOG_FILE" | \ sort | uniq -c | \ awk '$1 > 1' | \ sort -rn echo "" echo "=== Status Code Distribution ===" awk '{print $9}' "$LOG_FILE" | \ sort | uniq -c | \ awk '{printf "%3d requests: HTTP %s\\n", $1, $2}' echo "" echo "=== Lines That Appear Only Once (anomalies) ===" sort "$LOG_FILE" | uniq -u | head -20
This comprehensive script demonstrates real-world uniq usage:
  • Frequency analysis of error messages
  • Finding unique warning types
  • Detecting duplicate/repeated requests
  • HTTP status code distribution
  • Identifying anomalous single-occurrence events
  • Combines uniq with sort, awk, grep, sed

Real-World Use Cases

Use Case 1: Data Cleaning

# Remove duplicate email addresses sort emails.txt | uniq > unique_emails.txt # Remove duplicate lines from config file sort -u config.txt -o config.txt # Find duplicate entries in CSV (by first column) sort -t, -k1,1 data.csv | uniq -D -f 0

Use Case 2: Log Analysis

# Most frequent error types grep ERROR app.log | awk '{print $5}' | sort | uniq -c | sort -rn # Find repeated failed login attempts grep "Failed login" auth.log | awk '{print $8}' | sort | uniq -c | sort -rn # Unique IP addresses accessing server awk '{print $1}' access.log | sort -u # Count requests per user awk '{print $3}' access.log | sort | uniq -c

Use Case 3: Data Analysis

# Count unique users cut -d: -f1 /etc/passwd | sort | uniq | wc -l # Find duplicate file sizes (potential duplicates) find . -type f -exec ls -l {} \\; | awk '{print $5}' | sort -n | uniq -d # Word frequency analysis tr -cs A-Za-z '\\n' < document.txt | tr A-Z a-z | sort | uniq -c | sort -rn | head -20 # Most common commands in bash history history | awk '{print $2}' | sort | uniq -c | sort -rn | head -10

Combining uniq with Other Commands

Pattern 1: uniq + awk for Advanced Counting

# Show items that appear more than 5 times sort data.txt | uniq -c | awk '$1 > 5' # Calculate percentage distribution sort data.txt | uniq -c | \ awk '{sum+=$1; lines[NR]=$0} END {for(i=1;i<=NR;i++) {split(lines[i],a); printf "%3.1f%% %s\\n", (a[1]/sum)*100, a[2]}}'

Pattern 2: uniq + grep for Filtering

# Count unique ERROR lines grep ERROR log.txt | sort | uniq -c # Find duplicate IPs with failed requests grep "404" access.log | awk '{print $1}' | sort | uniq -d

Pattern 3: Comparing Files

# Find items in file1 but not in file2 sort file1.txt file2.txt file2.txt | uniq -u # Find common items between files sort file1.txt file2.txt | uniq -d

Performance Considerations

⚡ Performance Tips:
📊 Benchmark:
# Fastest for simple deduplication time sort -u huge_file.txt > output.txt # Slower but necessary for counting time sort huge_file.txt | uniq -c > output.txt

Common Pitfalls and Solutions

Pitfall 1: Forgetting to Sort

⚠️ Wrong:
# This misses non-adjacent duplicates! uniq file.txt # Example: echo -e "a\\nb\\na" | uniq # Output: a, b, a (WRONG!)
✅ Correct:
sort file.txt | uniq # Example: echo -e "a\\nb\\na" | sort | uniq # Output: a, b (correct)

Pitfall 2: Using uniq -c with Pipes

⚠️ Problem: Spaces in count output complicate parsing
# Hard to parse sort file.txt | uniq -c | awk '{print $1}' # Gets count # Better approach sort file.txt | uniq -c | sed 's/^[ \\t]*//' | cut -d' ' -f1

uniq vs. sort -u

Feature uniq sort -u
Performance Fast (single pass) Faster (combined operation)
Count Duplicates ✅ Yes (-c option) ❌ No
Show Only Duplicates ✅ Yes (-d option) ❌ No
Field/Character Skip ✅ Yes (-f, -s, -w) ❌ No
Requires Sorted Input ✅ Yes ❌ No (sorts first)
Best For Analysis, counting, filtering Simple deduplication

Related Commands

Quick Reference

Most Common uniq Commands

# Remove adjacent duplicates uniq file.txt # Remove all duplicates (sorted) sort file.txt | uniq # Count occurrences sort file.txt | uniq -c # Find most common items sort file.txt | uniq -c | sort -rn # Show only duplicates sort file.txt | uniq -d # Show only unique (non-duplicated) lines sort file.txt | uniq -u # Case-insensitive sort -f file.txt | uniq -i # Alternative to sort | uniq sort -u file.txt # Count unique lines sort file.txt | uniq | wc -l # Show all duplicate lines sort file.txt | uniq -D