Introduction to the uniq Command
The uniq command in Linux is used to filter out adjacent duplicate lines from input (or standard input) and write to output (or standard output).
Tip: For uniq to work effectively, it generally requires sorted input as it only compares adjacent lines.
Example 1: Basic Usage
Remove adjacent duplicate lines from a file.
$ cat fruits.txt
apple
apple
banana
banana
banana
cherry
$ uniq fruits.txt
apple
banana
cherry
Example 2: Count Occurrences
Count the number of times each line occurs.
$ uniq -c fruits.txt
2 apple
3 banana
1 cherry
Example 3: Print Only Duplicate Lines
Display only lines that are repeated.
$ uniq -d fruits.txt
apple
banana
Example 4: Print All Duplicate Lines
Display all duplicate lines (once for each group).
$ uniq -D fruits.txt
apple
apple
banana
banana
banana
Example 5: Print Unique Lines
Display only lines that are not repeated.
$ uniq -u fruits.txt
cherry
Example 6: Case-Insensitive Comparison
Ignore case when comparing lines.
$ cat case.txt
Apple
APPLE
apple
BANANA
$ uniq -i case.txt
Apple
BANANA
Example 7: Skip Fields
Skip the first N fields before comparing.
$ cat data.txt
1 John Doe
2 John Doe
3 Jane Smith
4 Jane Smith
$ uniq -f 1 data.txt
1 John Doe
3 Jane Smith
Example 8: Skip Characters
Skip the first N characters before comparing.
$ cat codes.txt
A1234
A1235
B1234
B1235
$ uniq -s 1 codes.txt
A1234
B1234
Example 9: Check Characters
Compare only the first N characters.
$ uniq -w 1 codes.txt
A1234
B1234
Example 10: Using with Sort
Sort first to remove all duplicates.
$ cat mixed.txt
apple
banana
apple
cherry
banana
$ sort mixed.txt | uniq
apple
banana
cherry
Example 11: Count with Sorted Input
Count occurrences after sorting.
$ sort mixed.txt | uniq -c
2 apple
2 banana
1 cherry
Example 12: Show Repeated Lines with Count
Count and show only duplicate lines.
$ sort mixed.txt | uniq -cd
2 apple
2 banana
Example 13: Show Unique Lines with Count
Count and show only unique lines.
$ sort mixed.txt | uniq -cu
1 cherry
Example 14: Using with Process Substitution
Compare two files to find common lines.
$ cat file1.txt
apple
banana
orange
$ cat file2.txt
banana
cherry
orange
$ sort file1.txt file2.txt | uniq -d
banana
orange
Example 15: Find Lines in One File But Not Another
Show lines unique to the first file.
$ sort file1.txt file2.txt | uniq -u
apple
cherry
Example 16: Ignore Leading Blanks
Skip leading blanks when comparing.
$ cat spaced.txt
apple
apple
banana
banana
$ uniq spaced.txt
apple
banana
banana
Note: By default, uniq considers lines with different leading spaces as different.
Example 17: Using with AWK for Advanced Processing
Extract a field and then find unique values.
$ cat users.txt
john:doe:35
jane:smith:28
john:doe:35
bob:jones:42
$ awk -F: '{print $1}' users.txt | sort | uniq
bob
jane
john
Example 18: Combine with Grep
Find and count error messages in a log file.
$ grep "ERROR" application.log | sort | uniq -c | sort -nr
15 Connection timeout
10 Database not responding
5 Invalid user credentials
Example 19: Check for Duplicate Entries
Find duplicate entries in a CSV file.
$ cut -d, -f1 data.csv | sort | uniq -cd
3 john@example.com
2 jane@example.com
Example 20: Using with Head
Show the top 5 most frequent lines.
$ sort large_file.txt | uniq -c | sort -nr | head -5
142 error 404
120 error 500
85 successful login
72 user logout
55 password change
Example 21: Using with Tail
Show the least frequent lines.
$ sort large_file.txt | uniq -c | sort -n | tail -5
1 rare error 901
1 rare error 902
1 rare error 903
1 rare error 904
1 rare error 905
Example 22: Ignore Specific Fields
Skip the first two fields when comparing.
$ cat log.txt
2023-01-01 10:15:30 ERROR Service down
2023-01-01 10:16:45 ERROR Service down
2023-01-01 10:17:20 INFO Service restored
$ uniq -f 2 log.txt
2023-01-01 10:15:30 ERROR Service down
2023-01-01 10:17:20 INFO Service restored
Example 23: Check Only First N Characters
Compare only the first 5 characters.
$ cat codes2.txt
ERROR001 Service failure
ERROR002 Authentication issue
ERROR001 Service failure
ERROR003 Network timeout
$ uniq -w 7 codes2.txt
ERROR001 Service failure
ERROR002 Authentication issue
ERROR001 Service failure
ERROR003 Network timeout
Note: The first 7 characters include "ERROR00" which is different for each error code.
Example 24: Combine Skip Fields and Characters
Skip first field and first 2 characters of the second field.
$ cat complex.txt
1 AA1234 Error1
2 AB1234 Error1
3 AA1234 Error2
4 AC1234 Error1
$ uniq -f 1 -s 2 complex.txt
1 AA1234 Error1
3 AA1234 Error2
4 AC1234 Error1
Example 25: Using uniq with Pipelines
Complex example showing uniq in a data processing pipeline.
$ cat access.log | awk '{print $7}' | grep "\.html" | sort | uniq -c | sort -nr | head -10
1245 /index.html
893 /about.html
765 /contact.html
543 /products.html
432 /services.html
321 /blog.html
210 /pricing.html
198 /faq.html
165 /terms.html
123 /privacy.html
This pipeline shows the top 10 most accessed HTML pages on a website.
Important Notes About uniq
1. uniq only removes adjacent duplicate lines, so input usually needs to be sorted first.
2. By default, comparison is case-sensitive. Use -i for case-insensitive comparison.
3. uniq can count occurrences, show duplicates, or show unique lines.
4. You can skip fields or characters before comparing lines.
5. uniq is often used in pipelines with sort, grep, awk, and other commands.