/commands/coreutils/count-duplicate-lines-sort-uniq
sort | uniq -c | sort -rn: count duplicate lines
- tool
- GNU coreutils
- versions
- GNU coreutils 8.x, GNU coreutils 9.x
- os
- linux, macos, windows
sort file | uniq -c | sort -rn counts how many times each line occurs and lists the most frequent first. uniq only collapses adjacent duplicate lines, which is why the input must be sorted first; -c prefixes each line with its count, and the second sort -rn orders by that count, numerically and in reverse.
Example
printf '%s\n' 404 200 200 500 200 404 200 > status.txt
sort status.txt | uniq -c | sort -rnOutput
4 200
2 404
1 500Without sorting first
printf '%s\n' a b a a > x.txt
uniq -c x.txtOutput
1 a
1 b
2 aThe two a runs are counted separately because they are not adjacent.
Common variations
| Command | Result |
|---|---|
| sort f | uniq -c | sort -rn | head | Top 10 most frequent lines |
| sort f | uniq -d | Only the lines that occur more than once (one copy each) |
| sort f | uniq -u | Only the lines that occur exactly once |
| sort -u f | Unique lines, same as sort f | uniq |
| sort f | uniq -ic | Count, ignoring case |
| awk '{print $1}' access.log | sort | uniq -c | sort -rn | Count the first field (for example client IPs in a log) |
Count a column of a log
cat > access.log <<'EOF'
10.0.0.1 GET /
10.0.0.2 GET /about
10.0.0.1 GET /login
10.0.0.3 GET /
10.0.0.1 POST /login
EOF
awk '{print $1}' access.log | sort | uniq -c | sort -rnOutput
3 10.0.0.1
1 10.0.0.3
1 10.0.0.2In this output 10.0.0.3 and 10.0.0.2 both have a count of 1, and sort -rn reverses the whole line comparison, so equal counts end up in reverse alphabetical order. That is why ties are not listed in a predictable way.
Stable order for equal counts
Sort by the count field only, descending, then by the line itself ascending. -k1,1nr limits the first key to field 1, and -k2 breaks ties with the rest of the line:
printf '%s\n' b c a b c a d > x.txt
sort x.txt | uniq -c | sort -k1,1nr -k2Output
2 a
2 b
2 c
1 dCount in one pass with awk
awk can count without sorting the input first; sort the much shorter result afterwards:
printf '%s\n' 404 200 200 500 200 404 200 > status.txt
awk '{ c[$0]++ } END { for (k in c) print c[k], k }' status.txt | sort -k1,1nr -k2Output
4 200 2 404 1 500
uniq options that change what counts as a duplicate
| Option | Effect |
|---|---|
| -c | Prefix each output line with the number of occurrences |
| -d / -u | Print only duplicated lines / only lines that are never repeated |
| -i | Ignore case when comparing (sort with -f too) |
| -f N | Skip the first N fields before comparing (fields are separated by blanks) |
| -s N | Skip the first N characters before comparing |
| -w N | Compare at most N characters (GNU) |
uniq compares whole lines, so trailing spaces or Windows line endings (\r) make identical-looking lines different. If counts are split unexpectedly, check with cat -A file | head and strip \r with tr -d '\r' first.
FAQ
How do I remove the leading spaces from uniq -c output?
uniq -c right-aligns the count in a padded field. Pipe the result through awk '{ $1 = $1; print }' or sed 's/^ *//' to strip the padding, or print the count yourself with awk.
How do I count duplicates while ignoring case?
Sort case-insensitively as well, sort -f file | uniq -ic. With a case-sensitive sort, Apple and apple may not end up next to each other and uniq -i counts them separately.
Is there a way to count lines without sorting?
Yes, an awk associative array counts every line in one pass, awk '{ c[$0]++ } END { for (k in c) print c[k], k }' file. It keeps every distinct line in memory, which is fine unless the input has millions of distinct lines.