/commands/awk/awk-remove-duplicate-lines
awk '!seen[$0]++': remove duplicate lines without sorting
- tool
- awk
- versions
- gawk 5.x, mawk 1.3, BSD awk
- os
- linux, macos, windows
awk '!seen[$0]++' file prints each line only the first time it appears and keeps the original order. seen is an array indexed by the whole line ($0); seen[$0]++ returns the old count (0 the first time) and then increments it, and ! turns 0 into true, so a line is printed only on its first occurrence. Unlike sort -u or sort | uniq, the input does not need to be sorted and the order does not change.
Example
printf '%s\n' beta alpha beta gamma alpha beta > words.txt
awk '!seen[$0]++' words.txt
echo ===
sort -u words.txtOutput
beta alpha gamma === alpha beta gamma
awk keeps beta alpha gamma in the order they first appeared; sort -u returns the same set, alphabetically.
Deduplicate by one column
Use a field instead of $0 as the key to keep the first line for each value of that field:
cat > logins.csv <<'EOF'
ada,2026-10-01,laptop
bob,2026-10-01,phone
ada,2026-10-02,phone
cyd,2026-10-03,laptop
bob,2026-10-04,laptop
EOF
awk -F, '!seen[$1]++' logins.csv
echo ===
awk -F, '!seen[$1 FS $3]++' logins.csvOutput
ada,2026-10-01,laptop bob,2026-10-01,phone cyd,2026-10-03,laptop === ada,2026-10-01,laptop bob,2026-10-01,phone ada,2026-10-02,phone cyd,2026-10-03,laptop bob,2026-10-04,laptop
The first command keeps the first login per user. The second uses user and device together as the key (FS puts a comma between them so that ab + c and a + bc stay distinct), which keeps one line per user and device.
Variants
| Command | Result |
|---|---|
| awk '!seen[$0]++' f | Unique lines, first occurrence kept, order preserved |
| awk '!seen[$2]++' f | First line for each value of field 2 |
| awk 'seen[$0]++ == 1' f | Lines that occur more than once, printed once each |
| awk '{ c[$0]++ } END { for (k in c) if (c[k] == 1) print k }' f | Lines that occur exactly once (order not kept) |
| awk '{ k = tolower($0) } !seen[k]++' f | Unique lines ignoring case |
| awk '{ sub(/[ \t]+$/, "") } !seen[$0]++' f | Ignore trailing spaces and tabs when comparing |
| awk 'NF && !seen[$0]++' f | Unique non-empty lines |
Pitfalls
- Lines that look the same are not always the same. Trailing spaces and Windows
\rendings make two lines different. Check withcat -A f | headand normalise withsub()ortr -d '\r'. - Memory grows with the number of distinct keys. Each unique line is kept in memory until awk exits.
- Several files share one array:
awk '!seen[$0]++' a.txt b.txtremoves lines ofb.txtthat already appeared ina.txt. Reset the array per file withFNR == 1 { delete seen }if that is not what you want (delete seenon a whole array works in gawk, mawk and BSD awk).
Confirm it worked
Compare the counts: wc -l < words.txt against awk '!seen[$0]++' words.txt | wc -l, and sort -u words.txt | wc -l should give the same number as the awk result.
FAQ
How do I write the result back to the same file?
awk cannot edit in place portably. Write to a temporary file and move it, awk '!seen[$0]++' f > f.tmp && mv f.tmp f. gawk 4.1 and later also offer gawk -i inplace '!seen[$0]++' f.
Is this faster than sort -u?
It reads the file once and never sorts, so it is usually faster on large files and it keeps the original order. The cost is memory: every distinct line is stored in the array. For files with hundreds of millions of distinct lines, sort -u, which can spill to disk, is safer.
How do I keep the last occurrence instead of the first?
Reverse the file, deduplicate, and reverse again, tac f | awk '!seen[$0]++' | tac. On macOS, where tac is missing, use tail -r instead.