Linux · text processing

uniq

remove or count repeats

uniq removes duplicate lines — but only when the duplicates are next to each other. That is why it is always fed from sort. With -c it counts how often each line occurred.

sort | uniq -c | sort -nr turns any list into a ranking of what occurs most.

Removes

Adjacent repeats

not repeats further apart

So first

sort

to bring duplicates together

Count

-c

occurrences, before each line

Only repeats

-d

-u for lines that occur once

At the terminal

Read the session first, then the notes: each numbered line in the output is answered by the note with the same number.

user@netflow — bashcoreutils

$ uniq visitors.txt

192.0.2.7

198.51.100.3

1192.0.2.7

2$ sort visitors.txt | uniq -c

3 192.0.2.7

1 198.51.100.3

3$ sort visitors.txt | uniq -c | sort -nr | head -1

3 192.0.2.7

Illustrative session · names, sizes and dates are examples, and every address is from a documentation or private range

Reading the output

What the numbered lines are telling you

The repeat was not removed

192.0.2.7 appears twice in the output because, in the file, another line sat between its occurrences.

Sorted first, then counted

sort puts identical lines together; uniq -c then collapses each run and prints how many it replaced.

Ranked by the count

A second sort, numeric and reversed, orders by the number in front. head keeps the top one: the most frequent visitor.

Switches

A switch changes how the command behaves. These are the ones used day to day; man uniq lists the rest. Single-letter switches can usually be joined, so -l -a may be written -la.

-c: Count: prefix each line with the number of times it occurred.
-d: Print only lines that are repeated.
-u: Print only lines that are not repeated.
-i: Ignore case when comparing.

Variations worth knowing

The same tool, pointed at a narrower question. These four cover most of what it is reached for day to day.

sort file.txt | uniq -d

Prints only the lines that occur more than once.

sort file.txt | uniq -u

Prints only the lines that occur exactly once.

sort file.txt | uniq -i

Treats upper and lower case as the same.

cut -d" " -f1 access.log | sort | uniq -c | sort -nr | head

The ten addresses making the most requests to a web server.

In cybersecurity

Counting repeats is how a pattern is pulled out of a noisy log.

Who is guessing passwords

Failed-login lines reduced to source addresses, then sort | uniq -c | sort -nr, lists the attackers by number of attempts.

Rare is interesting too

The lines that occur once — uniq -u — are often the unusual event: a single request to an odd path, a process seen on one host only.

Forgetting sort gives wrong numbers

Without sorting, the same value is counted in several separate runs and the totals are meaningless.

Filed under

Text processing

Read alongside

sortSorts lines of text — alphabetically, numerically, or by a chosen column.cutExtracts chosen columns from each line, by separator or by character position.