Linux · text processing

awk

work with columns

awk splits every line into fields and lets you refer to them as $1, $2, $3. It handles any amount of space between columns, which is why it is the tool for the output of ps, df and ls -l.

awk '{print $1}' prints the first column of anything, however it is spaced.

Fields

$1, $2 … $NF

$0 is the whole line

Separator

Whitespace

or -F to choose one

Filter

condition {action}

the action runs on matching lines

Arithmetic

Built in

sums and averages

At the terminal

Read the session first, then the notes: each numbered line in the output is answered by the note with the same number.

user@netflow — bashgawk

$ df -h | awk '{print $5, $6}'

1Use% Mounted

44% /

95% /home

2$ awk -F: '$3 >= 1000 {print $1}' /etc/passwd

user

deploy

3$ awk -F, 'NR>1 {sum += $4} END {print sum}' data.csv

6410000

Illustrative session · names, sizes and dates are examples, and every address is from a documentation or private range

Reading the output

What the numbered lines are telling you

Two columns out of six

The fifth and sixth fields of each line of df. The uneven spacing between columns does not matter to awk.

A condition before the action

-F: splits on colons. Only lines whose third field, the user id, is 1000 or more are printed — the human accounts.

Adding up a column

NR>1 skips the header line. sum grows with each row’s fourth field, and END prints it once the input is finished.

Switches

A switch changes how the command behaves. These are the ones used day to day; man awk lists the rest. Single-letter switches can usually be joined, so -l -a may be written -la.

-F SEP: Use this as the field separator instead of whitespace.
-v NAME=VALUE: Set a variable before the program runs.
-f FILE: Read the awk program from a file.

Variations worth knowing

The same tool, pointed at a narrower question. These four cover most of what it is reached for day to day.

ps aux | awk '$3 > 50 {print $2, $11}'

Prints the id and command of processes using more than 50% CPU.

awk '{print $NF}' file.txt

Prints the last field of each line, however many there are.

awk 'NR==5' file.txt

Prints line 5 only.

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head

The ten busiest client addresses in a web log.

In cybersecurity

awk is how a log line becomes data you can add up.

Totals and thresholds

Summing bytes sent per address, or printing only requests that returned 500, takes one line of awk and often answers the question on the spot.

Accounts with user id 0

awk -F: '$3 == 0' /etc/passwd lists every account with root's id. There should be exactly one.

Quote the program

The awk program belongs in single quotes. In double quotes the shell replaces $1 with its own value before awk runs.

Filed under

Text processing

Read alongside

cutExtracts chosen columns from each line, by separator or by character position.sedEdits text as it streams past: replaces, deletes or prints lines that match a pattern.