awk Cheatsheet - Linux Text Processing Reference

Q: How do I test whether a field matches a regex in awk?

Use the ~ operator: awk '$1 ~ /pattern/' file to match and awk '$1 !~ /pattern/' file to invert. index($1, "pattern") is the equivalent for plain substring search, but it takes a literal string and cannot handle regexes. To test the whole record, just write /pattern/.

Q: How do I make awk emit CSV?

Set the output field separator: awk 'BEGIN{OFS=","} {print $1, $2}' file prints comma-separated values. Set it once in BEGIN, not inside the main rule, or it will be reset on every record. For a quoted CSV variant, add a quote variable and build the field manually inside print.

Logs and reports share one trait — one field per line — and awk's strength is working on columns: counting status codes in an access log, summing values per user, joining files on a key. One awk program replaces a Python script here. The sheet runs from basic syntax through built-in variables, conditionals, statistics, string functions and common scenarios, forming the complete skeleton: a pattern plus an action per line, with BEGIN and END blocks. Built-in variables come first because everything depends on them: FS splits input, OFS separates output, NR is the global record number and NF the field count. The statistics group rests on associative arrays: count[$9]++ tallies by field value and sum[group]+=value accumulates by group, and with an END block they produce a finished report. The division of labour: sed rewrites lines without understanding columns, awk understands columns and can aggregate, and cut extracts but computes nothing.

SysOps·37 commands·Last updated 2026-07-21

Typical Use Case

For engineers building reports and analysing logs. Keep the division of labour with sed straight: sed edits or deletes lines, awk extracts columns and computes. This sheet is organised into six groups — basic syntax, built-in variables, conditionals, statistics, string functions and common patterns. The three patterns worth memorising are column extraction piped through sort/uniq for counting, conditional extraction such as $9 == 500 {print $7}, and BEGIN/END blocks, which are awk's distinguishing feature for pre- and post-processing.

Basic Syntax 6

awk '{print $1}' file

Print the first column (space-separated by default)

awk -F: '{print $1, $3}' /etc/passwd

Use colon as the field separator

awk '{print $1, $NF}' file

Print first and last column

awk '{print NR, $0}' file

Print line number and whole line

awk 'NR==10' file

Print line 10

awk 'NR>=10 && NR<=20' file

Print lines 10-20

Built-in Variables 8

$0

The whole line

$1, $2, $NF

1st / 2nd / last column

NR

Current line number (global)

NF

Number of columns in the current line

FS

Input field separator (space by default)

OFS

Output field separator

RS

Input record separator (newline by default)

ORS

Output record separator

Conditionals 6

awk '$3 > 100' file

Print lines where column 3 > 100

awk '$1 == \"error\"' file

Print lines where column 1 equals 'error'

awk '$1 ~ /pattern/' file

Column 1 matches a regex

awk '$1 !~ /pattern/' file

Column 1 does not match a regex

awk 'NR==1 || $1 > 100' file

First line or column 1 > 100

awk '$1 == \"a\" {print $2}' file

Condition with action

Statistics 5

awk '{sum += $1} END {print sum}' file

Sum column 1

awk '{sum += $1} END {print sum/NR}' file

Average of column 1

awk '{count++} END {print count}' file

Count lines

awk '{count[$1]++} END {for(k in count) print k, count[k]}' file

Group and count by column 1

awk 'BEGIN{max=0} {if($1>max) max=$1} END{print max}' file

Find the max of column 1

String Functions 7

length($0)

Length of the current line

substr($0, 5, 10)

Substring from char 5, length 10

index($0, \"text\")

Find substring position (1-based; 0 if not found)

split($0, arr, \":\")

Split into an array by a delimiter

sub(/old/, \"new\", $0)

Replace first match

gsub(/old/, \"new\", $0)

Replace all matches

tolower($0) / toupper($0)

Convert to lower/upper case

Common Patterns 5

awk '{print $1}' access.log | sort | uniq -c | sort -rn

Top IPs by access count

awk '$9 == 500 {print $7}' access.log

Extract URLs with 500 errors

awk -F, '{print $2, $3}' data.csv

Process a CSV file

awk 'BEGIN{OFS=\",\"} {print $1, $2}' file

Output as CSV

awk '{sum+=$5} END{print sum/1024/1024 \" MB\"}' file

Sum a column and convert units

Parameter matrix

Parameter	Effect	Example
`{print $1}`	Print the 1st column (whitespace-separated by default)	`awk '{print $1}' file`
`-F<sep>`	Specify the input delimiter	`awk -F: '{print $1, $3}' /etc/passwd`
`$0`	The whole line	`awk '{print NR, $0}' file`
`$NF`	The last column (most useful when the column count varies)	`awk '{print $1, $NF}' file`
`NR / NF`	Current line number / number of fields in the current line	`awk 'NR==10' file`
`FS / OFS`	Input / output delimiters	`awk 'BEGIN{OFS=","} {print $1, $2}' file`
`$3 > 100`	Numeric comparison (awk auto-detects numbers)	`awk '$3 > 100' file`
`$1 ~ /pat/`	Regex match (!~ for no match)	`awk '$1 ~ /^error/' app.log`
`BEGIN / END`	Run once before / after processing	`awk '{sum+=$1} END {print sum}' file`
`sum += $1`	Accumulate, can be combined with END output	`awk '{sum+=$1} END {print sum/NR}' file`
`count[$1]++`	Group and count with an associative array	`awk '{c[$1]++} END {for(k in c) print k, c[k]}' file`
`sub / gsub`	Replace the first / all matches	`gsub(/old/, "new", $0)`
`split / substr`	Split into an array by delimiter / extract a substring	`split($0, arr, ":")`
`Pipeline composition`	Combine with sort / uniq for counting	`awk '{print $1}' f \| sort \| uniq -c \| sort -rn`

Common pitfalls

SymptomThe columns awk extracts are all empty, or the whole line is treated as one field.

CauseThe delimiter does not match: the file is colon-separated but -F: was not given, and awk splits on whitespace by default.

FixUse head to look at the real delimiter first, then specify the correct one with awk -F: or -F,.

SymptomThe sum comes out as 0 or abnormally large.

CauseThe column contains non-numeric characters (such as units or timestamps) that are treated as 0 in the computation.

FixUse head to confirm the data format; if needed gsub away the non-numeric characters first, or force numeric conversion with $1+0.

SymptomThe END block cannot access the accumulator variable.

CauseThe scope of the accumulator variable differs between the main block and the END block (commonly because the variable is declared inside a conditional branch).

FixInitialize the accumulator in the BEGIN block, or make sure the first assignment of the accumulation statement is at the top level rather than inside a branch.

SymptomA regex in awk matches nothing.

CauseSpecial characters in the regex (. * + ? [ ]) must be escaped.

FixFor literal matching use /\./ or index/substr instead of a regex; do not put string quotes around the two sides of ~.

SymptomCounting occurrences of each item, every count comes out as 1.

CauseThe pipeline is missing sort; the awk associative array does not sort itself, but not aggregating first causes duplicate entries to be overwritten.

FixWrite all four steps: awk extraction | sort | uniq -c | sort -rn.

SymptomOutput columns run together with no separation.

Causeprint separates multiple expressions with OFS by default, but OFS was not set explicitly and the comma-separated form was treated as string concatenation.

FixSet the output delimiter with BEGIN{OFS=","}; separate expressions with commas in print $1, $2.

SymptomWhen handling fields that contain spaces, the whole line gets split apart.

CauseThe default FS is whitespace (including space and Tab), so a field whose value contains a space gets broken up.

FixSet FS to a fixed single character: -F" " or -F"|".

Troubleshooting

1First confirm the data delimiter and column positions
head -3 file
See the real delimiter before writing expressions; this avoids most rework from being unable to get the column.
2Count the top 20 access counts
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
Extract column, sort and aggregate, count, then sort numerically descending; all four steps are required.
3Extract fields by condition
awk '$9 == 500 {print $7}' access.log
$9 is the status code and $7 is the URL; put the condition first and the action in {}.
4Compute a column sum and convert units
awk '{sum+=$5} END {print sum/1024/1024 " MB"}' file
Output the result in the END block; concatenating the unit as a string is the clearest.
5Group and count by the first column
awk '{c[$1]++} END {for(k in c) print k, c[k]}' file
awk arrays are associative, so any value can be used as a key directly.

Command Examples

Group and count by the first column

awk '{count[$1]++} END {for(k in count) print k, count[k]}' access.log | sort -k2 -rn | head

count[$1]++ uses the array subscript as the key and accumulates line by line; the END block outputs everything after all input is read. Append sort to order by count, since awk itself does not sort.

Output

203.0.113.7 1842
198.51.100.22 931
192.0.2.180 407

Sum a column and convert to MB

awk '{sum += $5} END {printf "%.2f MB\n", sum/1024/1024}' access.log

Use printf in END to control output precision and the unit. Write the division step by step; awk does not support a power operator like 1024*1024.

Output

482.31 MB

Emit CSV by setting OFS

awk -F, 'BEGIN{OFS=","} NR>1 {print $2, $3}' data.csv > out.csv

-F, sets the input delimiter to a comma and OFS sets the output delimiter. NR>1 skips the CSV header row, otherwise the title gets written into the result too.

Common Pitfalls

Field references start at 1 and $0 is the whole line; there is no $1[0].
awk splits on whitespace by default; parsing CSV or colon-separated files requires an explicit -F or every line becomes one field.
Set OFS once in BEGIN — putting it in the main rule resets it on every record.
Commas inside quoted CSV fields are treated as separators; FPAT handles the harder cases.
awk does not sort. Pipe through sort | uniq or the result is unordered.

Tips

awk splits on whitespace or tabs by default; use -F to specify another separator.
The BEGIN block runs before processing, the END block runs after.
awk arrays are associative (hash tables); keys can be strings.

FAQ

Do awk field references start at 0 or 1?

They start at 1. $0 is the whole record, $1 the first field, $2 the second, and $NF the last. This trips people up because arrays also use arr[1]. In awk $1 always means a column, not an array element, and $NF is especially handy when the number of columns varies between records.

How do I compute grouped counts in awk?

Use a count array plus an END block: awk '{count[$1]++} END {for(k in count) print k, count[k]}' file counts occurrences grouped by the first column. Counting IP hits in a log becomes awk '{print $1}' access.log | sort | uniq -c | sort -rn. For a sum use awk '{sum += $1} END {print sum}', and divide by NR for an average.

What is the difference between NF and $NF?

NF is the number of fields in the current record — a number. $NF is the value of the last field — a string. Both appear together when columns vary: test NF < 3 to skip malformed lines, or use $(NF-1) for the second-to-last field. Confusing the two produces a type error at runtime.

How do I test whether a field matches a regex in awk?