awk Cheatsheet - Linux Text Processing Reference

Q: How do I test whether a field matches a regex in awk?

Use the ~ operator: awk '$1 ~ /pattern/' file to match and awk '$1 !~ /pattern/' file to invert. index($1, "pattern") is the equivalent for plain substring search, but it takes a literal string and cannot handle regexes. To test the whole record, just write /pattern/.

Q: How do I make awk emit CSV?

Set the output field separator: awk 'BEGIN{OFS=","} {print $1, $2}' file prints comma-separated values. Set it once in BEGIN, not inside the main rule, or it will be reset on every record. For a quoted CSV variant, add a quote variable and build the field manually inside print.

Logs and reports share one trait — one field per line — and awk's strength is working on columns: counting status codes in an access log, summing values per user, joining files on a key. One awk program replaces a Python script here. The sheet runs from basic syntax through built-in variables, conditionals, statistics, string functions and common scenarios, forming the complete skeleton: a pattern plus an action per line, with BEGIN and END blocks. Built-in variables come first because everything depends on them: FS splits input, OFS separates output, NR is the global record number and NF the field count. The statistics group rests on associative arrays: count[$9]++ tallies by field value and sum[group]+=value accumulates by group, and with an END block they produce a finished report. The division of labour: sed rewrites lines without understanding columns, awk understands columns and can aggregate, and cut extracts but computes nothing.

SysOps·37 commands·Last updated 2026-07-21

Typical Use Case

For engineers building reports and analysing logs. Keep the division of labour with sed straight: sed edits or deletes lines, awk extracts columns and computes. This sheet is organised into six groups — basic syntax, built-in variables, conditionals, statistics, string functions and common patterns. The three patterns worth memorising are column extraction piped through sort/uniq for counting, conditional extraction such as $9 == 500 {print $7}, and BEGIN/END blocks, which are awk's distinguishing feature for pre- and post-processing.

Basic Syntax 6

awk '{print $1}' file

Print the first column (space-separated by default)

awk -F: '{print $1, $3}' /etc/passwd

Use colon as the field separator

awk '{print $1, $NF}' file

Print first and last column

awk '{print NR, $0}' file

Print line number and whole line

awk 'NR==10' file

Print line 10

awk 'NR>=10 && NR<=20' file

Print lines 10-20

Built-in Variables 8

$0

The whole line

$1, $2, $NF

1st / 2nd / last column

NR

Current line number (global)

NF

Number of columns in the current line

FS

Input field separator (space by default)

OFS

Output field separator

RS

Input record separator (newline by default)

ORS

Output record separator

Conditionals 6

awk '$3 > 100' file

Print lines where column 3 > 100

awk '$1 == \"error\"' file

Print lines where column 1 equals 'error'

awk '$1 ~ /pattern/' file

Column 1 matches a regex

awk '$1 !~ /pattern/' file

Column 1 does not match a regex

awk 'NR==1 || $1 > 100' file

First line or column 1 > 100

awk '$1 == \"a\" {print $2}' file

Condition with action

Statistics 5

awk '{sum += $1} END {print sum}' file

Sum column 1

awk '{sum += $1} END {print sum/NR}' file

Average of column 1

awk '{count++} END {print count}' file

Count lines

awk '{count[$1]++} END {for(k in count) print k, count[k]}' file

Group and count by column 1

awk 'BEGIN{max=0} {if($1>max) max=$1} END{print max}' file

Find the max of column 1

String Functions 7

length($0)

Length of the current line

substr($0, 5, 10)

Substring from char 5, length 10

index($0, \"text\")

Find substring position (1-based; 0 if not found)

split($0, arr, \":\")

Split into an array by a delimiter

sub(/old/, \"new\", $0)

Replace first match

gsub(/old/, \"new\", $0)

Replace all matches

tolower($0) / toupper($0)

Convert to lower/upper case

Common Patterns 5

awk '{print $1}' access.log | sort | uniq -c | sort -rn

Top IPs by access count

awk '$9 == 500 {print $7}' access.log

Extract URLs with 500 errors

awk -F, '{print $2, $3}' data.csv

Process a CSV file

awk 'BEGIN{OFS=\",\"} {print $1, $2}' file

Output as CSV

awk '{sum+=$5} END{print sum/1024/1024 \" MB\"}' file

Sum a column and convert units

Parameter matrix

Parameter	Effect	Example
`{print $1}`	打印第 1 列（默认空白分隔）	`awk '{print $1}' file`
`-F<sep>`	指定输入分隔符	`awk -F: '{print $1, $3}' /etc/passwd`
`$0`	整行内容	`awk '{print NR, $0}' file`
`$NF`	最后一列（列数不定时最有用）	`awk '{print $1, $NF}' file`
`NR / NF`	当前行号 / 当前行列数	`awk 'NR==10' file`
`FS / OFS`	输入 / 输出分隔符	`awk 'BEGIN{OFS=","} {print $1, $2}' file`
`$3 > 100`	数值比较（awk 自动识别数字）	`awk '$3 > 100' file`
`$1 ~ /pat/`	正则匹配（!~ 为不匹配）	`awk '$1 ~ /^error/' app.log`
`BEGIN / END`	处理前 / 处理后执行一次	`awk '{sum+=$1} END {print sum}' file`
`sum += $1`	累加，可结合 END 输出	`awk '{sum+=$1} END {print sum/NR}' file`
`count[$1]++`	关联数组分组计数	`awk '{c[$1]++} END {for(k in c) print k, c[k]}' file`
`sub / gsub`	替换第一个 / 全部匹配	`gsub(/old/, "new", $0)`
`split / substr`	按分隔符拆数组 / 截取子串	`split($0, arr, ":")`
`管道组合`	与 sort / uniq 组合做统计	`awk '{print $1}' f \| sort \| uniq -c \| sort -rn`

Common pitfalls

Symptomawk 取到的列全是空的或整行被当成一个字段。

Cause分隔符不匹配——文件是冒号分隔却没加 -F:，awk 默认按空白切。

Fix先用 head 看一眼真实分隔符，再用 awk -F: 或 -F, 指定正确分隔符。

Symptomsum 统计结果为 0 或异常大。

Cause列中含有非数字字符（如单位、时间戳），被按 0 参与运算。

Fix用 head 确认数据格式；必要时先 gsub 去掉非数字字符，或改用 $1+0 显式转数。

SymptomEND 块里访问不到累加变量。

Cause累加变量在主块与 END 块中的作用域不一致（常见于把变量声明在条件分支内）。

Fix在 BEGIN 块里初始化累加器，或确保累加语句在顶层而非某个分支内首次赋值。

Symptomawk 里的正则匹配不到内容。

Cause正则中的特殊字符（. * + ? [ ]）需要转义。

Fix字面匹配用 /\./ 或 index/substr 代替正则；~ 两侧不要写字符串引号。

Symptom统计每个项的次数，结果全是 1。

Cause管道漏了 sort，awk 的关联数组本身不排序，但未先聚合导致重复项被覆盖。

Fix写全四步：awk 提取 | sort | uniq -c | sort -rn。

Symptom输出列之间连在一起，没有分隔。

Causeprint 对多个表达式默认用 OFS 分隔，但 OFS 未显式设置且用逗号分隔的写法被当成了字符串拼接。

FixBEGIN{OFS=","} 设置输出分隔符；print $1, $2 用逗号分隔表达式。

Symptom处理带空格的字段时整行被切开。

Cause默认 FS 是空白（含空格与 Tab），字段值内部有空格时会被拆开。

Fix把 FS 设为固定单字符：-F" " 或 -F"|"。

Troubleshooting

1先确认数据分隔符与列位置
head -3 file
看清真实分隔符再写表达式，能避免大部分「取不到列」的返工。
2统计访问量 TOP 20
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
取列 → 排序聚合 → 计数 → 按数值降序，四步缺一不可。
3按条件提取字段
awk '$9 == 500 {print $7}' access.log
$9 是状态码、$7 是 URL；条件放前面 {} 放动作。
4计算列总和并换算单位
awk '{sum+=$5} END {print sum/1024/1024 " MB"}' file
END 块输出结果，字符串拼接单位最直观。
5按第一列分组统计
awk '{c[$1]++} END {for(k in c) print k, c[k]}' file
awk 的数组是关联数组，可直接用任意值做键。

Command Examples

Group and count by the first column

awk '{count[$1]++} END {for(k in count) print k, count[k]}' access.log | sort -k2 -rn | head

count[$1]++ 用数组下标做键，逐行累加；END 块在全部读完后统一输出。再接 sort 才能按计数排序——awk 本身不排序。

Output

203.0.113.7 1842
198.51.100.22 931
192.0.2.180 407

Sum a column and convert to MB

awk '{sum += $5} END {printf "%.2f MB\n", sum/1024/1024}' access.log

END 里用 printf 控制输出精度与单位。除法要逐层写，awk 不支持 1024*1024 这样的幂运算符。

Output

482.31 MB

Emit CSV by setting OFS

awk -F, 'BEGIN{OFS=","} NR>1 {print $2, $3}' data.csv > out.csv

-F, 指定输入分隔符为逗号，OFS 指定输出分隔符。NR>1 跳过 CSV 的表头行，否则标题也会被写进结果。

Common Pitfalls

Field references start at 1 and $0 is the whole line; there is no $1[0].
awk splits on whitespace by default; parsing CSV or colon-separated files requires an explicit -F or every line becomes one field.
Set OFS once in BEGIN — putting it in the main rule resets it on every record.
Commas inside quoted CSV fields are treated as separators; FPAT handles the harder cases.
awk does not sort. Pipe through sort | uniq or the result is unordered.

Tips

awk splits on whitespace or tabs by default; use -F to specify another separator.
The BEGIN block runs before processing, the END block runs after.
awk arrays are associative (hash tables); keys can be strings.

FAQ

Do awk field references start at 0 or 1?

They start at 1. $0 is the whole record, $1 the first field, $2 the second, and $NF the last. This trips people up because arrays also use arr[1]. In awk $1 always means a column, not an array element, and $NF is especially handy when the number of columns varies between records.

How do I compute grouped counts in awk?

Use a count array plus an END block: awk '{count[$1]++} END {for(k in count) print k, count[k]}' file counts occurrences grouped by the first column. Counting IP hits in a log becomes awk '{print $1}' access.log | sort | uniq -c | sort -rn. For a sum use awk '{sum += $1} END {print sum}', and divide by NR for an average.

What is the difference between NF and $NF?

NF is the number of fields in the current record — a number. $NF is the value of the last field — a string. Both appear together when columns vary: test NF < 3 to skip malformed lines, or use $(NF-1) for the second-to-last field. Confusing the two produces a type error at runtime.

How do I test whether a field matches a regex in awk?