First separate "looks out of memory" from "is actually out of memory"
A lot of people panic the moment free shows a low available figure, but available and free are two very different numbers. free is only the truly idle pages, while available estimates how many pages the kernel could reclaim under pressure—most of it page cache, which is cheap to drop and never triggers OOM.
So before drawing conclusions, check whether swap usage is steadily climbing, then look in dmesg for real OOM killer records. Rising swap means the kernel is actually paging anonymous memory out, which is hard evidence of memory pressure rather than the page-cache illusion.
free -h
cat /proc/meminfo | head -n 25
# 看 SwapTotal / SwapFree / SwapCached 趋势
vmstat 2 5Find the OOM-killed process name in dmesg / journalctl
Every time the OOM killer acts it leaves a decision record in the log: it picks a victim by oom_score and prints that process 's memory profile including anonymous memory, shared memory, and file mappings. The critical lines are the ones beginning with Memory cgroup out of memory plus the PID and name of the killed victim.
If the kernel has oom_score_adj enabled, or a process is protected by a cgroup, the true culprit may not be the killed victim but a sibling process in the same cgroup that exhausted the quota. So read the surrounding context instead of stopping at the last line naming the victim.
dmesg -T | grep -i -E "out of memory|killed process|oom-kill"
# systemd 环境优先看 journal
journalctl -k -b -1 -g "oom|killed process"Investigate real per-process usage: RSS, swap, and the top pitfall
The RES column in top is resident memory, but it includes shared-library pages that are double-counted across processes. When several Python/C children share libc, summing RES overstates real usage. For a closer approximation, aggregate PSS (Proportional Set Size) from smaps, which apportions shared pages by the number of mapping processes.
Another blind spot is the swap zombie: once anonymous pages are paged out, RES in top shrinks as if pressure vanished, though the memory only moved into the swap device. Cross-checking the SWAP column in ps with actual swap usage explains the odd case of tiny RES paired with a saturated swap.
ps aux --sort=-%mem | head
# 所有用户的整台机器 PSS 汇总
echo "PSS total: $(grep -E "^Total" /proc/*/smaps_rollup 2>/dev/null | awk "{s+=$3} END {print s/1024 \"MB\"}") "
# 找出占 swap 最多的进程
for p in /proc/[0-9]*; do awk -v pid=$(basename $p) "/Swap:/{s+=\$2} END{print pid, s/1024 \"MB\"}" $p/smaps 2>/dev/null; done | sort -k2 -rn | head -5Containers throttled by cgroups: OOM happens inside memory.max
In a container the symptoms look different from the host: free on the host looks fine, yet the container keeps dying. Each container has its own memory.max (formerly memory.limit_in_bytes), so OOM is adjudged inside the cgroup, not necessarily in sync with host-wide pressure.
Start by checking systemd-cgtop or cat /sys/fs/cgroup/<path>/memory.current, then confirm whether it collides with max. Many "mysterious container OOM" cases are just an application exhausting its 1 GiB request cap inside a single pod, unrelated to the host—so the fix is the container quota, not more physical RAM.
systemd-cgtop
echo /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.current
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.eventsCommon mistakes vs fixes: why swap did not prevent the kill
The common mistake is opening swap and assuming that alone makes it safe. swapiness defaults to 60, so under pressure the kernel prefers reclaiming page cache, using swap only as a backstop. And if vm.overcommit_memory=2 is set, the kernel may reject new allocations outright, failing malloc instead of OOM-killing—three quite different symptoms.
The correct move has two tiers: short-term, use sysctl to reserve headroom and adjust oom_score_adj to protect critical processes; long-term, identify which job causes the peak and cap the runaway with ulimit or a cgroup. Below, a flawed setup is replaced with a corrected one.
### 错误示范：只开 swap 不设防护
# vm.swappiness=60 保留默认，无 oom 保护
### 修复对照
sysctl -w vm.swappiness=10 # 极少换出，倾向回收缓存
sysctl -w vm.overcommit_memory=1 # 放开 overcommit，避免 malloc 直接失败
# 保护数据库进程优先级
systemctl set-property mysqld.service OOMScoreAdjust=-500
# 给失控作业套上限，防止它吃掉整机
ulimit -v 4194304Verify it works: simulate pressure instead of trusting luck
The strongest verification is to deliberately create pressure and watch whether the system freezes or kills the wrong process again. Use stress to consume a few gigabytes at once, then observe free, dmesg, and console logs, confirming: critical services survive, swap grows in a measured way, and no runaway process is minting fresh OOM records.
If you care about the per-process distribution, capture a snapshot of smaps_rollup mid-pressure, write each PID PSS to disk, and compare after the stress ends—this directly shows whether the reformed process stopped expanding.
# 用 stress 制造 2GB 瞬时压力
stress --vm 2 --vm-bytes 2G --vm-hang 10 --timeout 15
# 压力中看是否再次被杀
watch -n 1 "dmesg -T | tail -3; free -m | grep Mem"
# 抓 PSS 快照
for p in /proc/[0-9]*; do awk -v pid=$(basename $p) "/^Pss:/{s+=\$2} END{print pid, s}" $p/smaps 2>/dev/null; done | sort -k2 -rn | head -5 > /tmp/pss_snapshot.txt