First separate "looks out of memory" from "is actually out of memory"
A lot of people panic the moment free shows a low available figure, but available and free are two very different numbers. free is only the truly idle pages, while available estimates how many pages the kernel could reclaim under pressure—most of it page cache, which is cheap to drop and never triggers OOM.
So before drawing conclusions, check whether swap usage is steadily climbing, then look in dmesg for real OOM killer records. Rising swap means the kernel is actually paging anonymous memory out, which is hard evidence of memory pressure rather than the page-cache illusion.
free -h
cat /proc/meminfo | head -n 25
# Watch the SwapTotal / SwapFree / SwapCached trend
vmstat 2 5Find the OOM-killed process name in dmesg / journalctl
Every time the OOM killer acts it leaves a decision record in the log: it picks a victim by oom_score and prints that process 's memory profile including anonymous memory, shared memory, and file mappings. The critical lines are the ones beginning with Memory cgroup out of memory plus the PID and name of the killed victim.
If the kernel has oom_score_adj enabled, or a process is protected by a cgroup, the true culprit may not be the killed victim but a sibling process in the same cgroup that exhausted the quota. So read the surrounding context instead of stopping at the last line naming the victim.
dmesg -T | grep -i -E "out of memory|killed process|oom-kill"
# On systemd systems, check the journal first
journalctl -k -b -1 -g "oom|killed process"Investigate real per-process usage: RSS, swap, and the top pitfall
The RES column in top is resident memory, but it includes shared-library pages that are double-counted across processes. When several Python/C children share libc, summing RES overstates real usage. For a closer approximation, aggregate PSS (Proportional Set Size) from smaps, which apportions shared pages by the number of mapping processes.
Another blind spot is the swap zombie: once anonymous pages are paged out, RES in top shrinks as if pressure vanished, though the memory only moved into the swap device. Cross-checking the SWAP column in ps with actual swap usage explains the odd case of tiny RES paired with a saturated swap.
ps aux --sort=-%mem | head
# Whole-machine PSS total across all users
echo "PSS total: $(grep -E "^Total" /proc/*/smaps_rollup 2>/dev/null | awk "{s+=$3} END {print s/1024 \"MB\"}") "
# Find the processes using the most swap
for p in /proc/[0-9]*; do awk -v pid=$(basename $p) "/Swap:/{s+=\$2} END{print pid, s/1024 \"MB\"}" $p/smaps 2>/dev/null; done | sort -k2 -rn | head -5Containers throttled by cgroups: OOM happens inside memory.max
In a container the symptoms look different from the host: free on the host looks fine, yet the container keeps dying. Each container has its own memory.max (formerly memory.limit_in_bytes), so OOM is adjudged inside the cgroup, not necessarily in sync with host-wide pressure.
Start by checking systemd-cgtop or cat /sys/fs/cgroup/<path>/memory.current, then confirm whether it collides with max. Many "mysterious container OOM" cases are just an application exhausting its 1 GiB request cap inside a single pod, unrelated to the host—so the fix is the container quota, not more physical RAM.
systemd-cgtop
echo /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.current
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/memory.eventsCommon mistakes vs fixes: why swap did not prevent the kill
The common mistake is opening swap and assuming that alone makes it safe. swapiness defaults to 60, so under pressure the kernel prefers reclaiming page cache, using swap only as a backstop. And if vm.overcommit_memory=2 is set, the kernel may reject new allocations outright, failing malloc instead of OOM-killing—three quite different symptoms.
The correct move has two tiers: short-term, use sysctl to reserve headroom and adjust oom_score_adj to protect critical processes; long-term, identify which job causes the peak and cap the runaway with ulimit or a cgroup. Below, a flawed setup is replaced with a corrected one.
### Bad example: enable swap but add no protection
# vm.swappiness=60 left at default, no OOM protection
### Fix comparison
sysctl -w vm.swappiness=10 # swap out rarely, reclaim cache instead
sysctl -w vm.overcommit_memory=1 # loosen overcommit so malloc does not fail outright
# Protect the database process priority
systemctl set-property mysqld.service OOMScoreAdjust=-500
# Cap a runaway job so it cannot eat the whole box
ulimit -v 4194304Verify it works: simulate pressure instead of trusting luck
The strongest verification is to deliberately create pressure and watch whether the system freezes or kills the wrong process again. Use stress to consume a few gigabytes at once, then observe free, dmesg, and console logs, confirming: critical services survive, swap grows in a measured way, and no runaway process is minting fresh OOM records.
If you care about the per-process distribution, capture a snapshot of smaps_rollup mid-pressure, write each PID PSS to disk, and compare after the stress ends—this directly shows whether the reformed process stopped expanding.
# Use stress to create a 2GB spike
stress --vm 2 --vm-bytes 2G --vm-hang 10 --timeout 15
# During the load, watch whether it is killed again
watch -n 1 "dmesg -T | tail -3; free -m | grep Mem"
# Capture a PSS snapshot
for p in /proc/[0-9]*; do awk -v pid=$(basename $p) "/^Pss:/{s+=\$2} END{print pid, s}" $p/smaps 2>/dev/null; done | sort -k2 -rn | head -5 > /tmp/pss_snapshot.txt