OOM killer
The OOM (out-of-memory) killer is the part of the Linux kernel that ends a process when the system runs out of memory and cannot free any. It chooses the process by its OOM score, usually the one using the most memory. The kernel log records every OOM kill.
1How the victim is chosen
Each process has a score in /proc/<pid>/oom_score that grows with its memory use. /proc/<pid>/oom_score_adj, from -1000 to 1000, adjusts it; -1000 means the process is never chosen.
2Finding OOM kills
The kernel writes 'Out of memory: Killed process' to its log. journalctl -k | grep -i oom or dmesg | grep -i oom shows these messages, with the process name and its memory use.
3Prevention
Add memory or swap, limit memory-hungry services, or set limits per service with systemd options such as MemoryMax=. Some distributions also run systemd-oomd, which acts earlier based on memory pressure.
How to investigate an OOM kill
- Search the kernel log:
journalctl -k | grep -i -A5 'out of memory' - Note the killed process and its memory size.
- Check current memory and swap:
free -h - Find the biggest processes:
ps aux --sort=-rss | head - Add swap or limit the service, then watch whether it happens again.
Cautions
- Setting oom_score_adj to -1000 for many processes can leave the kernel with no safe choice.
- Turning off memory overcommit protection without understanding it can cause failures elsewhere.
Related articles
- Swap space on LinuxWhat swap is, how to see it with free and swapon, how to add a swap file, and what vm.swappiness changes.
- Load averageWhat the three load average numbers mean, how they relate to CPU cores, and when a high load is a problem.
- journalctl and the systemd journalHow to read logs with journalctl, check how much space the journal uses and shrink it safely with --vacuum options.
Sources
- proc(5) manual page: oom_score and oom_score_adj man7.org
- Out Of Memory Management (kernel.org) kernel.org
Note: RainServer offers monitoring and safe cleanup for Linux servers; it shows what will change and asks first.
Last reviewed: