Memory

What is it? #

Memory problems are harder to diagnose than CPU problems because the numbers are widely misread.

Linux uses free memory for disk caching. A server reporting almost no free memory is usually healthy — the cache is reclaimable at any time.

The number that matters is available, not free. Confusing the two produces a great deal of unnecessary alarm.

When memory genuinely runs out, the kernel terminates a process to reclaim it. That is the OOM killer, and it usually picks the largest consumer — often your database.

Think of it like this #

A desk covered in papers you are currently referencing. It looks full, and clearing space takes a moment whenever you need it.

Panicking about the covered desk is the mistake. Running out of room even after clearing it is the real problem.

Simple example #

Monitoring shows 95% memory used and someone raises an alarm. free -h shows most of it is cache and several gigabytes are available. Two weeks later, a genuine leak fills that available memory and the OOM killer terminates PostgreSQL.

Code #

free -h
#               total   used   free   shared  buff/cache   available
# Mem:           7.8Gi  2.1Gi  312Mi    89Mi       5.4Gi       5.3Gi
#                              └── looks alarming, is not
#                                                    └── what actually matters
# available = free + reclaimable cache. 5.3Gi is genuinely usable.

# Who is using memory?
ps aux --sort=-%mem | head
top    # press M to sort by memory

# RSS vs VSZ
# RSS  resident set size: physical memory actually in use — this is the one
# VSZ  virtual size: address space reserved, often far larger and misleading

# Detailed breakdown for one process
pmap -x <PID> | tail -1
cat /proc/<PID>/status | grep -E "VmRSS|VmSwap"

# Did the OOM killer act?
dmesg -T | grep -i "killed process"
journalctl -k | grep -i "out of memory"
# "Out of memory: Killed process 1234 (postgres) total-vm:..., anon-rss:2.1GB"
#
# The victim is chosen by an internal score, roughly the largest consumer.
# Your database is frequently the largest process on the machine.

Detecting a leak

Plot RSS over days, not minutes. A leak looks like a slow, steady climb
that never returns to baseline after traffic drops.

  normal   /\  /\  /\  /\       rises with load, falls back
  leak     /  /  /  /  /         rises with load, never falls

Common causes: unbounded in-memory caches, listeners or timers never
removed, growing lists on a long-lived object, connection objects not
closed.

# Turn leaks into restarts rather than OOM kills
# systemd
MemoryMax=1G
MemoryHigh=800M          # throttle before the hard limit

# Gunicorn / Puma / similar
--max-requests 1000 --max-requests-jitter 100    # recycle workers

# Docker
--memory 1g --memory-reservation 512m

How it works #

Linux fills unused memory with page cache because unused memory is wasted memory. The cache is dropped instantly when a process needs the space, which is why free being low is not a problem.

available accounts for that: it estimates how much memory can be given to a new process without swapping. It is the number to alert on.

RSS is physical memory in use; VSZ includes reserved address space that may never be touched. Alerting on VSZ produces false alarms, particularly for runtimes that reserve large address ranges.

The OOM killer scores processes and terminates the highest, which correlates strongly with memory usage. That is why the database is a frequent victim, and why per-service memory limits matter — they make the application the one that gets restarted instead.

A leak is identified by shape over time rather than by any single reading. Memory that rises with load and never returns to baseline afterwards is the signature.

Setting MemoryMax converts an unbounded leak into a service restart. MemoryHigh throttles the process as it approaches the limit, sometimes allowing it to recover without restarting.

Worker recycling — restarting after a number of requests — is a pragmatic defence that contains slow leaks without needing to find them, though it is a mitigation rather than a fix.

Swap deserves a note: a small amount helps the kernel move genuinely idle pages out. Heavy swapping is worse than an OOM kill, because everything becomes extremely slow rather than failing cleanly.

Real-world use #

The most common false alarm in server monitoring is treating low free memory as a problem. Alerting on available avoids it entirely.

Real leaks are usually application-level: a cache with no eviction policy, event listeners added per request, or objects retained by a long-lived reference. The Docker and systemd limits contain the damage while you find the cause.

The OOM killer taking down the database is a genuinely common production incident, and it is prevented by limiting the application rather than by adding memory.

Memory limits in containers are what make behaviour predictable. Without them, one leaking container can destabilise an entire host.

Before adding memory, it is worth checking whether the working set simply does not fit — a database with too little cache memory does far more disk reads, which appears as slowness rather than as a memory alarm.

Common mistakes #

• Alerting on free memory instead of available memory.
• Reading VSZ rather than RSS when assessing process memory.
• No per-service memory limits, so the OOM killer picks the database.
• Diagnosing a leak from a single reading rather than a trend.
• Adding memory to work around a leak instead of finding it.

Practice #

On a running server, compare free, available and buff/cache and explain what each means. Identify the three largest processes by RSS, then plot one service's memory over 24 hours and decide whether the shape indicates a leak.