Disk

What is it? #

Disk causes two distinct kinds of production problem: running out of space, and running out of speed.

Running out of space is the more common and the more damaging. A full disk breaks logging, breaks databases and often breaks the ability to fix the problem.

There is a second way to run out that surprises people: inodes. A filesystem can have plenty of free bytes and no free inodes, which happens with millions of tiny files.

Running out of speed is subtler. High I/O wait means the CPU is idle waiting for storage, and adding CPU does nothing.

Think of it like this #

A warehouse that can be full in two ways: no floor space left, or no shelf labels left even though there is floor space.

And separately, a warehouse where everything fits but the forklift is too slow, so nothing moves.

Simple example #

The application stops responding. The disk is 100% full. The cause is an unrotated log that grew to 40 GB during an earlier incident, and now the database cannot write either.

Code #

# Space
df -h                                   # per filesystem
du -sh /var/* | sort -rh | head         # biggest directories under /var
du -h --max-depth=1 /var/log | sort -rh
find / -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

# Inodes — the second way to run out
df -i
# Filesystem  Inodes  IUsed  IFree  IUse%
# /dev/vda1   2.0M    2.0M   1.2K   100%     ← full, despite free bytes
find /var -xdev -type d -exec sh -c 'echo "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mi>l</mi><mi>s</mi><mo>−</mo><mn>1</mn><mi mathvariant="normal">&quot;</mi></mrow><annotation encoding="application/x-tex">(ls -1 &quot;</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0197em;">l</span><span class="mord mathnormal">s</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="katex-base"><span class="katex-strut" style="height:0.6944em;"></span><span class="mord">1&quot;</span></span></span></span>1" | wc -l) $1"' _ {} \; \
  | sort -rn | head                          # directories with the most files

# Deleted but still open — space not returned
sudo lsof +L1 | head
# COMMAND  PID  USER  FD  TYPE  ...  NLINK  NODE NAME
# python   812  app   3w  REG        0      1234 /var/log/app.log (deleted)
#
# Deleting a file a process still has open does NOT free the space.
# Restart the process, or truncate instead of deleting:
sudo truncate -s 0 /var/log/app.log

# Speed
iostat -x 2 5
# %util     how busy the device is; near 100% means saturated
# await     average time per request in ms; rising means queueing
# r/s w/s   read and write operations per second

vmstat 2 5 | awk '{print $16}'     # the wa column: CPU waiting on I/O

iotop -o                            # which process is doing the I/O

The usual culprits for a full disk

logs without rotation      the most common by a wide margin
Docker images and volumes  docker system df; prune regularly
old release directories    a deployment script that never cleans up
database WAL or binlogs    replication stopped, so they are never recycled
core dumps                 a crashing process writing gigabytes each time
temporary files            uploads or exports never cleaned up

# Alert BEFORE it is full — 85% gives hours of warning
df / --output=pcent | tail -1 | tr -dc '0-9'
df -i / --output=ipcent | tail -1 | tr -dc '0-9'   # inodes too

How it works #

df shows what is full; du shows what is responsible. Both are needed, and the sorted du one-liner usually identifies the cause within seconds.

Inode exhaustion occurs because a filesystem allocates a fixed number of inodes at creation, one per file. Millions of small files — session files, cache entries, tiny logs — exhaust them while bytes remain free. The symptom is confusing: "no space left on device" with df -h showing free space.

Deleted-but-open files are the other confusing case. Unlinking a file a process still has open removes the directory entry but not the data; the space returns only when the process closes it. lsof +L1 lists these, and truncating rather than deleting avoids the problem for logs.

For speed, %util near 100% with rising await means the device is saturated and requests are queueing. High wa in vmstat confirms that the CPU is idle waiting for storage.

Reaching 100% full is worse than it sounds because recovery becomes difficult: you may be unable to write the file needed to fix it, and some databases refuse to start.

Alerting at 85% is deliberate. It gives hours of warning in normal conditions, and logs can consume the final 15% quickly during an incident.

Real-world use #

A full disk is among the most common production incidents, and unrotated logs are the usual cause. It frequently happens during another incident, when error logging is at its peak.

Docker accumulates aggressively. Build servers in particular need a scheduled docker system prune, or they fill within weeks.

Database write-ahead logs grow without limit when replication stops or a replication slot is left inactive, which turns a replica problem into a primary outage.

Inode exhaustion typically appears in session storage, cache directories or mail queues, and is baffling the first time because there is visibly free space.

I/O saturation on cheap storage is a real constraint. A database on slow network-attached storage can be I/O bound at modest traffic, and the fix is faster storage rather than more CPU or memory.

Common mistakes #

• Alerting only at 95%, leaving no time to react.
• Monitoring bytes but not inodes.
• Deleting a log file a process still has open and wondering why space did not return.
• Never pruning Docker images and volumes.
• Adding CPU when high iowait shows the constraint is storage.

Practice #

On a server, find the five largest directories and the five largest files. Check inode usage as well as bytes. Then create a large file, delete it while a process holds it open, and confirm with lsof that the space is still consumed.