CPU

What is it? #

CPU is the resource people blame first and diagnose least carefully.

Load average is the most misread number in Linux. It counts processes wanting to run, including those waiting on disk, and it is meaningless without knowing the core count.

High CPU is not automatically a problem. A server at 70% during peak hours is well used; one at 5% is over-provisioned.

The useful question is not "is CPU high" but "what is consuming it, and is it work we intended".

Think of it like this #

A kitchen with four chefs. Four orders in progress means fully used; eight means a queue forming.

Counting the orders without knowing how many chefs there are tells you nothing at all.

Simple example #

Response times double. CPU shows 95%, and the instinct is to add cores. Profiling reveals one endpoint doing an unindexed query and serialising results inefficiently — a code fix that removes the load entirely.

Code #

# Load average, read correctly
uptime
#  load average: 3.42, 2.87, 2.15      1min, 5min, 15min
nproc                                   # 4 cores
# 3.42 on 4 cores is busy but fine. On 1 core it is badly overloaded.

# Where is the time going?
top                    # press 1 for per-core, P to sort by CPU
mpstat -P ALL 2 5      # per-core breakdown over time
vmstat 2 5
# us  user code        → your application is doing work
# sy  system/kernel    → syscalls, often network or I/O heavy
# wa  iowait           → waiting on disk: the problem is NOT the CPU
# st  steal            → the hypervisor gave your time to someone else

# Which process, and which part of it?
ps aux --sort=-%cpu | head
pidstat -t -p <PID> 2 5             # per-thread breakdown

# Profiling a live process, without restarting it
py-spy top --pid <PID>              # Python
py-spy dump --pid <PID>             # current stack of every thread
perf top -p <PID>                   # any language, kernel-level

# Container CPU throttling — invisible in top
cat /sys/fs/cgroup/cpu.stat
# nr_throttled 1043      the container was throttled this many times
# throttled_usec 892000  for this long in total
#
# Throttling means the container hit its CPU limit and was paused.
# The application looks slow while host CPU appears fine.
docker stats                        # per-container usage against limits

What high CPU usually means

your code            a hot loop, inefficient serialisation, regex backtracking
garbage collection   frequent collection in a memory-pressured runtime
encryption           TLS termination without hardware acceleration
iowait (not CPU)     high wa means disk is the constraint
steal (not you)      high st means a noisy neighbour on shared hardware
throttling           the container limit, not the host, is the ceiling

How it works #

Load average counts running and runnable processes plus those in uninterruptible sleep, typically waiting on disk. That is why a machine with low CPU usage can show high load — the processes are waiting, not computing.

Comparing load with nproc is what makes it interpretable. A rule of thumb: sustained load above the core count means work is queueing.

The vmstat columns separate the cases. High us is your code. High sy suggests heavy syscall activity. High wa means the disk is the bottleneck and adding CPU will not help. High st means the hypervisor is giving your time to other tenants, which is a provider problem rather than yours.

Live profiling is the step that turns guessing into knowing. Tools like py-spy attach to a running process without restarting it, so you can profile production directly and see exactly which functions consume the time.

Container throttling is genuinely easy to miss. The application is slow, the host shows plenty of idle CPU, and the cause is the container hitting its own limit and being paused. The cpu.stat counters are where that appears.

The diagnosis order that works: confirm it is CPU and not iowait, find the process, profile it, then decide whether to fix the code or add capacity. Adding cores to inefficient code is expensive and often only postpones the problem.

Real-world use #

The most common real cause of high CPU in web applications is not computation but inefficiency: an N+1 query pattern, serialising more data than needed, or a regular expression with catastrophic backtracking.

Profiling in production is underused. Sampling profilers have low overhead and answer in minutes what hours of speculation cannot.

Container CPU limits set too low cause throttling that looks like application slowness. Checking throttling counters before raising limits or blaming the code is worth the thirty seconds.

Steal time on cheap shared instances is real. Persistent high steal is a reason to move to a different instance type or provider rather than to optimise anything.

CPU is usually the cheapest resource to add and the most expensive to waste. A day of profiling frequently removes more load than doubling the instance size.

Common mistakes #

• Reading load average without comparing it to the core count.
• Adding CPU when the real constraint is iowait.
• Missing container throttling because the host looks idle.
• Guessing at the hot code path instead of profiling.
• Scaling up to work around inefficient code that a fix would remove.

Practice #

On a machine under load, record the load average and core count, then use vmstat to determine whether the constraint is user CPU, system CPU or iowait. Profile the busiest process with a sampling profiler and identify the top three functions by time.