Processes

What is it? #

A process is a running program. Every one has a process ID, an owner, a parent, and a share of CPU and memory.

Most server troubleshooting starts here: what is running, what is using the CPU, what is using the memory, and why has something stopped.

Stopping a process is done by sending it a signal. SIGTERM asks politely and lets the program clean up; SIGKILL removes it immediately with no chance to save anything.

Reaching for kill -9 first is the habit to avoid. It is the equivalent of pulling the power cable.

Think of it like this #

Staff in a building. Some are working, some are waiting, one is monopolising the photocopier.

Asking someone to finish up and leave is SIGTERM. Escorting them out mid-sentence, dropping whatever they were holding, is SIGKILL.

Simple example #

The server is slow. You need to find which process is responsible, decide whether it is CPU or memory, and either restart it cleanly or investigate why it grew.

Code #

# What is running?
ps aux                      # every process, with user, CPU, memory, command
ps aux | grep python        # filter
pgrep -a gunicorn           # PIDs and command lines by name
pstree -p                   # the process tree, showing parents

# Live view
top                         # press M to sort by memory, P by CPU, q to quit
htop                        # nicer, if installed

# What is this process doing?
ps -p 1234 -o pid,ppid,user,%cpu,%mem,etime,cmd
ls -l /proc/1234/cwd        # its working directory
sudo lsof -p 1234           # files and sockets it has open
sudo ss -tulpn | grep 1234  # ports it is listening on

# Stopping processes — escalate, do not start at the end
kill 1234              # SIGTERM: asks the process to shut down cleanly
kill -TERM 1234        # the same thing, written explicitly
# wait a few seconds, check whether it exited
kill -9 1234           # SIGKILL: forced, no cleanup, data may be lost

pkill -f "gunicorn"    # by command line pattern
killall nginx          # by exact process name

# Prefer the service manager when there is one
sudo systemctl restart myapp     # handles order, dependencies and timeouts

Signals worth knowing

SIGTERM (15)   please shut down cleanly — the default for kill
SIGKILL (9)    stop immediately; cannot be caught or ignored
SIGINT  (2)    what Ctrl+C sends
SIGHUP  (1)    often means "reload your configuration"
SIGSTOP/CONT   pause and resume a process

# Why did my process disappear?
dmesg -T | grep -i "killed process"          # the OOM killer
journalctl -u myapp --since "1 hour ago"     # the service's own logs
# "Out of memory: Killed process 1234 (python)" means the kernel
# reclaimed memory by terminating the largest consumer.

How it works #

ps aux lists every process with its owner, CPU and memory share and full command. It is a snapshot; top is the live version.

In top, the load average is the number of processes wanting to run. Compare it with the core count from nproc: a load of 4 on four cores is fully busy, on one core it is badly overloaded.

lsof -p shows what a process has open, which answers questions like "why is this disk still full after deleting the log" — a deleted file still held open keeps its space until the process closes it.

kill sends SIGTERM by default, and a well-written program catches it, finishes its current work and exits. This is exactly the graceful shutdown covered in the background workers lesson.

kill -9 cannot be caught. The process is removed immediately, with unsaved work lost, open files unflushed and locks potentially left behind. It is the last resort, not the first move.

Using systemctl restart is better than killing a managed service, because the service manager handles dependencies, timeouts and restarting it afterwards.

The OOM killer check explains the most confusing failure: a process that vanished with no error in its own logs. The kernel, out of memory, terminated the largest consumer. Application logs show nothing because the process was given no chance to log anything.

Real-world use #

The first three commands during a slow-server incident are almost always top, df -h and free -h: what is busy, is the disk full, is memory exhausted.

Memory leaks appear as steadily growing resident memory in top over hours or days, usually ending in an OOM kill. Setting a memory limit in the service definition turns an unpredictable kill into a controlled restart.

Zombie processes — shown as defunct — are finished children whose parent has not collected them. A few are harmless; many indicate a bug in the parent.

In containers, the same tools apply but the view is restricted to that container's processes, and the container's memory limit is what triggers the OOM killer rather than the host's total.

The practical discipline is simple: try SIGTERM, wait, then escalate. Reaching for -9 immediately is how databases end up needing recovery on the next start.

Common mistakes #

• Using kill -9 first, preventing cleanup and risking data loss.
• Killing a managed service directly instead of using systemctl.
• Reading load average without comparing it to the number of cores.
• Not checking dmesg when a process disappears silently.
• Ignoring steadily rising memory until the OOM killer intervenes.

Practice #

Start a long-running process, find it with ps and pgrep, inspect its open files with lsof, then stop it with SIGTERM and confirm it exited cleanly. Then check dmesg on a machine you control and see whether the OOM killer has ever run.