systemd

What is it? #

systemd is the service manager on most Linux distributions. You describe a service in a unit file and it handles the rest.

A unit file is a short configuration file with three sections: what the service is and what it needs, how to run it, and when it should start.

The things worth getting right are the restart policy, the user it runs as, how configuration reaches it, and a shutdown timeout long enough for clean exit.

Logs go to the journal automatically, which means your application can simply print to standard output and journalctl will have it.

Think of it like this #

A job description pinned to the wall: what this role does, what it needs in place before starting, how to do the work, and what to do if the person calls in sick.

Written once, followed consistently, and anyone can read it to understand how the service is supposed to behave.

Simple example #

A Python application needs to start after the network and the database, run as an unprivileged user, read secrets from an environment file, restart on failure, and stop cleanly within 30 seconds.

Code #

# /etc/systemd/system/myapp.service

[Unit]
Description=MyApp API server
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/srv/app

EnvironmentFile=/srv/app/.env           # KEY=value lines, chmod 600
Environment=PYTHONUNBUFFERED=1          # so logs appear immediately

ExecStart=/srv/app/.venv/bin/gunicorn app.main:app \
          --workers 4 --bind 127.0.0.1:8000
ExecReload=/bin/kill -HUP $MAINPID

Restart=always
RestartSec=5
StartLimitBurst=5                       # give up after 5 rapid failures
StartLimitIntervalSec=60

KillSignal=SIGTERM
TimeoutStopSec=30                       # time to finish in-flight work

# Hardening: the service only gets what it needs
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/srv/app/uploads /var/log/myapp
MemoryMax=1G

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload        # required after editing a unit file
sudo systemctl enable --now myapp
systemctl status myapp

# Logs
journalctl -u myapp -f              # follow
journalctl -u myapp --since "1 hour ago"
journalctl -u myapp -p err          # errors only
journalctl -u myapp -n 200 --no-pager
journalctl -u myapp --since today | grep "timeout"

Common Type values

simple    the process started by ExecStart is the service (the usual choice)
exec      like simple, but systemd waits until the binary has been executed
forking   the process forks and the parent exits (older-style daemons)
oneshot   runs once and exits; used with timers for scheduled jobs
notify    the service tells systemd when it is genuinely ready

How it works #

After= controls ordering, and Wants= expresses a soft dependency. Ordering does not imply readiness: After=postgresql.service means systemd starts it later, not that the database is accepting connections. Applications still need to retry their initial connection.

User= and Group= are what keep the service off root. Combined with WorkingDirectory, the process starts in the right place with the right identity.

EnvironmentFile= loads secrets from a file with 600 permissions, which keeps them out of the unit file and out of version control.

PYTHONUNBUFFERED=1 matters more than it looks. Without it, Python buffers output and your logs appear in delayed chunks, which is maddening during an incident.

Restart=always with RestartSec=5 restarts after any exit. The start limit stops a genuinely broken service restarting in a tight loop forever — after five failures in a minute, systemd gives up and marks it failed, which is what you want an alert on.

TimeoutStopSec=30 gives the process time to finish in-flight work after SIGTERM before systemd forces it, which pairs with the graceful shutdown handling in your application.

The hardening block is cheap and effective. ProtectSystem=strict makes the whole filesystem read-only except the paths you list, so a compromised service cannot modify system files.

daemon-reload is required after editing a unit file. Forgetting it means systemd keeps using the old definition, which produces confusing results.

Real-world use #

Every service on a typical Linux server — Nginx, PostgreSQL, Redis, your application, your workers — is a unit file, and reading them is a fast way to learn how a machine is configured.

The journal replaces log files for service output. It is indexed by unit, priority and time, which makes journalctl -u myapp --since "10 minutes ago" -p err a very direct way to find what went wrong.

Journal size needs bounding. SystemMaxUse=1G in journald.conf prevents it consuming the disk, which is a common cause of a full root filesystem.

systemd timers are the modern alternative to cron for scheduled work, with the advantage that each run is a unit with logs and status.

For deployments, systemctl reload or a socket-activated restart can avoid dropped connections, though most setups simply accept a brief restart behind a load balancer.

Common mistakes #

• Forgetting daemon-reload after editing a unit file.
• Assuming After= means the dependency is ready, not merely started.
• Running as root because User= was omitted.
• No TimeoutStopSec, so in-flight work is killed during deploys.
• Restart=always with no start limit, hiding a permanently broken service in a restart loop.

Practice #

Write a unit file for a small script that prints a line every second. Run it as a non-root user, set Restart=always, enable it, then kill the process and confirm it restarts. Read its output with journalctl, then add MemoryMax and ProtectSystem and confirm it still runs.