Running Your App as a systemd Service

What is it? #

A systemd unit file is how your application becomes a proper service: started at boot, restarted on failure, logged centrally and stopped cleanly.

The Linux track covered the syntax. This lesson is the production checklist: what each setting should be and why, for an application you intend to leave running.

Five things matter most. Run as a non-root user. Restart on failure, but give up if it is failing constantly. Allow enough time for a graceful shutdown. Cap memory. Load configuration from an env file.

The hardening options cost nothing and meaningfully reduce what a compromised process can do.

Think of it like this #

An employment contract for the service: what it does, who it runs as, what happens when it falls over, how much notice it gets before being asked to leave, and which rooms of the building it may enter.

Simple example #

An application that must start at boot, restart within five seconds of a crash, finish in-flight requests within thirty seconds during deployments, and never exceed 1 GB of memory.

Code #

# /etc/systemd/system/myapp.service
[Unit]
Description=MyApp production API
Documentation=https://wiki.internal/runbooks/myapp
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/srv/app/current

EnvironmentFile=/srv/app/shared/.env
Environment=PYTHONUNBUFFERED=1 NODE_ENV=production

ExecStartPre=/srv/app/current/.venv/bin/python -c "import app"   # fail fast on import errors
ExecStart=/srv/app/current/.venv/bin/gunicorn app.main:app --bind 127.0.0.1:8000 --workers 4
ExecReload=/bin/kill -s HUP $MAINPID

# Failure handling
Restart=always
RestartSec=5
StartLimitBurst=5                 # five failures...
StartLimitIntervalSec=120         # ...within two minutes, then stop trying

# Graceful shutdown
KillSignal=SIGTERM
KillMode=mixed
TimeoutStopSec=45

# Resource limits
MemoryMax=1G
MemoryHigh=800M                   # throttle before the hard limit
CPUQuota=200%                     # two cores
LimitNOFILE=65535                 # file descriptors for many connections

# Hardening — cheap and effective
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
ReadWritePaths=/srv/app/shared/uploads /var/log/myapp

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload       # after ANY unit file edit
sudo systemctl enable --now myapp
systemctl status myapp
journalctl -u myapp -f

# Verify the hardening did not break anything
systemd-analyze security myapp     # scores the unit and lists gaps

Checking the restart behaviour actually works

sudo systemctl status myapp                  # note the PID
sudo kill -9 <PID>                           # simulate a crash
sleep 6 && systemctl status myapp            # should be running with a new PID

Then simulate a permanent failure (break the env file) and confirm systemd
gives up after the start limit rather than looping forever.

How it works #

ExecStartPre runs a quick check before the main command. An import error caught here fails the start immediately with a clear message, rather than producing a confusing crash loop.

Restart=always with a start limit is the balance that matters. Transient crashes recover automatically; a permanently broken service stops after five attempts and is marked failed, which is what you want to alert on. Without the limit, a broken service restarts forever and the problem hides.

KillMode=mixed sends SIGTERM to the main process and SIGKILL to any remaining children after the timeout, which suits worker-based servers.

TimeoutStopSec=45 must exceed the application's own graceful shutdown period. If systemd kills the process before it finishes, in-flight requests are dropped during every deployment.

MemoryHigh throttles the process as it approaches the limit, giving it a chance to recover, while MemoryMax is the hard stop. Together they turn a leak into a controlled restart rather than an OOM kill on a random process.

LimitNOFILE matters for servers handling many concurrent connections; the default is often too low and produces confusing "too many open files" errors.

ProtectSystem=strict makes the entire filesystem read-only except the paths listed in ReadWritePaths. A compromised process then cannot modify system files or other applications.

systemd-analyze security scores the unit and suggests further hardening, which is a quick way to find easy improvements.

Real-world use #

Most production services on a VPS are a unit file like this one. Getting it right once gives you a template for everything else.

The restart-limit behaviour is important for alerting. A service in a restart loop with no limit looks alive in a naive check while serving nothing.

Resource limits per service are underused and valuable. One application leaking memory should not take down the database on the same machine.

The hardening directives are free and take minutes. ProtectSystem=strict alone removes a large part of what a compromised process could do, and systemd-analyze security shows how far you have got.

Testing the restart behaviour deliberately — killing the process and watching it recover — is worth doing once per service. Assuming it works is how you discover on a bad night that it does not.

Common mistakes #

• Forgetting daemon-reload after editing the unit file.
• Restart=always with no start limit, hiding a permanently broken service.
• TimeoutStopSec shorter than the application’s graceful shutdown period.
• No memory limit, so one leaking service destabilises the whole machine.
• Never testing that the service actually restarts after a crash.

Practice #

Write a production unit file for your application with a restart policy, start limit, graceful shutdown timeout, memory cap and the hardening directives. Kill the process and confirm it recovers, then break the configuration and confirm systemd stops trying after the limit.