What is it? #
Every production server has scheduled work: backups, cleanup of expired data, certificate renewal, report generation, cache warming.
The Linux track covered cron syntax. This lesson is about making scheduled jobs reliable on a real server, which is mostly about four things.
Run them as the right user. Set up the environment explicitly, because cron provides almost none. Prevent overlapping runs. And make failure visible.
A scheduled job that has silently failed for three weeks is worse than not having it, because you believed it was working.
Think of it like this #
A cleaner who comes at night when nobody is watching.
If they stop coming, you find out weeks later when the place is a mess. A note signed each visit — or an alert when the note does not appear — is what makes the arrangement trustworthy.
Simple example #
A nightly backup, an hourly cleanup of expired sessions, and a weekly report. Each runs as the application user, logs its output, cannot overlap, and pings a monitor on success.
Code #
# Edit the application user's crontab, not root's, unless root is required
sudo -u appuser crontab -eSHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO="" # we handle reporting ourselves
# Nightly backup — locked, logged, monitored
0 2 * * * /usr/bin/flock -n /tmp/backup.lock /srv/app/scripts/backup.sh >> /var/log/myapp/backup.log 2>&1
# Hourly cleanup
15 * * * * /usr/bin/flock -n /tmp/cleanup.lock /srv/app/current/.venv/bin/python /srv/app/current/manage.py cleanup_sessions >> /var/log/myapp/cleanup.log 2>&1
# Weekly report, Mondays at 06:00
0 6 * * 1 /srv/app/current/.venv/bin/python /srv/app/current/manage.py weekly_report >> /var/log/myapp/report.log 2>&1#!/usr/bin/env bash
# /srv/app/scripts/backup.sh — the pattern every scheduled script should follow
set -euo pipefail
cd /srv/app/current # never rely on the working directory
set -a; source /srv/app/shared/.env; set +a # cron loads no shell profile
echo "[$(date -Is)] starting"
if ! pg_dump "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>D</mi><mi>A</mi><mi>T</mi><mi>A</mi><mi>B</mi><mi>A</mi><mi>S</mi><msub><mi>E</mi><mi>U</mi></msub><mi>R</mi><mi>L</mi><mi mathvariant="normal">"</mi><mi mathvariant="normal">∣</mi><mi>g</mi><mi>z</mi><mi>i</mi><mi>p</mi><mo>></mo><mi mathvariant="normal">"</mi><mi mathvariant="normal">/</mi><mi>b</mi><mi>a</mi><mi>c</mi><mi>k</mi><mi>u</mi><mi>p</mi><mi>s</mi><mi mathvariant="normal">/</mi><mi>d</mi><mi>b</mi><mo>−</mo></mrow><annotation encoding="application/x-tex">DATABASE_URL" | gzip > "/backups/db-</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.0278em;">D</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.0502em;">B</span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.0576em;">S</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.0576em;">E</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="katex-sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.109em;">U</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal" style="margin-right:0.0077em;">R</span><span class="mord mathnormal">L</span><span class="mord">"∣</span><span class="mord mathnormal" style="margin-right:0.0359em;">g</span><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="mord mathnormal">i</span><span class="mord mathnormal">p</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">></span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">"/</span><span class="mord mathnormal">ba</span><span class="mord mathnormal">c</span><span class="mord mathnormal" style="margin-right:0.0315em;">k</span><span class="mord mathnormal">u</span><span class="mord mathnormal">p</span><span class="mord mathnormal">s</span><span class="mord">/</span><span class="mord mathnormal">d</span><span class="mord mathnormal">b</span><span class="mord">−</span></span></span></span>(date +%F).sql.gz"; then
echo "[$(date -Is)] FAILED" >&2
curl -fsS -m 10 "https://hc-ping.com/UUID/fail" || true
exit 1
fi
curl -fsS -m 10 "https://hc-ping.com/UUID" || true # success ping
echo "[$(date -Is)] finished"# The systemd timer alternative — logs and status included
# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
[Service]
Type=oneshot
User=appuser
EnvironmentFile=/srv/app/shared/.env
ExecStart=/srv/app/scripts/backup.sh
# /etc/systemd/system/backup.timer
[Unit]
Description=Run the nightly backup
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true # catch up after downtime
RandomizedDelaySec=300
[Install]
WantedBy=timers.targetsudo systemctl enable --now backup.timer
systemctl list-timers # when did it last run, when does it run next
journalctl -u backup.service --since "1 week ago"How it works #
Running jobs as the application user rather than root follows the same least-privilege principle as everything else. A cleanup script does not need to be able to modify system files.
Cron provides a minimal environment: a short PATH, no shell profile, and the home directory as the working directory. The script handles all three explicitly, which is why jobs that work interactively fail under cron.
flock -n prevents overlap. When a backup that normally takes two minutes starts taking four hours after data growth, the lock stops three copies running simultaneously.
Redirecting output to a log file keeps a record. With MAILTO empty and no redirection, output is discarded entirely.
The monitoring ping is the piece that matters most. Services that expect a regular check-in alert when it stops arriving, which catches the case where the job did not run at all — something log checking cannot detect.
set -euo pipefail stops the script at the first failure. Without it, a failed pg_dump still produces a gzip file, and an empty backup looks exactly like a successful one.
systemd timers give each run a unit with status and journal logs, and Persistent=true runs a missed job after downtime, which cron simply skips.
Real-world use #
Backups, session cleanup, certificate renewal, data exports and reconciliation jobs are the usual scheduled work on a VPS.
The failure mode that causes real damage is the silent one. A backup job that has been failing for a month is discovered at exactly the wrong moment.
Overlapping runs cause real incidents too: a cleanup job that takes longer than its interval accumulates copies until the machine runs out of memory.
With more than one server, running the same crontab on each means the job runs once per server. A single designated scheduler or a distributed lock is required.
Timezones matter. Servers are usually set to UTC to avoid daylight-saving ambiguity, which means a 02:00 job runs at 02:00 UTC regardless of local time.
Common mistakes #
- Assuming cron has your PATH, environment and working directory.
- No output redirection, so failures leave no trace.
- No lock, so a slow run overlaps with the next.
- No alerting, so a job that stopped running goes unnoticed.
- The same crontab on several servers, so scheduled jobs run multiple times.
Practice #
Schedule a backup script that uses flock, redirects output to a log, sources the env file explicitly and pings a monitoring service on success and failure. Deliberately make it fail once and confirm both the log entry and the alert appear.