Server Updates

What is it? #

Unpatched software is one of the two most common causes of server compromise, and the vulnerabilities used are usually months old with patches long available.

The fix is to apply security updates automatically and reboot on a schedule when the kernel changes.

Full upgrades are a different matter. Upgrading every package automatically can change behaviour and restart services at unpredictable times, so most teams automate security updates only.

Kernel updates need a reboot to take effect. Patched on disk and vulnerable in memory is a common and easily missed state.

Think of it like this #

Servicing a vehicle. Safety recalls get done as soon as possible; optional upgrades wait for a convenient slot.

And a repair that needs the engine restarted is not finished until the engine has actually been restarted.

Simple example #

A production server applies security patches nightly without intervention, reboots during a defined window when a kernel update requires it, and alerts if it has been waiting to reboot for too long.

Code #

# Manual update, for the first run and for controlled maintenance
sudo apt update
sudo apt list --upgradable          # see what will change before agreeing
sudo apt upgrade                     # upgrade installed packages
sudo apt autoremove                  # clean up orphaned dependencies

# Is a reboot required?
[ -f /var/run/reboot-required ] && cat /var/run/reboot-required

# Automatic security updates
sudo apt install unattended-upgrades apt-listchanges
sudo dpkg-reconfigure --priority=low unattended-upgrades

# /etc/apt/apt.conf.d/50unattended-upgrades
Unattended-Upgrade::Allowed-Origins {
    "<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi>d</mi><mi>i</mi><mi>s</mi><mi>t</mi><mi>r</mi><msub><mi>o</mi><mi>i</mi></msub><mi>d</mi></mrow><mo>:</mo></mrow><annotation encoding="application/x-tex">{distro_id}:</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal">d</span><span class="mord mathnormal">i</span><span class="mord mathnormal">s</span><span class="mord mathnormal">t</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord"><span class="mord mathnormal">o</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3117em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="katex-sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">d</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span></span></span></span>{distro_codename}-security";      // security only
};
Unattended-Upgrade::Package-Blacklist {
    "postgresql-16";        // upgrade the database deliberately, not automatically
};
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "03:30";     // a defined window
Unattended-Upgrade::Mail "[email protected]";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";

# Verify it is actually working
systemctl status unattended-upgrades
sudo unattended-upgrade --dry-run --debug | tail -20
grep -c "Packages that will be upgraded" /var/log/unattended-upgrades/unattended-upgrades.log

# Alert if a reboot has been pending too long
#!/usr/bin/env bash
if [ -f /var/run/reboot-required ]; then
    AGE=<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mo stretchy="false">(</mo><mo stretchy="false">(</mo></mrow><annotation encoding="application/x-tex">(( (</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(((</span></span></span></span>(date +%s) - $(stat -c %Y /var/run/reboot-required)) / 86400 ))
    if [ "$AGE" -gt 3 ]; then
        curl -fsS -m 10 "https://alerts.example.com/reboot-pending?days=$AGE" || true
    fi
fi

How it works #

apt list --upgradable shows what will change before you agree to it. On a production machine, reading that list is worth the thirty seconds, particularly before a manual full upgrade.

Restricting Allowed-Origins to the security pocket means only security patches are applied automatically. Feature updates wait for a deliberate maintenance window.

Blacklisting the database is a common choice. A database upgrade may need a restart, a migration or configuration review, and having it happen at 3am unattended is not what anyone wants.

Automatic-Reboot with a defined time is the piece most setups omit. Without it, kernel patches are downloaded and installed but never take effect, which leaves the machine running the vulnerable kernel indefinitely.

Remove-Unused-Kernel-Packages prevents old kernels accumulating and filling /boot, which is a surprisingly common cause of a failed upgrade on servers with a small boot partition.

The dry run confirms the configuration is valid and shows what would be upgraded, which is how you verify the automation actually works rather than assuming it does.

The alert script catches the case where a reboot has been pending for days — either because automatic reboot is disabled or because it silently failed.

Real-world use #

A single VPS with no redundancy cannot reboot without downtime, which is why teams often postpone it indefinitely. Scheduling a short window at a quiet hour is far better than never rebooting.

With two servers behind a load balancer, rolling reboots give you patching with no user-visible downtime, which is one of the practical arguments for the second machine.

Live kernel patching exists and avoids some reboots, though it is usually a paid feature and does not cover every update.

Automatic updates occasionally restart a service at an inconvenient moment. Blacklisting the critical ones and handling them in a maintenance window is the usual compromise.

The most important thing is verification. Enabling unattended upgrades and never checking the log is how teams discover months later that it has been failing silently.

Common mistakes #

• Enabling automatic updates and never verifying they run successfully.
• Installing kernel patches but never rebooting.
• Automating full upgrades, so behaviour changes unpredictably.
• Letting old kernels fill /boot until an upgrade fails.
• Postponing reboots indefinitely because there is no redundancy.

Practice #

Enable automatic security updates on a test server with a defined reboot window and the database blacklisted. Run a dry run to confirm the configuration works, then check the log after a day and confirm patches were applied.