DevOpsIntermediate 13 min Lesson 9 of 15

Security Updates

Patch the three layers that matter — operating system, dependencies and container images — without breaking production.

DevOps · Lesson 9 of 15
0/15 done(0%)

What is it? #

Unpatched software is one of the two most common causes of compromise, and the vulnerabilities used are usually months old with patches long available.

There are three layers to keep current: the operating system, your application's dependencies, and container base images.

The difficulty is not knowing about updates — scanners report plenty. It is triaging them: most reported vulnerabilities do not apply to how you use the package.

The practical approach is automating the routine updates, triaging the rest by real exploitability, and having a fast path for the occasional urgent one.

Think of it like this #

Building maintenance. Safety recalls are dealt with immediately; routine servicing happens on a schedule; cosmetic improvements wait.

Treating every notice as urgent means nothing gets the attention it deserves, and treating none as urgent is how the roof falls in.

Simple example #

A critical vulnerability is announced in a web framework. Because dependency updates are routine and the test suite is trustworthy, the patched version is tested and deployed within two hours rather than debated for a week.

Code #

TEXT
The three layers

operating system    unattended-upgrades for security patches, scheduled reboots
dependencies        automated pull requests, tests decide whether they merge
container images    rebuild on base image updates; scan in CI

Each needs its own process. Patching one and ignoring the others
leaves the same exposure.
YAML
# Automated dependency updates
# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: pip
    directory: /
    schedule: { interval: weekly }
    open-pull-requests-limit: 5
    groups:
      minor-and-patch:              # group routine bumps into one PR
        update-types: [minor, patch]

  - package-ecosystem: docker
    directory: /
    schedule: { interval: weekly }

  - package-ecosystem: github-actions
    directory: /
    schedule: { interval: monthly }
YAML
# Scanning in CI, failing only on what matters
      - name: Scan dependencies
        run: |
          pip-audit --strict --ignore-vuln GHSA-xxxx-known-false-positive
      - name: Scan the image
        run: trivy image --severity HIGH,CRITICAL --exit-code 1 \
               --ignore-unfixed myapp:${{ github.sha }}
# --ignore-unfixed skips vulnerabilities with no available patch,
# which would otherwise block every build with no action available.
TEXT
Triage: not every CVE needs action

Ask, in order:
1. Is the vulnerable code path something we actually call?
   A parser flaw in a feature you never use is not exploitable.
2. Is it reachable from untrusted input?
   A vulnerability requiring local access on a single-tenant server
   is lower priority than one reachable over HTTP.
3. Is there a patch?
   If not, is there a mitigation — a configuration change, a WAF rule?
4. What is the blast radius if exploited?

Critical and reachable    patch today
High and reachable        patch this week
Anything unreachable      batch into the routine update cycle
BASH
# The emergency path, when it is genuinely urgent
# 1. Confirm exposure: are we running the affected version and configuration?
pip list | grep package-name
# 2. Patch, test, deploy through the normal pipeline — do not bypass it
# 3. Verify the deployed version
curl -s https://example.com/version
# 4. Check logs for signs of prior exploitation

How it works #

Automating the routine cases is what makes the urgent ones manageable. A team that updates dependencies weekly can apply a critical patch in hours, because the update is small and the pipeline is exercised.

A team that updates once a year faces a large, risky upgrade under time pressure, which is how urgent patches get delayed.

Grouping minor and patch updates into one pull request reduces review noise substantially while keeping major versions separate for deliberate handling.

--ignore-unfixed matters in practice. Without it, scanners report vulnerabilities with no available patch, the build fails, and there is nothing to do about it — which trains people to ignore the scanner entirely.

Triage by reachability is the key judgement. A vulnerability in a code path your application never executes is not exploitable in your system, and treating it with the same urgency as a reachable one wastes the attention needed for the real ones.

The emergency path deliberately goes through the normal pipeline. Bypassing tests to deploy a security patch quickly is how a patch causes an outage, which is a poor trade.

Checking logs for prior exploitation is the step most often skipped. If the vulnerability was public before you patched, it is worth knowing whether it was used.

Real-world use #

Automated dependency pull requests with a trustworthy test suite make updates routine. Without good tests, every update requires manual verification, and updates stop happening.

Base image updates are frequently forgotten. An application image built six months ago carries six months of unpatched operating system packages regardless of how current the application dependencies are.

Alert fatigue applies to vulnerability scanners as much as to monitoring. A scanner reporting two hundred findings, none of them actionable, is ignored.

Major version upgrades need their own planning. Batching them with security patches is how a security fix turns into a breaking change.

The strongest signal of a healthy process is the time between a patch being published and it running in production. Measuring that number tells you more than any scanner report.

Common mistakes #

  • Patching dependencies but never updating base images.
  • Treating every reported vulnerability as equally urgent.
  • Scanners failing builds on unfixable findings, so people ignore them.
  • Bypassing tests to deploy a security patch quickly.
  • Infrequent updates, so urgent patches require a large risky upgrade.

Practice #

Enable automated dependency updates and image scanning in one repository, with unfixable findings excluded. Then take the most recent critical advisory affecting you, determine whether the vulnerable code path is reachable, and record how long a patch would take to reach production.

Quick quiz

  1. 1. Which three layers need patching?

  2. 2. Why does frequent routine updating make urgent patches easier?

  3. 3. Why exclude unfixable findings from build failures?

  4. 4. What is the most important triage question?

  5. 5. Should an urgent security patch bypass the test pipeline?

Summary

  • Patch the OS, dependencies and base images — all three.
  • Automate routine updates so urgent ones are small and fast.
  • Triage by reachability, not by score alone.
  • Do not let unfixable findings block builds.
  • Measure the time from patch published to running in production.