Automated Testing in CI

What is it? #

Tests in CI are what make automated deployment possible. Without them, the pipeline is an automated way to ship bugs faster.

The practical concerns are different from writing tests locally: speed, reliability and what the result gates.

Speed, because a slow suite delays every change. Reliability, because a test that fails randomly destroys trust in all of them. Gating, because a test suite nobody has to pass is a suggestion.

The mix matters too. Many fast unit tests, fewer integration tests, and a small number of end-to-end tests covering critical journeys.

Think of it like this #

Quality checks on a production line. Most are quick measurements at each station, a few are assembly checks, and one is a full function test of the finished product.

Doing only full function tests would be slow and would tell you something is wrong without telling you where.

Simple example #

A suite of 800 unit tests running in 40 seconds, 60 integration tests against a real database in 90 seconds, and 8 end-to-end tests covering signup, checkout and login in two minutes.

Code #

The mix, by proportion and purpose

unit          many, milliseconds each   one function or class, no I/O
integration   fewer, tens of ms         real database, real queries
end-to-end    few, seconds each         a full user journey through the UI

Rough guide: 70% unit, 20% integration, 10% end-to-end.
End-to-end tests are the most valuable per test and the most expensive
to run and maintain, so cover the journeys that would lose money.

# Splitting for speed
jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - run: pytest tests/unit -q -n auto        # parallel across cores

  integration:
    runs-on: ubuntu-latest
    services:
      postgres: { image: postgres:16, options: --health-cmd pg_isready }
    steps:
      - run: pytest tests/integration -q

  e2e:
    needs: [unit, integration]                   # only if the cheap ones pass
    runs-on: ubuntu-latest
    steps:
      - run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: failure()                            # keep the evidence
        with:
          name: playwright-report
          path: playwright-report/

Flaky tests: causes and fixes

timing         waiting a fixed number of seconds
               → wait for a condition, not a duration

shared state   tests depending on data left by other tests
               → each test creates and cleans up its own data

ordering       tests that pass alone but fail in a suite
               → run in random order to expose it

external calls real network requests in tests
               → stub them; keep one contract test against the real thing

time and zone  tests that fail at midnight or in another timezone
               → freeze time explicitly in tests

What to enforce before merging

required     unit and integration tests pass
             linting and type checking pass
             no new critical vulnerabilities in dependencies

useful       coverage does not drop on changed files
             end-to-end tests pass on the main journeys

counterproductive
             an absolute coverage percentage as a hard gate — it
             encourages tests written to cover lines rather than behaviour

How it works #

Splitting tests across jobs means they run in parallel on separate machines, so total pipeline time is the slowest job rather than the sum.

Running end-to-end tests only after the cheaper suites pass avoids spending several minutes discovering something that a unit test caught in seconds.

-n auto distributes tests across CPU cores within a job. It requires tests to be independent, which is a useful forcing function for test quality.

Uploading artefacts on failure — screenshots, traces, reports — is what makes a failed end-to-end test diagnosable. Without them, "it failed in CI" is a starting point for guesswork.

Flakiness is the issue that matters most for trust. Once people believe failures might be random, they re-run instead of investigating, and a genuine failure gets re-run too.

The usual causes are all fixable. Fixed sleeps become condition waits, shared fixtures become per-test setup, and random ordering exposes hidden dependencies before they become mysterious.

Coverage as a hard percentage gate tends to produce tests that execute lines without asserting anything meaningful. Coverage on changed lines is a more useful signal, and reviewing what is untested beats chasing a number.

Real-world use #

The most common CI complaint is slow tests, and the most common cause is integration tests doing setup work repeatedly rather than sharing fixtures or running in parallel.

Teams that tolerate flaky tests gradually stop trusting CI entirely, at which point the pipeline provides false confidence. Quarantining a flaky test and fixing it is better than leaving it in the main suite.

Test databases should be created and destroyed per run. Reusing a persistent one accumulates state and produces failures that only occur in CI.

Contract tests against external services are a good compromise: stub the service in most tests, and run one test against the real thing on a schedule so interface changes are noticed.

When a bug reaches production, adding a test that reproduces it before fixing is what stops it returning, and is the highest-value test you can write.

Common mistakes #

• Tolerating flaky tests, which destroys trust in every result.
• Running everything in one sequential job when parallelism is available.
• Running expensive end-to-end tests before cheap unit tests.
• Enforcing an absolute coverage percentage instead of meaningful assertions.
• No artefacts on failure, so CI failures cannot be diagnosed.

Practice #

Split an existing test suite into unit, integration and end-to-end jobs, running the first two in parallel. Add artefact upload on failure. Then run the suite ten times and identify any test that does not produce the same result every time.