CI/CD Pipelines: Designing One That Earns Its Keep

Most teams have a pipeline. Rather fewer have one they trust, and the difference shows up in behaviour: whether people watch their build, whether a red result triggers investigation or a re-run, whether anyone deploys on a Friday.

A pipeline that is ignored is not a safety net. It is a delay with a dashboard.

Speed is not a nice-to-have

The single most consequential property of a pipeline is how long it takes, because duration determines behaviour.

Under about ten minutes, people wait for the result and fix what breaks immediately, with the change fresh in mind. Beyond that, they context-switch, and the feedback arrives when they are thinking about something else. Beyond thirty minutes, they batch changes to avoid the wait, which makes each change larger and each failure harder to attribute — the precise opposite of what continuous integration is for.

What actually makes pipelines fast:

• Parallelism. Lint, type check, unit tests and build have no dependency on each other. Run them at the same time. This alone frequently halves wall-clock time.
• Aggressive caching. Dependencies, build layers, compiled artifacts. An uncached dependency install on every run is usually the largest single avoidable cost.
• Test splitting. Distribute the suite across several runners, ideally balanced by recorded timing rather than file count.
• Fail fast on cheap checks. Lint and type errors should be reported in under a minute, not after a fifteen-minute integration suite.
• Right-sized runners. Compute is cheaper than engineer waiting time by a wide margin. Teams under-provision runners to save an amount of money that is trivial next to the salary cost of the delay.

Flaky tests are the real enemy

A test that fails one run in twenty without a code change does more damage than no test at all, and the damage is cultural rather than technical.

Once a suite is known to be flaky, a red build stops meaning anything. People re-run it. Then they re-run it twice. Then they merge on the assumption it was flaky, and eventually a genuine failure is merged the same way. The pipeline still runs; it simply no longer functions as a gate.

Treating flakiness as a first-class defect is the only approach that works:

• Track it. Record which tests fail intermittently. Without data this is anecdote and nothing gets prioritised.
• Quarantine quickly. Move a known-flaky test out of the blocking path, and open a ticket with an owner. A quarantined test is honest; a flaky blocking test is corrosive.
• Fix the cause. The usual suspects are timing assumptions, shared state between tests, unmocked network calls and order dependence. All are fixable; none are fixed by adding a retry.
• Resist blanket retries. Automatic retries hide flakiness rather than removing it, and the hidden version grows.

Which gates are worth having

Every gate costs time on every run, so each one should be paying for itself.

Worth it, nearly always: fast unit tests; type checking, which catches a whole category of error for almost no time; linting with automatic formatting, which removes an entire class of review comment; dependency vulnerability scanning; secret scanning, because a leaked credential in history is expensive to remediate; and a build of the actual deployable artifact, since a build that only happens at deploy time is a build that breaks at deploy time.

Worth it selectively: integration tests, which are valuable and slow, so run the critical paths on every commit and the rest on merge to main; end-to-end browser tests, which are the most valuable and the most flaky — keep a small number covering genuine user journeys rather than a large suite covering everything; and performance budgets on the paths where regression actually matters.

Usually not worth blocking on: full coverage thresholds, which incentivise tests written to raise a number; every possible static analysis rule; and slow scans that could run nightly and report rather than blocking every merge.

Build once, promote the same artifact

The pattern that prevents an entire category of environment-specific surprise: build a single immutable artifact, then promote that exact artifact through staging and production. Configuration comes from the environment; the artifact never changes.

Rebuilding per environment reintroduces the possibility that what you tested is not what you shipped — a dependency resolved differently, a build cache in a different state, a base image updated in between. It is a subtle failure and an unpleasant one to diagnose, and it is entirely avoidable.

Deployment is not release

Separating these two is what makes continuous deployment safe.

Deployment puts code on a server. Release makes a feature visible to users. Keeping them separate means you can deploy small changes constantly, with the incomplete work behind a flag, and turn features on independently of the deploy that carried them.

What this enables in practice: merging unfinished work safely, which keeps branches short-lived; turning a broken feature off in seconds without a rollback deploy; gradual rollout to a percentage of users; and honest experimentation.

The associated discipline is removing flags once a feature is fully released. A codebase carrying two hundred dead flags is its own kind of problem, so treat flag removal as part of finishing the work rather than a cleanup task nobody schedules.

Deployment strategies, briefly

• Rolling replaces instances gradually. Simple, no extra capacity required, but two versions run concurrently — which your database migrations must tolerate.
• Blue-green runs the new version alongside the old and switches traffic at once. Instant rollback, at the cost of double capacity during the switch.
• Canary sends a small share of traffic to the new version and watches error rates before proceeding. The best risk profile and it requires monitoring good enough to make the decision automatically.

Whichever you choose, backward-compatible database migrations are the prerequisite. Expand the schema, deploy code that works with both shapes, migrate the data, then contract. Skipping this is how a rolling deploy takes the site down.

Boolean Solutions experience with delivery pipelines

We build and inherit pipelines regularly, and the two changes that most reliably improve a team’s delivery are not sophisticated. Make the pipeline fast, by parallelising and caching properly. Then make it trustworthy, by treating every flaky test as a defect with an owner.

Teams that do both start deploying several times a day without anyone mandating it, because it stops being frightening. If your pipeline has grown into something people work around rather than with, we are happy to take a look.

Further reading

• DORA’s four key metrics — the research behind why deployment frequency and lead time predict performance
• Continuous Integration — Martin Fowler; the definition, and a reminder of what CI actually requires
• Feature Toggles — Pete Hodgson; the taxonomy and the lifecycle discipline
• Flaky Tests at Google — scale changes the numbers, not the lessons