Canary deployments: traffic control, evidence and promotion
Design a canary using controlled exposure and revision-specific signals. Calculate sample size limits, distinguish traffic from replica weights and define promotion and abort conditions.
TL;DR: A canary exposes a new version to a bounded population, compares its behavior with the existing version, and increases exposure only when the evidence supports doing so. Traffic assignment, meaningful signals and a tested abort path determine how much risk it actually reduces.
Choose what you are exposing
A canary might receive a percentage of requests, a set of tenants or one internal region. These choices test different things. Random request routing spreads load, but a user may alternate between versions. Tenant routing keeps a customer's workflow together, but the first tenants may be unrepresentative. An internal-only cohort may never exercise payment or geographic edge cases.
Suppose a checkout service handles 2,000 requests per minute. A 5% request canary receives approximately 100 requests per minute under evenly distributed routing. That limits direct exposure, but the canary may still write to a shared database or exhaust a shared connection pool. Identify shared dependencies before calling the rollout isolated.
The Google SRE canarying chapter explains the value of comparing a candidate with a control. Use the comparison to separate a release regression from a dependency outage affecting both versions.
Replica count and request count differ
With ten replicas and one new Pod, roughly 10% of equally distributed new connections may reach the new version. Sticky sessions, long-lived connections, uneven Pod capacity and load-balancer behavior can produce a different request share. A traffic-management layer can assign weights independently of replica counts, although its actual behavior still needs measurement.
Argo Rollouts documents this distinction: without traffic management, it approximates the configured weight using whole replica counts. With only three replicas, a 5% replica canary is impossible. Adding a weight field does not create fractional Pods.
The percentages are illustrative targets. Instrument observed request counts by revision so the analysis can verify that the candidate received the expected traffic.
Write the decision before the deployment
| Signal | Useful condition | Common mistake |
|---|---|---|
| Error fraction | Candidate exceeds an absolute limit or a meaningful difference from stable | Aggregate errors hide a small failing canary |
| Latency | Compare relevant percentiles for comparable routes and payloads | Mix fast health checks with slow customer calls |
| Correctness | Verify business invariants or representative synthetic actions | Accept HTTP 200 as proof of correct output |
| Saturation | Candidate and shared dependencies retain operating headroom | Watch only the candidate's CPU |
| Evidence volume | Require enough observations over a relevant time window | Promote on zero errors from twelve requests |
Make missing telemetry a hold condition. Otherwise a broken exporter can look like a perfect release. Decide how the rollout reacts to an unrelated incident: pause, retain evidence and resume under an explicit decision, rather than repeatedly promoting and aborting on noisy signals.
Work the arithmetic
Assume independent requests and a true candidate failure probability of 1%. The probability of seeing no failures in 100 requests is 0.99^100, about 36.6%. Zero failures in that sample gives weak assurance. For 1,000 independent requests, the probability falls to about 0.0043%.
Real requests are often correlated, so these numbers illustrate the sampling problem rather than certify a rollout. One tenant's repeated calls do not cover a hundred different tenant configurations. Choose both a minimum sample and a duration long enough to exercise relevant behavior such as cache expiry or a scheduled task.
Compare equivalent populations. If stable serves mostly cached reads and candidate serves checkout writes, a raw latency comparison will attribute traffic mix to the release. Segment on a bounded set of route templates or operation types. Avoid adding user IDs as metric labels; the cardinality discipline concept explains that cost.
Abort the exposure and account for state
An abort usually stops further promotion and routes eligible traffic back to stable. It does not undo messages already emitted or database rows already written. Keep both versions compatible with the shared state during the canary window. Retain enough stable capacity to accept returned traffic without creating a second incident.
If the candidate introduced malformed records, recovery may require a forward fix and a controlled repair. A rollout controller cannot infer that from a healthy Pod count. Link the abort procedure to the rollback compatibility plan.
Self-check: a 5% canary has a 20% error rate while stable is healthy. What does the aggregate show?
Assuming the intended traffic share is accurate, the aggregate error rate is about 1%. A broad dashboard could make the release appear tolerable while one in five candidate requests fails. Halt promotion, inspect candidate-specific failures and reduce exposure according to the recovery plan.
Practice choosing among blue-green, canary and rolling deployment. State how you will observe the candidate before naming a rollout tool.