Canary/progressive deployment auto

Automated canary and progressive deployment is the practice of rolling out changes to a small percentage of production traffic first, automatically monitoring key metrics during the rollout, and either automatically promoting to full traffic (if metrics look good) or automatically rolling back (if metrics degrade). The entire lifecycle - from "deploy to 5%" to "promoted to 100%" or "rolled back to previous" - is automated without human intervention in the happy path.

Manual canary deployments existed before AI-assisted development. The L4 upgrade is the automation: removing the human who watches the canary and makes the promotion/rollback decision. At 50+ PRs/day, manually watching each canary deployment is not feasible. Automated analysis - comparing error rates, latency distributions, and key business metrics between canary and baseline using statistical analysis - makes the promotion/rollback decision reliably without human attention.

Flagger (built on top of Kubernetes) and Argo Rollouts are the primary implementations. Both support analysis templates: define the metrics to compare, the comparison method (Prometheus queries, Datadog metrics, custom HTTP checks), and the thresholds. The operator handles the traffic splitting, metric collection, comparison, and promotion/rollback. Spinnaker's automated canary analysis (ACA) provides similar capability for non-Kubernetes environments.

The second half of this L4 item is provenance. When deployment is automatic, the deployment record is the only place the origin of a change is ever written down, and it has to be trustworthy. That means machine-enforced disclosure of AI assistance rather than a convention people are asked to follow. The Linux kernel's 7.2 cycle is the instructive case: LWN counted 1,111 commits carrying Assisted-by tags on 2026-08-17, just under 7% of the cycle and up from 301 in 7.1, with 252 patches crediting the Sashiko LLM review system - and at least one maintainer strips the tags on the way in. A disclosure convention with a single non-participant is not a dataset, it is an anecdote. If the tag matters, a CI check has to require it and a merge has to fail without it.

Sourcegraph made the enterprise version of this argument on 2026-07-27: the blocker to agent adoption in regulated environments is not capability, it is the ability to prove which files the agent read. Provenance that answers "which model, which prompt, which context, which human accepted it" is what turns a canary record into an audit record.

At L4, progressive deployment is the default path for all production changes. The deployment doesn't go to 100% of traffic immediately - it starts at 5-10%, waits for the analysis window (typically 10-15 minutes), and advances to 25%, 50%, then 100% if each phase passes. This staged approach limits blast radius for every deployment while maintaining deployment frequency.

• Limits blast radius for every deployment - at 50+ PRs/day, some percentage of deployments will introduce regressions; progressive rollout limits each regression to 5-10% of users for the first 10-15 minutes, dramatically reducing incident scope
• Eliminates manual canary watching - manually monitoring a canary at high deployment frequency is a full-time job; automated analysis removes this overhead while providing more rigorous and consistent evaluation than human monitoring
• Enables safe auto-merge → auto-deploy - the green = auto-merge → auto-deploy pattern is only safe with automated progressive deployment as the deployment strategy; full-traffic auto-deploy without canary is reckless at L4 volume
• Generates deployment performance data - every canary analysis run produces a record of which metrics were compared, what values were observed, and what the decision was; this data makes deployment patterns visible and improvable
• Provenance turns a deploy log into an audit record - when nothing but automation touches the path from merge to production, the deployment record is the only surviving evidence of where a change came from; enforced disclosure is what keeps it complete
• Supports agent-driven development at scale - agents that produce PRs need deployment infrastructure that can handle their output safely; automated canary is the deployment safety net that makes autonomous agent workflows viable

1. Instrument your key metrics in Prometheus or Datadog - canary analysis requires observable metrics. At minimum: HTTP error rate (5xx responses), p50/p95/p99 latency, and 1-3 business-level metrics (checkout conversion, login success rate, API success rate). These must be queryable by the canary analysis tool.
2. Install Flagger or Argo Rollouts - if using Kubernetes, Flagger integrates with your service mesh (Istio, Linkerd) or ingress (NGINX, Traefik) for traffic splitting. Install Flagger with your preferred provider and configure it for your cluster.
3. Define a canary analysis template - create a Flagger MetricTemplate or Argo Rollouts AnalysisTemplate that defines: which metrics to compare, what thresholds indicate regression, and how long the analysis window should be. Start with error rate (fail if canary error rate > 2x baseline) and latency (fail if p99 latency > 20% above baseline).
4. Configure your first canary rollout - annotate one service for canary deployment. Start with a conservative schedule: 5% traffic for 10 minutes, then 25% for 10 minutes, then 50% for 10 minutes, then 100%. Use maxSurge and maxUnavailable settings to control pod counts during rollout.
5. Test rollback explicitly - deploy a known-bad version (one that generates 5xx errors at a 5% rate) and verify that Flagger detects the error rate increase and automatically rolls back within the configured window. Rollback must be tested before it's needed.
6. Enforce the AI-assistance disclosure in CI, not in the contributing guide - add a check that requires the disclosure trailer (Assisted-by: or your equivalent) on any commit whose diff was agent-produced, fail the merge without it, and carry the tag through into the deployment record so a canary promotion can be traced back to its origin. A convention that one maintainer can silently strip produces numbers you cannot use.
7. Carry the provenance into the rollback path too - when an automated rollback fires, the incident record should show the provenance of the change that triggered it. That is the data that tells you, months later, whether agent-authored changes roll back more often than human-authored ones.
8. Add deployment events to your team dashboard - canary promotions, rollbacks, and in-progress deployments should be visible in a shared dashboard. "Canary at 25%, metrics nominal" tells the team that an auto-deploy is in progress without requiring anyone to watch it actively.

Using canary analysis with insufficient traffic. Canary analysis comparing 5% traffic to 95% baseline requires statistically significant traffic volume. If your service handles 100 requests/hour, 5% is 5 requests/hour - not enough to detect a 2% error rate increase with confidence. For low-traffic services, use blue-green deployment (100% traffic switch with instant rollback) rather than canary, or use a minimum request count threshold before canary promotion.

Analysis templates that are too strict or too lenient. A template that fails on any 1% error rate increase will cause constant rollbacks from normal production noise. A template that requires 10% error rate increase won't catch real regressions. Calibrate thresholds by analyzing your current production metric variance: a "failure" threshold should be outside normal variance by 2-3 standard deviations. Start lenient and tighten over time.

Not accounting for time-based traffic patterns. A canary deployed at 3am Sunday sees different traffic (and different baseline behavior) than one deployed at 2pm Tuesday. Some analysis templates use fixed numeric thresholds that become inappropriate at low-traffic times. Use relative thresholds (canary error rate vs. concurrent baseline error rate) rather than absolute thresholds to handle traffic pattern variance.

Canary that can't actually roll back. Some services have one-way state changes: database migrations, event streams, feature flags that once flipped can't be flipped back cleanly. Canary + automated rollback only works for stateless deployments or deployments with backward-compatible state changes. For breaking changes, progressive deployment must be decoupled from feature activation (feature flags) and use a blue-green strategy instead.

Treating a valid attestation as evidence the change is safe. The ChainDrop compromise of the keyv npm ecosystem on 2026-08-04 is the clearest lesson of the month on this point: the malicious tarballs carried valid npm provenance attestations, because the maintainer's account had been taken over and the project's own Actions pipeline built and published the payload - 444 packages and 2,212 versions in under four hours. The attestation was correct; the source was malware. Provenance tells you where a change came from, not whether it should be there. Do not let a green provenance check substitute for the canary metrics.

Disclosure by convention rather than by check. Every organisation that has tried voluntary AI-assistance tagging has found the same thing the kernel did: participation is partial, and the non-participants are invisible in the data. If you intend to answer questions like "do agent-authored changes trigger more rollbacks", the tag has to be a merge requirement, applied uniformly, from the day you start counting.

Forgetting about canary in your monitoring. When a canary is running, your monitoring shows mixed signals: some users are on the new version, some are on the old. Alerts that fire on aggregate error rate can be triggered by canary issues that are already being handled by rollback automation. Ensure your monitoring is canary-aware: distinguish between "canary pod errors" (handled by automation) and "production pod errors" (requires human response).