Flaky test quarantine

Flaky test quarantine is a structured process for handling tests that fail intermittently: when a test is identified as flaky (fails without code changes, or fails on some CI runs but passes on others), it is immediately moved to a quarantine suite that runs separately from the main CI pipeline. Quarantined tests don't block merges, don't trigger alerts, and don't pollute the CI signal. But they are tracked, reported on a schedule, and assigned to engineers for resolution.

At Level 1, the response to flaky tests is ad-hoc: developers retry CI runs, add @Ignore annotations, or just accept intermittent failures as background noise. There's no process, no ownership, and no resolution timeline. The flakiness accumulates until it consumes a significant fraction of engineering time (as Google's research documented - up to 16%).

Quarantine is the L2 answer. It solves two problems simultaneously. First, it removes flaky tests from the primary CI signal, making the build reliable again - a passing build now means something. Second, it creates explicit accountability: quarantined tests are not "ignored" tests. They are tests on a repair schedule. The quarantine suite runs nightly and reports to a dashboard. Engineers are assigned to fix them. The quarantine is a holding area, not a permanent state.

The key distinction is between quarantine and deletion. Teams that simply delete flaky tests lose the coverage they provided. Teams that quarantine them preserve the intent (this behavior should be tested) while temporarily suspending the signal (this specific test is unreliable) until the root cause is fixed.

Flaky test quarantine is the bridge between the chaos of L1 and the reliable test infrastructure required at L3-L5:

• Restores CI trust - Once developers can trust that a red build means a real failure, they stop dismissing failures as noise. This is foundational to everything else.
• Preserves coverage intent - Quarantined tests remain in the codebase, documented, assigned, and scheduled for repair. The behavior they tested is not forgotten.
• Makes flakiness visible - The quarantine queue is a metric: how many tests are quarantined, for how long, and in which areas of the codebase. This makes the problem prioritizable.
• Enables agent reliability - AI agents that iterate on failing tests need a reliable CI signal. An agent in a high-flakiness environment thrashes on phantom failures. Quarantine gives agents a clean signal to work with.
• Prerequisite for TORS - The Test Oracle Reliability Score (TORS) target of 90%+ at L3 is only achievable after systematic quarantine has cleared the backlog of known flaky tests.

1. Create the quarantine infrastructure - Add a quarantine test suite to your CI configuration. In most CI systems, this means a separate job that runs the quarantine suite on a schedule (nightly is typical) rather than on every push. The output goes to a dashboard but does not gate merges.
2. Define the quarantine criteria - A test should be quarantined when it fails on a commit where no related code was changed, or when it has a documented history of intermittent failures. Write this criteria down. You want quarantine decisions to be objective, not judgment calls.
3. Build the detection pipeline - Automatically detect candidates for quarantine by tracking test results over the last N runs. A test that fails more than 5% of the time without a corresponding code change is a quarantine candidate. Some CI platforms (BuildKite, Gradle Enterprise) have built-in flakiness detection.
4. Create the ownership model - Assign quarantined tests to the team that owns the code they test. The owning team's tech lead reviews the quarantine queue weekly. Unowned tests go to a shared backlog with an on-call rotation.
5. Set the fix SLA and track it - Two weeks is a reasonable default. Publish the quarantine queue and SLA status in your engineering metrics dashboard. When tests age out of the SLA, escalate to team leads.
6. Implement the three resolutions - A quarantined test has three outcomes: fix the test (address the root cause), rewrite the test (replace it with a stable equivalent), or delete it (if the behavior is covered by other tests or the test was not worth keeping). No other outcomes are acceptable.

Quarantine as permanent storage. Without a SLA and active tracking, quarantine becomes a graveyard of tests that no one will ever fix. The metric to watch: average days-in-quarantine. If it's rising, quarantine has become deletion with extra steps.

Quarantining too eagerly. Some teams interpret "quarantine flaky tests immediately" as "quarantine on first failure." A single failure doesn't indicate flakiness - it might indicate a real bug. The criterion should be a pattern: two or more failures on commits with no related code changes, or a documented history of intermittent failures.

Fixing flakiness with retries instead of root cause. It's tempting to "fix" a quarantined test by wrapping it in a retry loop and moving it back to the main suite. This passes the quarantine SLA while preserving the underlying non-determinism. Retries are acceptable as a short-term stabilizer while root cause investigation proceeds, but they are not a resolution.

Not tracking the quarantine backlog. The value of quarantine as a process comes from visibility. If the quarantine queue isn't tracked, reported, and reviewed regularly, it provides no more accountability than the L1 @Ignore annotation. The quarantine process is only as effective as the management attention it receives.