TORS > 90% (Test Oracle Reliability Score)

The Test Oracle Reliability Score (TORS) measures the signal-to-noise ratio of your test suite: what percentage of test failures indicate a genuine defect in the code vs. false positives caused by flaky tests, environmental issues, or unstable assertions. A TORS of 90% means that 9 out of 10 test failures are worth investigating because they represent real problems. A TORS of 70% means that nearly one in three failures is noise - and developers learn to ignore CI output.

TORS is calculated by tracking test failure investigations: when a test fails, does it indicate a real bug (signal) or not (noise)? Over time, the ratio of signal failures to total failures is your TORS. Some CI platforms (Gradle Enterprise, BuildKite) automate this tracking. Teams without tooling support can approximate it by tracking re-runs: if a test failure disappears on re-run without code changes, it was noise. Re-run rate is a proxy for TORS.

At Level 1, TORS is typically below 70% due to accumulated flakiness and unstable oracles. At Level 2, quarantine and oracle stabilization push it toward 80-85%. At Level 3 (Systematic), TORS > 90% is a formal target - not an aspiration but a requirement. Automated quality gates (L4's green auto-merge) are architecturally impossible with TORS below 90%, because the automated system would block good code or pass bad code too frequently to be trusted.

The 90% threshold is meaningful. At 90% TORS, automated systems can make decisions with confidence: a failing build is almost certainly a real problem. At 80%, one in five failures is noise - enough to erode trust in automation. At 90%, the failure rate of the automation itself is low enough that it provides more value than friction.

What August added is a warning about which number you are trusting. The oracle question is not only "do failures mean something" but "where does the expected result come from". Tests written against what the code currently does are circular however reliably they run, which is why at L3 the oracle moves to requirements - tickets and specs - rather than the implementation. Two findings sharpen this. DECODE (arXiv 2607.25130, 53.6K real in-IDE edits from over 1,000 developers) found that 31% of accepted AI completions are deleted entirely, most of them edited within fifteen minutes: acceptance rate is a vanity metric, and so is any signal measured at the moment of generation rather than after the code survives contact with the suite. SlopCodeBench, published the same week, found agent-built projects running to roughly 29,000 source lines of which 51% was test code, with complexity and duplication rising at every checkpoint for every model. Test-suite mass is a warning sign, not a maturity signal. TORS earns its place precisely because it measures whether the suite tells you something, which volume never does.

The other half of this level is a negative instruction: do not mandate the test process to agents, measure the outcome instead. Birgitta Böckeler's "TDD inside the agent loop" (August 11) ran five batches of tasks blind-judged by Opus and found non-TDD solutions consistently taking the top spots on small and medium tasks while TDD landed in the bottom two, at three to eight times more tokens. Her mechanism is plausible: an agent told to write the test first makes locally-minimal decisions around that first test and hardens a bad design early, where an unconstrained agent plans the data model, edge cases and contracts up front. Her reading is that TDD's real benefits - fear management, forced pacing, permission to relax - are human benefits that do not transfer to something incapable of fear. Carry the caveats, because she does: five batches, greenfield business logic only, quality judged largely by a model, and no run achieved perfect TDD adherence. It is a provocation, not a settled finding. But her recommendation points where this level already pointed - stop mandating the process, monitor outcomes through mutation testing and static analysis. Where an oracle supplied up front genuinely pays is security: SecTDD (arXiv 2608.09740, August 10) found that giving the agent your security tests before it starts is worth roughly 19 points of joint functional-plus-security success.

TORS at 90% is the inflection point where the test suite transitions from a human-managed tool to an automation-ready infrastructure component:

• Automated decisions become viable - At 90% TORS, you can begin moving toward automated merge policies. False positive rates are low enough that automation provides net value.
• Agent iteration becomes reliable - AI agents fixing test failures need to know that failures indicate real problems. At 90% TORS, agents can trust the CI signal and make focused fixes rather than thrashing.
• Developer trust is restored - When 9 out of 10 failures are real, developers stop dismissing CI output. The psychological shift from "CI is unreliable" to "CI is worth trusting" is critical for effective AI-assisted development.
• Acceptance test coverage becomes meaningful - Acceptance tests derived from requirements (L3) only provide value if their failures are investigated. At L1-L2 TORS levels, even real failures get dismissed alongside noise.
• Foundation for L4 automation - The 95% TORS target at L4 is only reachable if the team has established the practices and metrics to manage TORS systematically at L3.

1. Instrument CI for TORS measurement - The minimum viable instrumentation: track which test failures led to code changes vs. which were resolved by re-running. A CI platform with test analytics (Gradle Enterprise, BuildKite, DataDog CI Visibility) can automate this. Manual tracking with a spreadsheet works for teams that need to start quickly.
2. Establish the baseline TORS - Run the measurement for 2-4 weeks before setting targets. Your actual baseline is often lower than expected - teams with high flakiness frequently discover TORS below 60% when they measure for the first time.
3. Set the 90% target and timeline - 90% TORS within one quarter is achievable for most teams that have already implemented quarantine and oracle stabilization. Publish the target and current score in your engineering metrics dashboard.
4. Work the remaining false positives - Below 90%, identify the top 5 tests by false positive frequency. Fix them. Repeat. The Pareto principle applies strongly here: a small number of tests usually account for the majority of false positives.
5. Automate false positive detection - Configure your CI system to flag test failures that match known flaky patterns. When a test fails that has a history of false positives, automatically route it to the quarantine queue rather than blocking the build.
6. Validate TORS after each major change - Significant codebase changes (large refactors, infrastructure upgrades, dependency updates) can degrade TORS. Measure it explicitly after major changes and restore it before the next sprint.

Measuring TORS only on failures, not on passes. TORS measures the reliability of failures - but there's an analogous problem with false negatives: tests that pass when they should fail (circular tests, tests with weakened oracles). A comprehensive quality metric includes both dimensions. TORS alone doesn't catch tests that are reliably wrong.

Gaming TORS by deleting flaky tests. Deleting all tests that produce false positives will mathematically raise TORS to 100% while destroying test coverage. Any TORS measurement should be paired with a coverage metric to ensure that TORS improvements reflect genuine quality improvements, not coverage reductions.

Setting TORS as a team-level metric rather than a codebase metric. Different areas of the codebase may have very different TORS profiles. A monorepo with 92% aggregate TORS may have one service at 60% dragging down the rest. Track TORS by service or module to surface these anomalies.

Treating 90% as a destination rather than a floor. At L3, 90% TORS is the minimum - the floor that enables systematic automation. It is not the ceiling. Teams should continue improving TORS as they progress to L4 (where the target rises to 95%). Treating 90% as "good enough" leads to stagnation.