Production feedback → CI auto-adjusts test suite

"Production feedback drives CI test suite adjustment" is an L5 pattern where the CI test suite is not a static artifact maintained by engineers but a dynamic system that evolves based on what's actually going wrong in production. When a production incident occurs, the system automatically: identifies the code path that failed, generates a regression test covering that failure mode, adds the test to CI, and ensures that path is covered on every future change that touches the affected code. The test suite grows in response to real failures, not anticipated failures.

The pattern works in the opposite direction too: when production telemetry shows that a code path has never caused a problem and has not been changed in 6 months, the system can flag the tests covering it as candidates for deprioritization in the fast CI path - moving them to a weekly validation suite rather than running them on every commit. The test suite's composition is continuously optimized: more coverage where production failures occur, less coverage of stable code paths that are rarely exercised in production.

This is a natural extension of the production telemetry patterns that already exist in mature engineering organizations (distributed tracing, error tracking, anomaly detection). The new element is the feedback loop: production signals automatically trigger test suite changes rather than requiring a human to analyze an incident, decide to write a regression test, implement it, and add it to CI. At L5, this loop runs continuously and automatically, with human review for the generated tests but no human requirement to initiate the process.

Thoughtworks made this an explicit invariant in its operating model for enterprise AI agent reliability, published 2026-08-14: every production failure becomes a permanent regression test. The word doing the work is "permanent". A regression test that can be deleted by the next agent tidying up a test file is not a control, it is a comment. In the Thoughtworks model each of six layers - terminology, routing, agent intent, semantic context, execution, result - has a named owning team and an executable "truth contract", and the regression suite is where those contracts accumulate. If you are building this loop, decide up front which team owns the generated test and what it takes to remove one.

The mechanism typically involves: error tracking (Sentry, Datadog) that identifies failing code paths in production; an agent that reads the error and the relevant source code and generates a regression test; a CI integration that adds the test to the affected module's test suite; and a review step (which can be automated to skip human review for straightforward regression tests with high confidence). The loop closes when the regression test is added to CI and future changes to the affected code path run against it automatically.

• Test coverage grows where it matters most - tests are added for code paths that fail in production, which is the most reliable signal of where coverage is needed; coverage grows in response to real risk rather than developer intuition
• Regression prevention becomes automatic - every production incident generates a regression test; the same bug cannot ship again without failing CI; the test suite becomes an automatically maintained safety net
• CI test suite stays relevant as codebases evolve - code paths that are never exercised in production and never changed are deprioritized in CI, keeping the test suite proportional to actual risk; the suite doesn't grow unboundedly
• Eliminates "we should write a regression test" backlog items - production incidents generate regression tests immediately and automatically; there's no backlog of "we should have a test for this" items because the system creates them
• Permanence is the whole point - Thoughtworks' operating model treats every production failure as a permanent regression test with a named owning team; a suite that agents can quietly prune is a suite that will regress to the mean
• Demonstrates that CI and production are a continuous loop, not separate stages - production feedback flowing back to CI represents the highest level of CI maturity: the pipeline learns from reality rather than being a static artifact

1. Establish production error tracking with code path attribution - Before auto-generating tests, you need production errors attributed to specific code paths. Sentry, Datadog APM, and Honeycomb all provide stack trace analysis that maps errors to source code locations. Configure your error tracking to capture full stack traces and correlate them with your current deployed version's source map or symbol table.
2. Build a "production failure to test spec" translation layer - Create a process (initially human-run, eventually agent-automated) that takes a production error with its stack trace and converts it into a test specification: what inputs triggered the error, what was the expected behavior, what was the actual behavior. This can start as a structured post-incident template and evolve toward automated generation.
3. Implement an agent-based regression test generator - Use Claude Code or a similar agent to take the test specification (error description, failing code path, relevant context) and generate a regression test in the appropriate test framework. The agent needs access to: the error details, the source code of the failing function, examples of existing tests in the same file as style reference, and the test framework's assertion patterns.
4. Add CI integration for automatically-generated tests - Generated tests should be added to the test file for the failing module with a metadata tag (a comment or test attribute) identifying them as production-feedback-generated. This allows tracking them separately and reviewing them as a group. Configure CI to run these tests with the same priority as manually written tests.
5. Implement test coverage-to-production-path correlation - Use code coverage data (Istanbul/nyc for JavaScript, coverage.py for Python, JaCoCo for Java) correlated with production code paths (from APM traces) to identify over-tested stable code and under-tested risky code. This correlation is the basis for deprioritizing stable tests and prioritizing risky ones.
6. Gate the generated tests on mutation strength, not on their existence - a generated regression test can pass, add a line to the coverage report and assert nothing meaningful. Sławomir Radzymiński's August write-up found one component at 100% line coverage and 61% mutation strength. Apply mutation testing selectively to the high-risk areas the generated tests land in - auth, business rules, payments - and treat the mutation score, not the test count, as the signal the loop is working.
7. Make the generated test permanent and owned - tag it with the incident it came from, assign it to the team that owns the failing layer, and require an explicit justification to delete it. Otherwise the next tidy-up pass removes exactly the tests you paid an outage for.
8. Start with human review of generated tests before automation - For the first 3 months, have a human engineer review each auto-generated regression test before it's added to CI. This validates the quality of the generation and builds the team's confidence in the process. After reviewing 50-100 tests, you'll know the false positive rate well enough to decide which categories of auto-generated tests can be added without review.

Generating tests that test implementation details rather than behavior. Auto-generated tests that directly assert on internal function structure rather than observable behavior will break every time the implementation changes, even when behavior is correct. Validate generated tests against the behavior-specification principle: do they test inputs and outputs, not implementation?

Adding auto-generated tests without fixing the underlying bug. A regression test that fails on the production-reproducing input is valuable. But if it's added to CI without also fixing the bug that caused the production failure, every CI run will fail until someone fixes it. The workflow must be: fix the bug first, then add the regression test to prevent recurrence. Not the other way around.

Deprioritizing tests based on coverage data alone. Coverage data says which code paths are executed by tests, not which code paths are important. A path that has 100% test coverage but is never exercised in production (a rarely-used admin feature, a deprecated API endpoint) may be safe to deprioritize. But a path with 100% test coverage that handles critical production traffic should not be deprioritized even if it's stable. Use production traffic data alongside coverage data for deprioritization decisions.

Not attributing auto-generated tests in version control. Generated tests should be clearly marked as auto-generated in their file header or with test metadata. This allows engineers to understand the test's origin, helps during code review ("this test was auto-generated from production error XYZ"), and enables bulk operations (find all auto-generated tests for a module, review their quality as a batch).

Creating a feedback loop that generates tests faster than the team can maintain them. An organization with many production incidents generating automatic regression tests could accumulate thousands of tests quickly. Monitor test suite growth rate and set a threshold that triggers review: if more than X tests are generated per week, review the generation quality and consider tightening the generation criteria. SlopCodeBench's July 2026 run is the cautionary number here: ~29,000 source lines of which 51% were test code, with complexity and duplication rising at every checkpoint for every model. Test-suite mass is a warning sign, not a maturity signal.

Reading a green auto-generated suite as proof of coverage. Mutation testing is the check that catches this, and it is now the recommended monitor for agent-written code generally - Birgitta Böckeler's August experiment on TDD inside the agent loop ends with exactly that recommendation, having found that agents told to TDD produced worse-judged solutions at 3-8x the tokens. Her result rests on five batches of greenfield business logic judged largely by a model, so carry it as a provocation rather than a settled finding; the mutation-testing advice stands on its own either way.