Production logs → auto-generated regression tests

When a production error occurs, an agent captures the input that triggered the failure, isolates the code path that was traversed, generates a regression test that reproduces the bug, and submits a PR that both adds the test and fixes the underlying defect - all without human involvement in the cycle. Production logs become an automatic source of test generation: every real failure becomes a permanent entry in the test suite.

The technical chain that makes this work starts with structured production logging. When an error occurs, the log captures not just the exception but the request context, the relevant application state, and the code path traversed. The agent receives this event, extracts the inputs (request parameters, database state, configuration values at the time of failure), and constructs a minimal reproduction case. It writes a failing test that uses those inputs and expects the correct behavior. It then implements the fix, verifies the test passes, and verifies no other tests break.

This is fundamentally different from a human debugging workflow, where a developer reads logs, mentally reconstructs the scenario, writes a reproduction script, and eventually commits a test. The agent's workflow is faster (minutes vs. hours), more precise (uses actual production inputs rather than approximations), and more complete (every production error generates a test, not just the ones developers have time to address).

At Level 5 (Autonomous), production-to-regression-test pipelines are continuously active. The test suite grows in proportion to production error volume. Over time, the test coverage map looks like a heat map of actual production usage patterns - the paths that users actually traverse are the ones with the deepest test coverage.

Production-derived regression tests close the loop between real-world usage and test coverage in ways that no amount of upfront test design can achieve:

• Tests from real inputs, not hypothetical inputs - Human-written tests are based on what developers imagine users will do. Production-derived tests are based on what users actually do. The coverage gap between imagination and reality is often significant.
• Zero time-to-test for regressions - In traditional workflows, a production bug might be fixed in days but the regression test gets written weeks later, if ever. In L5, the regression test exists in the PR that fixes the bug. No regression survives to be forgotten.
• Closed-loop quality - The system improves continuously from its own failures. Each production error strengthens the test suite. The cost of a bug decreases over time because the category of bug that reached production once is caught before it can reach production again.
• Coverage of edge cases humans wouldn't think to test - Production logs reveal edge cases that no test author would invent: specific combinations of user data, unusual request sequences, timing conditions that only occur under production load. These become first-class test cases.
• Audit trail for production incidents - Every production error has a corresponding test that reproduces it. The test suite is also an incident record: you can find when any given class of bug first appeared, how it was fixed, and that it hasn't recurred.

1. Instrument production error logging for regression capture - Add structured context to every unhandled exception: request parameters, user ID (anonymized or hashed), relevant application state at time of failure, stack trace with variable values. The agent needs enough information to reconstruct the scenario that caused the failure.
2. Build the error triage pipeline - Not every production error should automatically generate a regression test. Some errors are expected (rate limit responses, user-caused 404s), some are transient infrastructure issues, and some are genuine application bugs. Build a triage layer that routes genuine application bugs to the regression generation agent.
3. Implement the reproduction case extractor - Given a structured error event, the agent must: identify the failing code path, extract the minimal inputs required to reproduce the failure, and construct a test that will fail before the fix and pass after it. This is the most technically complex part of the pipeline.
4. Connect to the fix-and-submit workflow - The regression test generation should chain directly into the bug fix workflow: agent generates the failing test, agent diagnoses the root cause, agent implements the fix, agent verifies the test now passes, agent submits the PR. The full loop should complete without human handoffs.
5. Validate generated tests in staging - Before committing production-derived tests to the main suite, validate them in a staging environment using production data: do they reliably reproduce the failure? Do they pass after the fix? Does the fix introduce any new failures?
6. Monitor regression prevention effectiveness - Track: how many regression tests have been generated from production errors, how many production errors were caught by previously-generated regression tests before reaching production again, and the time from production error to regression test in the suite. The last metric is your primary quality indicator: it should be minutes, not days.

Using PII in regression tests. Production logs contain real user data. Regression tests generated from production logs will naturally include that data as test inputs. You must anonymize or synthesize realistic equivalents of any PII before tests are written to the repository. This is not optional - it's a legal and security requirement. Build anonymization into the extraction pipeline, not as an afterthought.

Generating tests for transient infrastructure failures. Not every production error is a code bug. Timeouts, connection failures, and resource exhaustion errors can produce error logs that look like bugs but aren't. If the regression pipeline generates tests for infrastructure failures, the test suite will contain unreliable tests that pass and fail based on environment conditions. The triage layer must distinguish application bugs from infrastructure failures.

No feedback loop on test quality. Production-derived tests can be low-quality: too tightly coupled to specific data values, testing too narrow a path to catch related bugs, or using unstable oracles. Implement mutation testing on production-derived tests (as in the L4 mutation testing workflow) to validate that they provide meaningful coverage.

Over-engineering the pipeline before proving value. Building a fully automated production-to-regression-test pipeline is a significant engineering investment. Start with a semi-automated version: the agent generates the test and fix candidate, but a human reviews and merges. Validate that the generated tests are high quality before fully automating the submission. The automation adds value when the quality is already proven.