AI tests circularly test what code DOES not what it SHOULD do

When you ask an AI to generate tests for existing code at Level 1, it does the natural thing: it reads the implementation and writes tests that verify what the code currently does. If your calculateDiscount() function returns 15% for platinum customers when it should return 20%, the AI will write a test that asserts the result is 15%. The test passes. The bug is hidden. Your coverage number goes up. You feel safer. You aren't.

This is the circular testing problem. A test is only valuable if it can fail when something is wrong. A test that was written by observing the code and asserting its current outputs can only fail if the code changes - it cannot fail because the code was always wrong. The AI doesn't know what calculateDiscount() should return for platinum customers. It only knows what it currently returns. So it tests the current behavior, including the bug.

At L1, this is the dominant mode of AI-assisted test generation because it's the easiest workflow: write code, ask AI to write tests, get coverage. The result is a test suite that looks healthy but is structurally circular. It guards against regressions from future changes, but it provides zero verification that the original implementation was correct. Every bug present at the time the tests were generated is permanently enshrined in the test suite as "expected behavior."

The fix requires separating the source of truth for test expectations from the implementation. Acceptance tests derived from requirements (tickets, specifications, user stories) cannot be circular because they were written before the implementation existed. This is why the maturity matrix progression moves toward requirements-derived tests (L2-L3) and eventually toward autonomous requirements generation (L3).

Circular tests are worse than no tests in one specific way: they provide false confidence. A team with 0% test coverage knows they have a coverage problem. A team with 70% circular coverage believes they're protected when they're not:

• Bug preservation - Every bug present when tests are generated gets encoded as expected behavior. The tests will continue passing until someone notices the behavior is wrong in production.
• Refactoring trap - If you refactor the code to fix a bug, the circular tests will fail - not because you broke something, but because the tests were wrong. Developers learn to distrust test failures, which compounds the flakiness problem.
• AI amplification - As AI generates more code faster, circular test generation at scale can create an enormous volume of tests that appear to provide coverage while actually encoding AI-generated bugs as correct.
• Review blind spot - It's difficult to detect circular testing in code review because the assertions look correct - they match the implementation. You need to know what the implementation should do to catch the circularity.
• False L2 metrics - Coverage targets set at L2 can be hit entirely with circular tests. If you're measuring coverage without auditing test quality, the metric is misleading.

1. Recognize the pattern - Train your team to identify circular tests. The telltale sign: expected values in tests that match exactly what the current implementation produces, with no reference to a specification or business rule.
2. Source expected values from requirements - Before writing a function, write down what it should produce for a given input - based on the product spec, the ticket, or the business rule. Use those values as test expectations, not the output of the function itself.
3. Use test-first for high-risk code - For business-critical logic (pricing, permissions, financial calculations), write the test before the implementation. This structurally prevents circular testing because the test is written without knowledge of the implementation.
4. Ask AI for test structure, not test values - Use AI to generate the test scaffolding (test file setup, describe blocks, test names), but provide the expected values yourself. This splits AI's strength (boilerplate) from the part that requires human judgment (correctness).
5. Introduce acceptance test categories - Distinguish between "unit tests" (which verify implementation mechanics and may be AI-generated) and "acceptance tests" (which verify business behavior and must be human-authored or requirements-derived). Keep them in separate directories and track them separately.
6. Review tests with requirements in hand - When reviewing a PR that includes tests, open the original ticket alongside the test file. Ask: does the test verify what the ticket described?

Confusing code coverage with behavior coverage. Coverage tools measure whether lines of code were executed during tests - they say nothing about whether the assertions are correct. A function can have 100% line coverage with circular tests that assert the wrong values. Coverage is a measure of test reach, not test correctness.

Using AI snapshots as test oracles. Some testing workflows use AI to generate "snapshot" tests - capture the current output and assert it doesn't change. This is circular testing by design. Snapshot tests are useful for detecting unintentional changes to UI or serialization formats, but they're dangerous for business logic where the current output may be wrong.

Generating tests after debugging. A common workflow: AI-generated code has a bug, you debug and fix it, then ask AI to generate tests. If the AI generates tests after the fix without being told what was wrong, it will test the fixed behavior - but it won't test the edge case that caused the original bug. You need to explicitly write a test for the bug case, not just re-run AI generation on the fixed code.

Treating test generation as a one-time action. Circular tests become more dangerous over time. As the codebase evolves, circular tests accumulate. Periodically auditing your test suite for circularity - especially in business-critical code - is a maintenance activity, not a one-time task.