Agent iterates tests to green in sandbox (doesn't block team CI)
AI agents fix failing tests in isolated sandboxes - running their own private CI loop - so agent work-in-progress never pollutes the shared pipeline or slows the team.
- ·A failing test reliably indicates a real defect (oracle false-positives are rare)
- ·Agents iterate tests to green in isolated sandbox CI without blocking team CI queue
- ·Mutation testing validates that tests catch real defects (not just achieve coverage)
- ·Sandbox CI iteration count per PR is tracked (ITS target: 1-3)
- ·Mutation testing kill rate exceeds 80%
Evidence
- ·Oracle-reliability dashboard (e.g., TORS) with per-service breakdown
- ·Sandbox CI logs showing agent iteration cycles separate from team CI
- ·Mutation testing reports showing kill rate and surviving mutants
What It Is
When an AI agent's code changes cause test failures, the naive approach is to let the agent iterate in the shared CI pipeline: submit, fail, fix, resubmit, fail again, fix again. This works for a single agent, but it creates serious problems at scale. Each iteration cycle occupies CI resources, clutters the PR history, and creates noise in the shared build queue. With multiple agents running simultaneously - the L4 pattern of 3-5 agents per developer - the shared CI pipeline becomes a bottleneck.
Agent sandbox iteration solves this by giving each agent its own isolated CI environment. The agent makes code changes, runs tests in its sandbox, observes the results, makes corrections, and repeats - all without touching the shared pipeline. Only when all tests pass in the sandbox does the agent submit its PR to the shared CI for final validation and merge. The shared pipeline sees only finished, green work - never in-progress iterations.
The sandbox is not a stripped-down environment. It runs the same tests as the shared CI: unit tests, integration tests, the acceptance test suite. It has access to the same test fixtures, the same environment configuration, the same database schemas. What makes it a sandbox is isolation: each agent's environment is independent from every other agent's environment, and from the shared team pipeline. Failures in one sandbox don't affect another.
At Level 4 (Optimized), sandbox isolation is the technical prerequisite for running multiple agents in parallel. Without it, parallel agents interfere with each other through shared CI state, competing for resources, and generating noise in the shared build system that humans have to triage.
Why It Matters
Sandbox CI isolation is the infrastructure that enables the parallel agent model:
- No CI pollution - The shared pipeline only sees PR-ready code. Developers are not interrupted by agent work-in-progress failures. Build queues are not congested by iterating agents.
- Parallel agent execution - Three to five agents per developer running simultaneously each need independent CI feedback. Sandbox isolation makes this feasible without proportional infrastructure scaling.
- Agent iteration speed - In a shared pipeline with queue wait times, each agent iteration might take 15-20 minutes including wait. In a dedicated sandbox, the same iteration takes 3-5 minutes. Agents reach green 3-5x faster.
- Failure attribution clarity - When a test fails in the shared pipeline, it may not be clear whether it was caused by the most recent agent, a previous agent, or a human developer's change. In sandboxed agents, every failure is clearly attributable.
- Cost predictability - Sandbox CI for multiple agents requires infrastructure investment, but the cost is predictable and proportional to agent count. Uncontrolled shared pipeline congestion is unpredictable and grows super-linearly.
Design sandboxes to be ephemeral - created on agent start, destroyed on completion. Ephemeral sandboxes prevent state accumulation between agent runs, which is one of the leading causes of "works in sandbox, fails in shared CI" surprises. Use container orchestration (Kubernetes Jobs, Fargate Tasks, or a CI platform with ephemeral environments) to manage sandbox lifecycle.
Getting Started
- Choose your sandbox infrastructure - The sandbox needs to run the same test suite as shared CI. Options: ephemeral containers per agent (Docker + CI runner), cloud-based ephemeral environments (GitHub Actions with isolated runners, Buildkite Elastic CI Stack), or a dedicated CI cluster for agents. Match the choice to your scale: one or two agents can share a container; dozens need a cluster.
- Replicate the CI configuration - The sandbox configuration should be a replica of the shared CI configuration, minus the notification and merge-gating steps. If the shared CI runs lint, type check, unit tests, and integration tests - so does the sandbox. Discrepancies between sandbox and shared CI environments are a leading cause of "passed in sandbox, failed in CI" bugs.
- Give agents sandbox management capabilities - The agent needs to be able to: start a sandbox run, observe the results, make code changes, and trigger another sandbox run. This typically means the agent has access to the CI API (GitHub Actions API, Buildkite API) and can trigger runs programmatically.
- Define the sandbox-to-shared-CI handoff protocol - Specify the condition under which an agent submits from sandbox to shared CI. The simplest: all tests green in sandbox. More sophisticated: green tests plus lint, plus type check, plus coverage threshold. The handoff condition should match your shared CI merge requirements exactly.
- Implement sandbox time limits - An agent iterating indefinitely in a sandbox wastes infrastructure and may be stuck. Set a maximum iteration count (10-15 iterations is typical) and a maximum wall-clock time (30-60 minutes). When limits are reached, the agent should escalate to a human rather than continuing to iterate.
- Monitor sandbox success rates - Track what percentage of agents successfully reach green in sandbox before submitting to shared CI. A healthy rate is 85%+. Lower rates indicate the agent is working on problems that require more context or human intervention than the sandbox workflow provides.
Common Pitfalls
Sandbox environment drift. The most common failure mode: the sandbox environment diverges from the shared CI environment over time (different dependency versions, different environment variables, different database schemas). Tests pass in sandbox and fail in shared CI. Prevent this by treating the sandbox configuration as a derived artifact of the shared CI configuration, not as independently maintained.
State leakage between sandbox runs. Agents that reuse sandbox environments between iterations may accumulate state from previous runs: cached files, database records from previous test runs, leftover temporary files. Ephemeral sandboxes (created fresh for each iteration) prevent this. If reuse is necessary for performance, implement explicit state reset between iterations.
No cost controls on iteration depth. Without limits, a stuck agent will iterate indefinitely, consuming infrastructure. Set hard limits: maximum iterations, maximum wall-clock time, maximum test failure count before escalation. The limits should be calibrated based on the complexity of typical tasks - not so tight that successful agents are cut off, not so loose that stuck agents burn resources.
Treating sandbox success as sufficient quality gate. Sandbox success is necessary but not sufficient for merge. The shared CI pipeline catches environment-specific issues, integration issues across services, and merge conflicts with concurrent changes. Sandbox iteration reduces PR-to-merge cycle time but does not replace shared CI validation.
How Different Roles See It
Bob's team has deployed three agents working in parallel, but shared CI is now congested with agent iterations. Developers are waiting 25 minutes for their own PRs to get CI feedback because agent jobs fill the queue. The team is frustrated and starting to question whether the agents are worth it.
What Bob should do: This is exactly the problem sandbox isolation solves. Bob should treat CI queue congestion as a blocker for continued agent deployment - not just an inconvenience. The solution is architectural: agents get dedicated infrastructure or isolated CI queues, and the shared pipeline is reserved for human PRs and final agent PR validation. Bob should also set a policy: no agent PR is submitted to shared CI until it's green in sandbox. This reduces shared CI load because agents aren't iterating in the shared queue - they arrive with passing tests.
Sarah has been tracking developer satisfaction and has seen a dip since agents were deployed. Developers feel like CI has gotten slower and less responsive. She expected agents to improve productivity, not degrade it.
What Sarah should do: The satisfaction dip is attributable to CI contention, not to the agents themselves. Sarah should decompose CI wait time into agent-caused and human-caused components. If agent iterations are crowding the queue, sandbox isolation will directly address the developer satisfaction metric. Sarah should frame this as a necessary infrastructure investment to enable agents at scale: the agents themselves are productive, but they need dedicated infrastructure to not degrade the human developer experience. Present the before-and-after CI wait time as the key metric for the investment.
Victor architected the initial agent deployment and is now being asked to fix the CI congestion problem. He has three options: scale CI infrastructure (expensive), throttle agent concurrency (defeats the purpose), or implement sandbox isolation (correct but complex).
What Victor should do: Victor should implement sandbox isolation - it's the architecturally correct solution and the one that scales. He should design it as ephemeral containers managed by a CI platform's dynamic runner feature (GitHub Actions' larger runners, Buildkite's elastic stack), so that sandbox capacity scales automatically with agent count without manual infrastructure management. Victor should also add the sandbox success rate metric to the monitoring dashboard so he can observe whether agents are consistently reaching green before submitting, or whether they're regularly hitting time limits and escalating.
Further Reading
From the Field
Recent releases, projects, and discussions relevant to this maturity level.
Where does your team actually sit on this?
This guide describes one level of one area. Run the assessment to place your team across all 16 areas, see which gates you have passed, and get a report you can take to your stakeholders.