Capability 04 of 04 · Infrastructure
Observability & Feedback Loop
Monitoring agent behavior, costs, and outcomes to close the improvement loop.
5
maturity levels
16
practices
16
guides
L1 · Stage 01Ad-hoc
Practices - what it looks like3 guides
- Basic loggingBasic logging is the first and most primitive form of production visibility: writing text output to stdout, stderr, or a log file so that when something goes wrong you have some reguide→
- Alerting on errorsAlerting on errors is the practice of automatically notifying a human when something goes wrong in production - before a customer reports it.guide→
- Prod feedback and token-cost visibility not yet wired to dev"No connection: prod to dev feedback" describes the state where production incidents have no automatic path back to the developer or agent that caused them.guide→
Criteria - what to measure
- 01Basic application logging exists
- 02Alerting fires on application errors
- 03Logs are searchable (centralized logging, not just local files)
- 04Production issues do not yet feed back into dev priorities
L2 · Stage 02Guided
Practices - what it looks like3 guides
- Structured loggingStructured logging replaces free-form text log output with machine-parseable records - typically JSON - where every field has a defined name and type.guide→
- OpenTelemetry basicOpenTelemetry (OTel) is the open standard for collecting and exporting telemetry data - traces, metrics, and logs - from distributed systems.guide→
- Post-deploy monitoring; per-session token cost (ccusage, /usage, /context) as table stakes; local token-compression proxies (run on-device, prompts never leave the machine; 60-95% cost cuts)Post-deploy monitoring is the practice of actively watching key production metrics in the minutes and hours after a deployment, with the goal of detecting deployment-induced regresguide→
Criteria - what to measure
- 01Structured logging is implemented (JSON logs with consistent fields)
- 02OpenTelemetry basic instrumentation is deployed (traces and metrics)
- 03Post-deploy monitoring checks run after each deployment
- 04Traces are correlated across services
- 05Post-deploy checks include automated smoke tests
L3 · Stage 03Systematic
Practices - what it looks like3 guides
- Full observability stack (OTel + Grafana)A full observability stack means having all three telemetry pillars - metrics, traces, and logs - collected, correlated, and queryable in a unified system.guide→
- Production metrics → dashboards; agent telemetry: cost per skill/subagent/plugin/MCP (Claude /usage), mergeability rate (FrontierCode), enterprise usage analytics + spend controls (Anthropic/OpenAI)Production metrics dashboards are the operational nerve center of a mature engineering team: real-time, continuously updated views into the health and behavior of every production service.guide→
- Incident data available for context; Claude-Code-Usage-Monitor live predictions"Incident data available for context" means that when an AI agent or human engineer begins investigating a production issue, all the relevant historical context is immediately acceguide→
Criteria - what to measure
- 01Full observability stack is operational (OpenTelemetry + Grafana/Datadog or equivalent)
- 02Production metrics feed into dashboards accessible to all developers
- 03Incident data (post-mortems, error patterns) is available as agent context
- 04SLOs are defined and tracked for key services
- 05Incident data is structured for machine consumption (not just human-readable post-mortem docs)
L4 · Stage 04OptimizedMost teams aim here
Practices - what it looks like4 guides
- Production anomaly → auto-ticket → agent investigationThe production anomaly to auto-ticket to agent investigation pipeline automates the first phase of incident response.guide→
- Self-healing basic: known patterns auto-fixedSelf-healing for known patterns means that specific, well-understood failure conditions are remediated automatically without human intervention.guide→
- Vercel SDI model: infra recommends code changesThe Vercel Software-Defined Infrastructure (SDI) model describes a paradigm where infrastructure does not just run code - it actively analyzes production behavior and surfaces specguide→
- Session audit pattern (Stella Laurenzo, 6,852 sessions / 234k tool calls) - regression detection on agent quality timelineThe Vercel Software-Defined Infrastructure (SDI) model describes a paradigm where infrastructure does not just run code - it actively analyzes production behavior and surfaces specguide→
Criteria - what to measure
- 01Production anomaly detection auto-creates tickets and triggers agent investigation
- 02Self-healing for known patterns: agent detects known error pattern, applies known fix, deploys, and verifies
- 03Infrastructure recommends code changes based on production data (Vercel SDI model)
- 04Auto-created tickets include full context (traces, logs, affected users, similar past incidents)
- 05Self-healing success rate is tracked (% of auto-fixes that resolve the issue without human intervention)
L5 · Stage 05Autonomous
Practices - what it looks like3 guides
- Full production → agent loopThe full production-to-agent loop is the L5 realization of observability as an agent input channel.guide→
- Anomaly → investigate → fix → test → deploy autonomousThe anomaly-to-deploy autonomous pipeline is end-to-end automated incident response: from the moment a production anomaly is detected to the moment a fix is deployed and verified iguide→
- Infrastructure self-drives: code defines infra, production informs code"Infrastructure self-drives" describes the fully realized bidirectional relationship between code and infrastructure at L5.guide→
Criteria - what to measure
- 01Full production-to-agent loop operates autonomously: anomaly detected, investigated, fixed, tested, deployed
- 02Infrastructure self-drives: code defines infrastructure, production performance informs code changes
- 03Anomaly-to-deploy cycle completes without human intervention for 80%+ of known issue categories
- 04Novel anomalies (not matching known patterns) are escalated to humans with full investigation context
- 05Mean time from anomaly detection to autonomous fix deployment is under 15 minutes
From the Field
Recent releases, projects and discussions the AI Radar classified into this capability.
Which level is your team at in Observability & Feedback Loop?
The criteria above are what we check in a facilitated assessment. Run it yourself to place this capability, see which gates you have passed, and compare it against the other 3 in Infrastructure.
Start the assessment →