Capability 03 of 04 · Delivery Management
Metrics
What you measure to understand AI-assisted engineering productivity and quality.
5
maturity levels
16
practices
16
guides
L1 · Stage 01Ad-hoc
Practices - what it looks like3 guides
- DORA metrics, if trackedAt L1 (Ad-hoc), most engineering teams track DORA metrics inconsistently or not at all.guide→
- Standard delivery metrics (not yet AI-specific)At L1, engineering teams that have adopted AI tools - GitHub Copilot, Cursor, Claude Code - are tracking those tools with zero AI-specific metrics.guide→
- ROI of AI not yet measured"How much did we save with AI?" is the question every engineering leader eventually faces from finance, from the CTO, or from the board.guide→
Criteria - what to measure
- 01Delivery is tracked with at least basic metrics
- 02Standard delivery metrics are in place (AI-specific metrics come later)
- 03Team acknowledges the need for AI-specific metrics beyond traditional DORA
- 04Basic deployment frequency is at least known (even if not dashboarded)
L2 · Stage 02Guided
Practices - what it looks like3 guides
- DORA + basic AI tracking; per-session token spend (ccusage, /usage); value-per-token, not raw token count (the efficiency turn)At L2 (Guided), teams have moved past the L1 silence on metrics.guide→
- Licenses vs usage rateLicenses vs. usage rate is the first uncomfortable AI metrics discovery. Teams that invest in AI coding tools - GitHub Copilot, Cursor, Claude Code - routinely find that the numberguide→
- PR throughput per devPR throughput per developer is the first meaningful per-developer productivity metric for AI-assisted development: how many pull requests does a developer merge per week, on averagguide→
Criteria - what to measure
- 01DORA metrics are tracked consistently with a dashboard
- 02AI tool license count vs. active usage rate is measured
- 03PR throughput per developer is tracked
- 04AI acceptance rate (% of AI suggestions accepted) is measured per tool
- 05Metrics are reviewed in team retrospectives at least monthly
L3 · Stage 03Systematic
Practices - what it looks like3 guides
- ITS (Iterations-to-Success): target 1-3Iterations-to-Success (ITS) is an AI-native metric that measures how many CI attempts it takes for an agent's PR to pass.guide→
- CPI (Cost-per-Iteration): target < $0.50; cost-per-merged-PR tracked over time; token arbitrage (judgment on premium model, code-writing on a cheaper specialist)Cost-per-Iteration (CPI) measures what a single agent CI attempt costs - in model API costs, CI compute, and related infrastructure.guide→
- CI Feedback Latency trackingCI Feedback Latency is the time from when an agent pushes a commit to when CI produces a result (pass or fail) that the agent can act on.guide→
Criteria - what to measure
- 01ITS (Iterations-to-Success) is tracked with a target of 1-3
- 02CPI (Cost-per-Iteration) is tracked with a target under $0.50
- 03CI feedback latency is tracked as a metric (time from push to CI result)
- 04Metrics are broken down per team and per repository
- 05Cost tracking includes model API costs, CI compute costs, and runner costs per iteration
L4 · Stage 04OptimizedMost teams aim here
Practices - what it looks like5 guides
- Test-oracle reliability tracked as a metricThe Test Oracle Reliability Score (TORS) measures what percentage of test failures represent real defects rather than test infrastructure issues, environmental flakiness, or poorlyguide→
- Auto-Approve Rate: target > 60% (anchor in mergeability + post-merge outcomes, not pass-rate - FrontierCode: >half of SWE-bench is unmergeable slop)Auto-Approve Rate is the percentage of PRs that merge without requiring human review - passing all automated gates (CI, security scans, coverage checks, linting) and merging algorithmically.guide→
- Agent Autonomy Score: % tasks without human interventionThe Agent Autonomy Score measures the percentage of tasks that an agent completes from assignment to merge without any human intervention: no clarifying questions answered, no mid-guide→
- Merge Queue Wait < 10 minMerge Queue Wait is the time a PR spends waiting in the merge queue after all gates pass (CI green, reviews approved, policy rules satisfied) before it is actually merged.guide→
- Model regression detection rate (thinking-length, files-read-before-edit timeline)The Test Oracle Reliability Score (TORS) measures what percentage of test failures represent real defects rather than test infrastructure issues, environmental flakiness, or poorlyguide→
Criteria - what to measure
- 01Test-oracle reliability is measured and tracked on a dashboard
- 02Auto-approve rate (% of PRs auto-merged as Green) is tracked with a target above 60%
- 03Merge queue wait time is tracked with a target under 10 minutes
- 04Agent Autonomy Score (% of tasks completed without human intervention) is measured and broken down by task type
- 05Metrics trigger automated alerts when thresholds are breached (e.g., test-oracle reliability drops)
L5 · Stage 05Autonomous
Practices - what it looks like2 guides
- Cost-per-feature (not cost-per-PR)Cost-per-feature is the total cost - in AI compute, CI infrastructure, human review time, and product management time - to deliver a complete user-facing feature from specification to production.guide→
- Business value throughput, not activity metricsBusiness value throughput is the rate at which an engineering organization delivers measurable business outcomes - revenue generated, customer problems solved, churn reduced, conveguide→
Criteria - what to measure
- 01Cost-per-feature is tracked (not cost-per-PR) - aggregating all agent, CI, and review costs per delivered feature
- 02Business value throughput is the primary metric (features delivered per week, not PRs merged per week)
- 03Metrics system auto-detects vanity metrics (high activity, low value delivery) and flags them
- 04Cost-per-feature trend is declining quarter-over-quarter
From the Field
Recent releases, projects and discussions the AI Radar classified into this capability.
discussion
How to understand if AI is adding value?
discovered
Deep analytics and session insights for your Claude Code usage
article
Dynamic Languages Faster and Cheaper in 13-Language Claude Code Benchmark
article
Agent SLOs: Grounding autonomous agents in metrics that matter
article
Show HN: Kerf-CLI – SQLite-backed cost analytics for Claude Code
discovered
Platform Evolution Engine. Distill the best from the combinatorial mess.
discovered
See where your AI coding tokens go. Interactive TUI dashboard for Claude Code cost observability.
discovered
See where Claude Code is burning tokens - turn raw JSONL transcripts into local cost analytics, hotspot views, and session-level usage insight.
Which level is your team at in Metrics?
The criteria above are what we check in a facilitated assessment. Run it yourself to place this capability, see which gates you have passed, and compare it against the other 3 in Delivery Management.
Start the assessment →