No AI-specific metrics

At L1, engineering teams that have adopted AI tools - GitHub Copilot, Cursor, Claude Code - are tracking those tools with zero AI-specific metrics. They know how many licenses they've purchased. They do not know how many are actively used, what percentage of production code was AI-assisted, whether AI-generated code has a higher or lower defect rate than human-written code, or whether AI adoption has changed developer throughput in any measurable way. The AI tools are running, but there's no instrumentation to see what they're doing.

This is the normal starting state. Teams adopt AI tools because developers request them or because leadership mandates them, and the focus is on adoption and tooling setup. Measurement is treated as something to figure out later, once the tools are embedded in the workflow. The problem is that "later" often never arrives, and the team ends up 12 months into AI tool usage with no ability to answer the question every engineering leader eventually faces: "What did we get for this investment?"

The absence of AI-specific metrics is different from the absence of DORA metrics. DORA metrics measure delivery performance, which existed before AI. AI-specific metrics measure a new phenomenon: the degree to which AI is changing how code is produced. They answer different questions. How many iterations does it take an agent to produce a passing CI result? What percentage of merged PRs were primarily AI-authored? How does the defect rate of AI-generated code compare to human-generated code in the same codebase? These questions have no analog in traditional software metrics, and they require instrumentation that most teams at L1 haven't built.

The gap between "AI tools installed" and "AI impact measured" is where most organizations live. It's a dangerous place because decisions about AI investment are being made on impressions and anecdotes rather than data. Some developers love the tools and exaggerate their productivity gains. Others are skeptical and underreport their usage. Without AI-specific metrics, there's no way to cut through the noise and understand what's actually happening.

• Invisible AI usage means invisible impact - if you don't know which PRs were AI-assisted and which were human-written, you can't compare defect rates, review time, or test coverage between the two cohorts; the signal is permanently lost
• License spend without usage data is waste - at L1, teams often discover that 30-40% of Copilot licenses are paid for but never actively used; without usage metrics, this waste is invisible until someone does a license audit
• AI tool selection requires evidence - deciding whether to use Claude Code vs Cursor vs Copilot should be data-driven; teams without AI-specific metrics make these decisions based on demos and developer preferences rather than actual productivity impact in their specific codebase
• Regulatory and compliance risk - as AI-generated code becomes subject to audit requirements (EU AI Act, SOC2 extensions, etc.), teams that didn't track AI usage have no way to produce the records they need; the measurement gap becomes a compliance gap
• Cannot learn what works - some AI usage patterns produce much better outcomes than others; without metrics, there's no way to identify which patterns are working and systematize them across the team

1. Define "AI-assisted" for your context - Before instrumentation, decide what counts as AI-assisted code. A PR where the developer used Copilot for one line completion? A PR written primarily by an agent? A PR reviewed by an AI review tool? The definition shapes the metrics. Start with a simple binary: was an AI agent the primary author of this PR's changes?
2. Add an AI label to your PR workflow - The simplest possible instrumentation is a PR label. Create a label like ai-assisted or agent-authored and ask developers to apply it when appropriate. This is zero-infrastructure measurement. It's imperfect and relies on developer discipline, but it gives you your first data point: what percentage of PRs in a given week are developers identifying as AI-assisted?
3. Instrument your AI tool's native analytics - GitHub Copilot, Cursor, and most enterprise AI coding tools have built-in usage analytics. Check the admin console. You likely already have data on acceptance rates, active users, and lines accepted. Pull this data into your engineering dashboard.
4. Track AI tool active usage, not license count - Define "active usage" (e.g., used the AI tool at least 3 days in a week) and track this metric weekly per developer. The gap between license count and active users is your first meaningful AI-specific signal.
5. Start a simple AI metrics log - Create a shared document or dashboard where you record weekly: AI tool active users, PR count (total and AI-labeled), and any qualitative observations. Even a Google Sheet works. The goal is a habit of recording, not a sophisticated BI system.
6. Review AI metrics in your sprint retrospective - Whatever AI metrics you start tracking, make them a standing agenda item in the sprint retrospective or engineering review. "Here's our AI usage this week - anything interesting?" creates the habit of attention that makes measurement meaningful.

Conflating license count with usage. Buying 50 Copilot licenses and reporting "50 developers use AI" is a common mistake that produces misleading ROI calculations. License count is a budget line item. Active usage is a behavior metric. Track both and report the gap honestly.

Treating acceptance rate as the primary success metric. GitHub Copilot's acceptance rate (percentage of suggestions accepted) is widely reported but weakly predictive of productivity impact. Developers can accept low-quality suggestions that slow them down later, or reject high-quality suggestions that they'll type themselves anyway. Acceptance rate measures engagement, not value. It's worth tracking but shouldn't be the headline metric.

Waiting for perfect instrumentation before tracking anything. Teams frequently say "we'll track AI impact once we have a proper BI pipeline." The BI pipeline takes six months to build, and by then the historical data is gone. Start with imperfect manual tracking - PR labels, weekly surveys, a shared spreadsheet - and improve the instrumentation over time. Something is always better than nothing.

Not tracking the baseline before AI tool rollout. The most important AI-specific metric is the before/after comparison. Teams that roll out AI tools without establishing a pre-AI baseline of PR throughput, lead time, and defect rate cannot compute impact. If you haven't started yet, measure now. If you've already rolled out, find whatever historical data you can from before the rollout.

Ignoring qualitative metrics. Developer satisfaction, perceived productivity, and tool preference are not captured in activity metrics but are critical for understanding adoption health. Run a brief monthly developer survey: "How much is AI tooling helping you in your daily work? (1-5)" The trend in this number is as important as any quantitative metric.