Basic audit: who uses what

Basic audit at L2 means the organization has established visibility into which developers are using which AI tools, at what frequency, and for what purposes. It's not a comprehensive provenance record (that's L3-L4), but it answers the first-order governance question: who is using AI in our delivery pipeline, and are they using approved tools?

This level of audit typically draws on two sources: tooling vendor dashboards (GitHub Copilot usage stats, Claude for Teams analytics) and PR disclosure fields that developers fill in manually. Together, these give a reasonable picture of adoption without requiring custom instrumentation. The picture is incomplete - vendor dashboards show license seat usage and acceptance rates (adoption signals, not quality ones) but not the content of AI interactions; PR disclosures are self-reported and inconsistently filled - but it's a genuine improvement over the zero visibility of L1.

The audit data at L2 serves three purposes simultaneously. For compliance, it provides evidence of controlled AI use that can be presented in SOC2 or ISO 27001 audits. For management, it shows which teams have adopted AI tools and which haven't, enabling targeted support. For productivity analysis, it creates the first dataset correlating AI tool use with delivery metrics - even imperfect data starts to reveal patterns when collected consistently over time.

The critical discipline at L2 is to define audit questions before collecting data, not after. Organizations that collect everything they can access and then try to find meaning in it get drowned in noise. The useful audit questions at L2 are specific: Is every developer using an approved tool (not a personal subscription)? Are high-risk repositories showing appropriate AI disclosure rates? Is the distribution of AI use consistent with what developers report in surveys? Start with these questions and collect the data that answers them.

In 2026 "who uses what" stopped meaning only people. Agents now run under their own credentials - CI bots, coding agents with API keys, MCP servers holding OAuth tokens - and they are multiplying faster than anyone is tracking them. The Cloud Security Alliance puts non-human identities at 45 to 144 for every human, and reports that 51% of organisations have no clear owner for their AI identities. Okta said at Oktane in September that only 34% of organisations apply the same controls to agents as to people. So the L2 audit gains one field that matters more than any adoption metric: every agent, bot account and long-lived AI key in the inventory has a named human owner. The owner is who gets paged when the agent misbehaves, who approves its scope, and who is asked to justify it when a key leaks. An agent without an owner is not an adoption statistic, it is an orphaned credential.

The breach data from September shows what an orphaned credential costs. In the first mass-exploitation campaign run by AI agents, reported by VentureBeat from GreyNoise data, attackers reached 395 organisations in 48 countries, harvested stolen tokens on 5,871 machines including Claude, Cursor and Gemini sessions, and burned $600k of model credits through one compromised agent dashboard. Anthropic's September threat report gives the same advice in one line: treat AI keys and agent integrations like production credentials. You cannot rotate, revoke or scope a credential you did not know existed, and you cannot ask anyone about it if nobody owns it.

• Makes compliance claims defensible - "we audit AI tool use and here is the data" is qualitatively different from "we have a policy" in an audit; actual usage data gives compliance claims evidentiary support
• Identifies shadow AI persistence - when vendor dashboard adoption numbers are lower than survey-reported AI use, the gap is shadow AI; the basic audit makes the gap visible and measurable rather than theoretical
• Reveals adoption patterns that inform support - some teams will adopt quickly, others will lag; the adoption map tells you where to focus enablement efforts and what barriers are preventing adoption in slower teams
• Creates the measurement baseline for ROI analysis - the same data that supports compliance audits also supports the business case for AI investment; correlating AI tool usage with PR throughput is the foundation of the productivity ROI argument
• Gives every agent an accountable human - 51% of organisations cannot say who owns their AI identities (CSA); an owner field in the inventory is the cheapest control that closes that gap, and every later control - scoping, certification, revocation - depends on it
• Enables proactive issue detection - usage patterns that deviate from policy (tools being used that aren't on the approved list, usage patterns that suggest prohibited data types being processed) are visible in audit data before they become audit findings

1. Consolidate vendor dashboards - every approved AI tool provides some usage analytics. GitHub Copilot has an organization-level dashboard showing seat usage, suggestion acceptance rates, and active users by team - all of them adoption signals, none of them evidence about the quality of what was accepted. Claude for Teams provides usage logs. Collect these into a single place - even a monthly spreadsheet - that gives an organization-wide view.
2. Define your audit fields in PR templates - the PR audit fields you collect consistently are the most useful data. At minimum: AI tool used (dropdown from approved list, or "none"), purpose (code generation, test generation, code review, debugging, other), and rough percentage of code that was AI-generated. Keep it to three fields or developers won't fill it out.
3. Create a compliance dashboard - a simple dashboard that shows: percentage of PRs with AI disclosure filled in, percentage of developers with active approved tool licenses, any tools appearing in disclosures that are not on the approved list. This dashboard is updated weekly and reviewed in your team leads meeting. Visibility creates accountability.
4. Run a quarterly shadow AI re-census - vendor dashboard numbers tell you about approved tool use; they don't tell you about shadow AI. A quarterly anonymous survey ("are you using any AI tools not on the approved list?") tracks whether shadow AI is increasing, decreasing, or stable after the official policy was implemented.
5. Flag anomalies for follow-up, not punishment - when audit data shows a developer using a non-approved tool, the first response should be a conversation: "we saw you used X, which isn't on our approved list - what were you trying to do, and does our approved list cover it?" This converts audit findings into policy improvement opportunities.
6. Inventory agents and AI keys, each with a named owner - list every agent identity, bot account, service token and long-lived model API key that acts in your delivery pipeline, and record one accountable human per entry. Start from what your vendors and IdP can already enumerate (Copilot and Claude admin consoles, GitHub App installations, cloud IAM service accounts), then ask each team for the rest. Anything without an owner after two weeks gets its key revoked; if something breaks, you have found the owner.
7. Export audit data monthly for the compliance record - store monthly exports of usage data (team-level, not necessarily individual-level) in your compliance documentation repository. This creates the time series that auditors want to see: evidence of continuous monitoring, not just a snapshot at audit time.

Collecting audit data without acting on it. An audit dashboard that nobody looks at is theater, not governance. The audit data needs a review cadence - weekly for the compliance dashboard, monthly for the trend analysis - and someone accountable for following up on anomalies. Without action, the audit data creates false assurance.

Individual-level tracking that feels like surveillance. There's a meaningful difference between "12% of PRs from the backend team don't have AI disclosure fields filled in" (team-level, actionable) and "Sarah has only accepted 23% of Copilot suggestions this week" (individual-level, demotivating). Keep audit data at team level for management reporting. Individual-level data, if collected, should be accessible to the individual but not used in performance management.

Trusting vendor dashboards as complete. GitHub Copilot's dashboard shows suggestion acceptance rates, which count what a developer pressed Tab on rather than what survived - and it doesn't show chat usage, it doesn't show whether the developer used a personal account on the same task, and it doesn't show what code the suggestions were generated for. Vendor dashboards are one data source, not the complete picture. They should be supplemented with PR disclosure data.

Treating low adoption numbers as neutral. If 40% of licensed developers are actively using approved AI tools, that's not a neutral finding - it means 60% are either not using AI tools (lost productivity opportunity) or using shadow AI (governance risk). Low adoption numbers are always worth investigating, not just reporting.

Auditing people and forgetting the agents. A dashboard of which developers hold which licences says nothing about the CI bot with a model API key, the MCP server with a broad OAuth grant, or the personal agent a developer wired to the company repository. Those credentials are what the September agent-run campaign harvested. Count them, and give each one an owner.

Skipping the baseline. Organizations that start collecting audit data without establishing a baseline can't measure improvement. On the day you publish your AI policy, record: current adoption rate, shadow AI survey result, PR disclosure rate. That baseline is what you measure against at 90 days, 6 months, and 1 year.