How to Measure Developer Productivity in the Age of AI

Every AI productivity conversation eventually reaches the same request: prove it. Show the board that the licences paid for themselves. Show that the team ships more. Show a number.

The meeting usually has a slide with three charts: seats activated, suggestions accepted, and pull requests opened. All three go up and to the right. Nobody in the room can say whether the product shipped faster, because the slide measures the tool rather than the work. That is not a data problem. It is a definition problem.

The number is easy to produce and easy to get wrong. In firms that adopted coding agents, code, commit, and pull-request volume rose. They also generate more review work, more rework, and more ways to feel busy without shipping. If you measure the first set because it is visible, you will conclude the investment worked while your delivery system quietly gets slower.

I have watched this pattern in metrics programmes long enough to expect it, and I no longer trust a productivity dashboard that has no review column. The fix is not a more sophisticated AI metric. It is a return to the same discipline good engineering measurement has always needed: measure the system, at team level, over time, with more than one signal. AI changes the weights. It does not change the principle.

Why is measuring developer productivity harder with AI?

Because AI separates the act of producing code from the act of delivering value. The gap between them was always there, but AI widens it. A developer can now generate a large, plausible change in minutes, then spend a day getting it reviewed, corrected, and deployed. The output metric moves immediately. The outcome metric moves later, if at all.

There is also a measurement illusion. AI usage is instrumented by default: the vendor reports acceptance rates, tokens, sessions, and active users. Those numbers are precise, available, and about the tool rather than the work. Precision feels like validity. It is not.

Stack Overflow's 2025 survey found that 84% of developers use or plan to use AI, while 46% actively distrust its accuracy and only 33% trust it. Two-thirds cited AI solutions that are almost right, but not quite as their top frustration, and 45% said debugging AI-generated code takes longer. (Stack Overflow) That is the experience behind the delivery numbers: more drafts, more verification, and a trust level that changes how much work review absorbs.

Finally, AI makes individual differences more visible, which tempts organisations to score individuals. That is both unfair and counterproductive. The same tool helps a junior developer write boilerplate and helps a senior developer explore a design. Judging the person by the tool's output confuses the instrument with the musician.

What developer productivity metrics should you track instead of individual output?

I measure four layers, all at team level, and read them together:

1. Delivery outcomes. The five DORA metrics: deployment frequency, lead time for changes, change failure rate, failed deployment recovery time, and rework rate.
2. Flow and review. Cycle time distribution, work in progress, review pickup time, review latency, pull request size, and the share of changes that come back for revision.
3. Quality. Escaped defects, revert rate, duplication and churn trends, incident rate, and whether tests catch the failures that matter.
4. Experience. Satisfaction, cognitive load, trust in the tooling, after-hours work, and intent to stay, measured with a recurring survey rather than inferred from activity.

Define each measure once and keep the definition stable. Deployment frequency can mean production releases, merges to main, or successful pipeline runs; those are different numbers with different incentives. Review latency can start at the first reviewer comment or at the ready-for-review event. Pick the definition that matches the decision you want to make, write it down, and do not change it mid-quarter.

AI adoption sits outside those four layers as a covariate. Track which teams use which tools, how often, and for what kind of task. Use it to explain variation. Do not use it as the outcome.

The reason for team level is statistical as well as ethical. Individual productivity is noisy; a single sprint is noise on top of noise. Team-level trends over a quarter, compared with the team's own baseline, are the smallest unit I trust. The same argument applies to the frameworks behind these layers, and to the predictability problem in the delivery job. This post is about what changes when AI enters the system.

Does AI actually make developers faster?

My honest answer is: sometimes, for some tasks, and the effect is much harder to measure than the vendors imply.

The strongest causal evidence is METR's randomised controlled trial. Sixteen experienced open-source developers worked on 246 real tasks in repositories they knew well. When AI was allowed, tasks took 19% longer, with a confidence interval from +2% to +39%. The same developers expected AI to make them 24% faster and, after the study, still believed it had made them 20% faster. (METR)

That result is frequently quoted as AI makes developers slower. The authors are more careful. They studied experienced developers, mature repositories, real issues, and early-2025 tools, mostly Cursor with Claude 3.5 and 3.7 Sonnet. I take the design detail seriously because it defines the boundary of the claim: 16 developers, 246 tasks averaging about two hours, coin-flip assignment per task, and original repositories averaging over a million lines. That is the closest thing the field has to a clinical trial for coding assistants, and it is still a narrow population. They explicitly do not claim that AI fails to speed up most developers, or that it cannot help in other settings, such as less experienced developers or unfamiliar codebases. They claim that in their setting, the measured effect was a slowdown, and that self-reports were badly miscalibrated.

The follow-up matters too. In February 2026, METR reported an estimated 18% speedup for returning developers (confidence interval −38% to +9%) and a 4% speedup for newly recruited developers (−15% to +9%). But the authors call the data unreliable because of selection effects: 30–50% of developers withheld tasks they did not want to do without AI, and the developer and task pool was no longer representative. They are redesigning the experiment. (METR)

So what should a manager take from this? Not AI does not work. Take three things:

• Self-reported speedup is not evidence. The gap between belief and measurement was roughly 39 points on METR's completion-time scale in the first trial.
• Context matters more than the model. Mature codebases with implicit requirements are the hard case, and that is where most enterprise work lives.
• Measure your own system. The trial gives you a prior, not an answer. Your repositories, tasks, and review process are different.

Why did the gain disappear by the time work shipped?

Because the constraint moved downstream. A Harvard study using Jellyfish data from more than 700 firms, covering 300 million work events from 2021 to March 2026, found that introducing AI coding agents correlated with a 30% increase in lines of code, a 20% increase in commits, and a 23% increase in pull requests. Issue and epic resolution did not change in a statistically significant way. (Harvard/Jellyfish study)

The review process absorbed the difference. Reviewers also carry a verification tax that the timestamps do not show: they have to reconstruct intent from a diff that arrived without the author's implementation journey, then decide whether plausible-looking code is correct in edge cases the ticket never named. That is slower than reviewing code written by a colleague who sat next to them.

Average review time from pull request to merge rose 49%. The share of pull requests with changes requested nearly doubled. Comments per pull request rose 35%, and the share of workers performing reviews rose 14%. Those are October 2026 figures from the study version Ars Technica covered; later revisions of the dataset may move the exact percentages, so treat the direction as the finding and re-check the current paper before quoting them in a board deck. AI agents wrote only 23.3% of review comments and 10.8% of pull requests, even though 80% of the firms used some form of AI code review and 95% had implemented coding agents. The tool that creates the review load is not yet carrying it. That is the operational gap a manager has to plan for.

This is the most useful finding for measurement because it tells you where to look. If code generation gets cheaper and review capacity does not, the queue moves to review. Your dashboard should show that. Most do not, because they measure the generating step and not the waiting step.

It also explains a common executive experience: developers say they are faster, delivery metrics are flat, and nobody can explain the gap. The gap is a queue. Measure the queue.

How do you measure review load?

Review is now a production constraint, so I instrument it like one. Five measures are enough to start, and each needs a stable definition:

• Review pickup time. How long from ready for review to the first meaningful comment. This is the queue, not the work.
• Review latency. How long from first review to merge. This is where rework shows up.
• Pull request size. Lines changed and files touched. Review time grows with size, and very large changes get worse review, not just slower review.
• Changes-requested rate. The share of pull requests that come back for substantive changes. A rising rate can mean lower draft quality or better review; read it with the other signals.
• Review concentration. The share of reviews handled by the top few reviewers. High concentration is a bus-factor risk and a burnout signal.

Add one qualitative measure: whether reviewers can explain the intent of the change after one focused pass. If they cannot, the problem is not reviewer effort. It is the size and framing of the change.

These measures also give you a lever. If AI increases PR volume, you can respond by reducing batch size, requiring a review packet, or splitting generated changes into logical commits. Those are system changes. Telling reviewers to go faster is not.

The flow layer has its own established toolkit: flow metrics covers cycle time distribution, WIP, throughput, and Little's Law. Use it here rather than inventing a new vocabulary. The AI-specific addition is the review queue, because that is where the new work lands.

Which metrics still hold in the AI era?

The durable metrics are the ones that describe outcomes and constraints, not activity. Four families survive:

DORA's five. Deployment frequency, lead time, change failure rate, failed deployment recovery time, and rework rate. DORA's 2025 sample is close to saturation: 90% of respondents use AI at work, with a median of two hours a day, while 30% report little or no trust in the generated code. AI may improve the first two and worsen the next three, which is exactly why you track all five together. DORA metrics are not a performance review covers the delivery layer in detail. DORA's 2025 research describes AI as an amplifier: it magnifies the strengths and dysfunctions of the surrounding system, and the returns depend more on the organisation than the tool. (DORA)

Read the five as a set. A team that deploys more often but recovers more slowly has traded one constraint for another. A team whose lead time fell while rework rose has moved work, not removed it. The AI-era question is not did throughput increase? but did throughput increase without a matching increase in recovery cost?

Flow. Cycle time distribution, WIP, throughput, and blocked time. If AI increases the arrival rate of work without increasing the completion rate, WIP rises and cycle time gets worse. That is a system signal you can act on.

Quality. Escaped defects, revert rate, incident rate, duplication and churn trends. GitClear's 2026 analysis of 623 million changes reports code-block duplication up 81%, within-commit copy/paste up 41%, error-masking constructs up 47%, and two-week churn up 15%, alongside a 70% decline in refactoring and a 74% decline in legacy maintenance since 2023. (GitClear) The data is vendor-collected and correlational, so treat it as a warning signal rather than proof. But the direction is consistent with the review bottleneck: more generated code, less reuse, more cleanup later.

Experience. Satisfaction, cognitive load, trust, and sustainable pace. These are leading indicators. A team that distrusts the tool reviews every line twice; a team that trusts it too much stops reviewing. Both show up in delivery eventually.

If you want the shortest possible dashboard, use these four and ignore the rest for a quarter. You will learn more than from a page of tool telemetry.

Which AI metrics should you retire?

Most AI dashboards measure the tool's behaviour and call it productivity. Retire or demote these:

• Lines of code and commit count. They reward volume and punish refactoring. AI makes them trivially easy to inflate. Why engineering metrics fail covers the dashboard traps; the AI-specific version is that the tool can generate the metric directly.
• Pull request count. A PR is a unit of review work, not a unit of value. More small PRs can be healthy; more large PRs are usually not.
• Token consumption. It measures spend, not output. High token use can mean hard problems or a loop that will not terminate.
• Suggestion acceptance rate. Acceptance means a developer pressed a key. It says nothing about whether the suggestion was correct, maintainable, or shipped.
• Self-reported hours saved as the primary ROI number. Useful as a directional survey item, dangerous as a board metric. METR's gap between belief and measurement is the cautionary tale.
• Percentage of code written by AI. This is an adoption metric, not an outcome. It also creates an incentive to use AI where it is not helping.

The individual leaderboard trap

The worst version of this is a per-developer AI leaderboard. It is also the version most likely to appear when a tool vendor offers a free analytics add-on. If you are instrumenting the agent rather than the person, designing AI agents covers the loop-level controls; the measurement rule is that they belong to the system, not the individual. It ranks people by acceptance rate, token use, or AI-generated line share. It punishes engineers who take time to review, mentor, or solve ambiguous problems. It rewards volume, which is the opposite of what you want in a system where review is the constraint.

If you need individual signals, use them for coaching, privately: where is this person stuck, what task types does the tool help with, what review habits would reduce risk. Do not turn a diagnostic into a score.

How do you measure experience without a popularity contest?

I use a short, recurring, anonymous survey and keep it stable enough to compare over time. The SPACE framework gives you the dimensions: satisfaction, performance, activity, communication and collaboration, and efficiency and flow. (SPACE framework) You do not need to measure all five every quarter. Pick the ones tied to your current constraint.

Three questions that survive Goodhart

For an AI rollout, I would start with three questions:

1. How often does AI output require rework you would not have needed without it? This captures the verification tax without asking people to estimate time saved.
2. How confident are you that you can verify AI-generated changes in the time you have? This captures the trust-calibration problem better than a generic trust score.
3. What part of your week got better or worse since the tool arrived? Open text, coded into themes. It finds the problems your metrics did not anticipate.

Run it every quarter, on the same scale, with a guaranteed response anonymity threshold. Then read the free text. The number tells you whether something changed; the text tells you why.

There is a governance point here too. Stack Overflow's 2026 data reports that only 24% of developers say their organisation provides approved AI tools and guidance for using them, while 30% say AI use is optional and left to the individual. (Stack Overflow) If usage is invisible and policy is absent, you cannot interpret the productivity data. You are measuring shadow work.

How do you attribute improvement to AI?

You cannot prove causality from a dashboard. You can build a credible case with four habits:

Compare the team with its own past. Use a rolling baseline, not a cross-team ranking. Teams differ in domain, legacy load, and on-call burden.

Record the rollout date and the confounders. A platform migration, a reorg, a hiring freeze, and a model upgrade can all move the same metrics. Write them down when they happen, not after the quarterly review.

Triangulate. If delivery improved, quality held, review load stayed flat, and the survey improved, you have a coherent story. If only the tool's own usage metrics improved, you have a procurement story.

Look for the mechanism. Cycle time fell because review pickup time fell after we capped PR size is a mechanism. Cycle time fell after we bought Copilot is a correlation. Prefer the explanation you can act on.

If you are managing the tooling budget rather than the delivery system, the LLM skills and agents guide for engineering managers covers the procurement side. The measurement point is the same: buy the capability, then measure the outcome.

For a board-level number, I would use cost per successful outcome rather than cost per seat or hours saved. Pick the outcome your business cares about: a merged change, a resolved ticket, a deployed feature. Divide the fully loaded cost by the number that actually shipped. It is blunt, hard to game, and comparable over time.

What should you instrument first?

I would start with five measures, not fifty:

1. Review pickup time and review latency. The new constraint deserves a dashboard line before anything else.
2. PR size distribution. If the median change is growing, review quality is falling even when review time looks stable.
3. Rework rate and change failure rate. The quality cost of speed shows up here first.
4. Cycle time p50 and p85. The tail tells you whether the system is predictable; the median hides the pain.
5. One quarterly experience survey with the three questions above.

Then add AI adoption as a covariate: which teams, which tools, which task types. Keep it out of the headline.

The point is not to prove that AI works or does not. It is to see where the constraint moved, and whether your system is absorbing the new capacity or choking on it. AI made code cheaper. It did not make judgment cheaper, review cheaper, or trust cheaper. Measure the parts that stayed expensive.