Measuring the real effect of AI in an engineering team: a protocol that holds up in front of the board and the team

Who this guide is written for.

• CTOs and VPs of Engineering who have to show the board whether the AI tooling budget changed anything, with a number they can defend
• Heads of engineering whose team already argues about whether things got better or worse, with no data either side would accept
• Engineering managers who have been asked to introduce productivity metrics and know what happens to a team when the metric is per person
• Platform and developer-experience teams who own the CI, the repository and the ticket system and will run the measurement after handover

Why faster ticket closure is not evidence

A ticket that closes faster tells you the ticket closed faster. It does not tell you whether the change was reworked after merge, whether the reviewer spent the time the author saved, or whether the next change in the same file took longer. The 2025 DORA report, based on survey responses from nearly 5,000 technology professionals, found a positive relationship between AI adoption and software delivery throughput and, in the same data, a negative relationship with delivery stability. The report's own reading is that acceleration exposes weaknesses downstream when the control systems, automated tests, version control practice and fast feedback, are not there to absorb the extra change volume.

A throughput number on its own is half a result. The DORA ROI report of April 2026 puts a price on the other half with an illustrative model: in its sample calculator for a 500-person engineering organisation, a change failure rate that rises from 5% to 6% after adoption shows as a negative downtime impact of 344,000 USD. The figure is a worked example, not a measurement of any company, and it is quoted here for one reason: it shows that a one-point move in a stability metric is large enough to cancel a visible gain in speed.

The argument inside teams has the same shape. In the largest practitioner discussions of 2026 on the subject, the dispute is about whether the code behind the closed ticket is worse and whether the pressure to keep the pace stops anyone from checking. Without a protocol both sides are right, and the measurement settles it.

What the controlled experiments actually measured

The published studies disagree, and the disagreement is informative once the method behind each number is visible. The table lists the studies a board is most likely to have heard of. None of them measured your team.

The measurement design

A protocol that survives both the board and the team has five parts, and each one exists because a specific study failed without it. Everything below is computed from git history, CI logs and the ticket system. No commercial platform is required and no vendor's acceptance-rate dashboard is an input.

• Baseline before the rollout. Twelve weeks of history before the first licence, for every metric that will be reported afterwards, reconstructed from git and CI if nobody took it at the time. A measurement with no before has nothing to compare against except a feeling.
• Paired metrics. Every speed metric is reported next to a stability metric from the same window and the same repositories: lead time for changes next to time to first fix after merge; merged pull requests per week next to the share of merges followed by a revert or hotfix within 30 days; deployment frequency next to change failure rate. A pair can move in opposite directions, and that is the finding.
• A window long enough to include the dip. Eight to twelve weeks after the rollout, because the first weeks of any tooling change carry a learning cost that a four-week snapshot reads as failure or, taken late, as pure gain.
• A comparison group by team, not by volunteer. Where the organisation rolls out in waves, the teams still waiting are the comparison group. Where it cannot, the comparison is the same team's own baseline, with the limit stated in the report. Volunteers are never the comparison, because METR's data shows what happens when enthusiasts opt in and sceptics opt out.
• A short list of what is not counted. Lines of code, number of commits, number of pull requests and assistant acceptance rate are excluded as outcomes, because the tool inflates every one of them without any change in what ships. They stay in the data only as denominators.

Per-person metrics, and why they poison the measurement

The fastest way to lose a measurement is to publish it per person. The SPACE framework (Forsgren et al., ACM Queue 2021) states the principle: developer productivity cannot be captured by a single metric or by an individual's activity level. A ranking of engineers by assistant usage or acceptance rate turns the metric into the target within a sprint. People run the tool to be seen using it, accept suggestions they would have rewritten, and the number goes up while the thing it was supposed to stand for stops moving. Employers have already started weighing individual AI usage in performance evaluations, as reported by The Register in November 2025, and the practitioner threads that follow such announcements describe the resistance it produces in detail.

The trust data explains the resistance. In the 2025 Stack Overflow survey of 48,968 developers, 66% named AI output that is almost right but not quite as their biggest frustration and 45.2% said debugging AI-generated code takes more time, while only 3.1% highly trust the accuracy of the output. A team that is told to use the tool more, and ranked on it, is being asked to ignore what it observes daily. The protocol therefore aggregates at team level, keeps names out of every panel, treats usage data as a denominator rather than an outcome, and collects the team's own assessment through an anonymous survey run before the rollout and again at the end of the window.

Where this goes wrong, and what we do about it.

1. 1
   The baseline is taken after the rollout has started

   Licences went out in a pilot, then to everyone, and the request to measure arrived a month later. The first data point is already inside the learning dip or already past it, so the before-and-after comparison measures the ramp, not the tool. Seasonality does the rest: a release freeze or a holiday period in either window moves the numbers more than the assistant does.

   What we do about it

   We reconstruct the baseline from git and CI history for the twelve weeks before the first licence date, per repository, and mark every week that contained a freeze, an incident or a holiday so it can be excluded from both windows symmetrically. Where the rollout date differs per team, each team gets its own baseline. We then verify that the baseline queries return the same values when run by your platform team.

   What stays with you: baseline sheet per team and repository, with the queries, the exclusion calendar and the licence date each baseline is anchored to
2. 2
   Throughput is reported without its stability pair

   The panel shows lead time and merged pull requests per week, both improved. Rework after merge, reverts and time to first fix are not on it, because nobody asked for them and the tooling vendor's dashboard does not have them. The DORA data says stability is where the cost of extra change volume appears, so the panel is showing exactly the half that cannot go wrong.

   What we do about it

   We define every reported pair up front and compute both halves from the same repositories and the same window: for each merged change, the days until a later commit in the same lines references a bug, a revert or a hotfix. We report the pair as one row, so a speed gain with a stability loss is visible as a single result, and we quantify the loss in engineering hours spent on fixes rather than as a percentage alone.

   What stays with you: paired-metric table with both halves per team per window, plus the queries that produce it
3. 3
   The comparison group is whoever volunteered

   The pilot was open to anyone who wanted it, so the people measured are the people who expected the most from the tool, on the tasks they chose to bring. METR's February 2026 update names this exact failure: developers declined to work without AI and 30-50% withheld tasks they expected to be slow without it. The measured group and the comparison group differ in attitude and task mix before the tool does anything.

   What we do about it

   We allocate by team and repository, using the rollout waves the organisation already has, and record who declined and why. Where no wave structure exists, we compare the same team against its own baseline and write the limit into the report in the first paragraph, not in a footnote. We compare like tasks by matching on ticket type and repository, and we report the task mix in both groups so a shift in what people chose to work on cannot pass as a speedup.

   What stays with you: allocation record with the comparison design, the refusal log and the task-mix table for both groups
4. 4
   Usage becomes the metric

   The most available number is how much each person used the assistant, so it becomes the number on the slide. Within a sprint the team optimises for it: suggestions are accepted and then rewritten, sessions are opened and left running. The usage line rises, the output lines do not, and the report concludes that adoption is high and impact is pending. Meanwhile the engineers who were ranked stop cooperating with any future measurement.

   What we do about it

   We keep usage data in the model only as a denominator, so that outcome metrics can be read per unit of assistant use at team level, and we remove every per-person view from the panel before it is shared. The anonymous survey asks the team where the tool helped and where it cost time, and we publish those answers next to the numbers. Access to the raw data is written down: who can query it, at what granularity, and what is never exported.

   What stays with you: panel specification with the aggregation level, the access rules and the survey instrument, plus the survey results for both rounds
5. 5
   Counting the output the tool inflates

   Lines of code, commits and pull requests all go up when an assistant is in the loop, because generating them is what the assistant does. A count of any of them after the rollout shows growth that has no relation to what reached users. Reviewers, meanwhile, absorb the extra volume, and their time does not appear in any count.

   What we do about it

   We define what counts as done before the window opens: a merged change that has survived 30 days without a fix in the same lines, and a roadmap item closed with the ticket system's own acceptance state. We add review load per merged change as its own row, so the time that moved from author to reviewer is visible. Then we run the definitions on the baseline to confirm they behave the same way before and after.

   What stays with you: definition sheet for every outcome metric, with the query, the exclusion list and the review-load row

What stays with you.

• Measurement protocol: the questions, the metric pairs, the window, the comparison design, the aggregation level and the list of what is deliberately not measured
• Baseline sheet per team and repository for the twelve weeks before the rollout, with the queries and the exclusion calendar
• Paired-metric panel built on your git, CI and ticket data, with the aggregation level fixed at team and no per-person view
• Allocation record: comparison groups by team or wave, refusal log, and the task-mix table for both groups
• Survey instrument and results from two anonymous rounds, before the rollout and at the end of the window
• Report after 8 to 12 weeks stating what changed, at what cost in rework and review load, what could not be measured and why, written so it can be read by the board and by the team without a second version
• Handover: the queries, the definitions and the panel stay with your platform team, so the same measurement can be repeated each quarter

How we work.

1. 1
   1. Protocol and baseline

   We agree the questions the board and the team each want answered, choose the metric pairs, and reconstruct the baseline from git and CI history for the twelve weeks before the first licence. If the rollout has not started, this is also where the wave structure and the comparison groups are fixed.

   Week 1-2
2. 2
   2. Instrumentation

   We build the queries against your repositories, CI and ticket system, run them on the baseline to check they behave, and set up the panel with team-level aggregation and written access rules. The first survey round goes out.

   Week 2-3
3. 3
   3. Measurement window

   The panel runs for eight to twelve weeks. We check the data monthly for gaps, freezes and incidents that need marking, and we do not report interim results, because a four-week snapshot reads either as failure or as pure gain depending on when it is taken.

   Week 3-15
4. 4
   4. Report and handover

   We run the second survey round, compute every pair for both windows and both groups, and write the report with its limits in the first paragraph. Queries, definitions and the panel are handed to your platform team with a runbook for the next quarter.

   Week 15-16

The rest of this cluster.

Where this turns into a service.

Where the dates and numbers come from.

• Becker, Rush, Barnes, Rein (METR), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (arXiv:2507.09089, July 2025)
• METR, Measuring AI uplift: an update on methodology and late-2025 data (24 February 2026)
• Cui, Demirer, Jaffe, Musolff, Peng, Salz, The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers (Management Science, 2026)
• Google Cloud, Announcing the 2025 DORA Report: State of AI-assisted Software Development
• DORA, The ROI of AI-assisted Software Development (v.2026.1, April 2026)
• InfoQ, New DORA Report Claims Strong Engineering Foundations Drive AI Return on Investment (May 2026): the illustrative change failure rate example and the 344,000 USD figure
• Forsgren, Storey, Maddila, Zimmermann, Houck, Butler, The SPACE of Developer Productivity: There's more to it than you think (ACM Queue, 2021)
• Stack Overflow, 2025 Developer Survey: AI section (48,968 respondents)
• The Register, Devs gripe about having AI shoved down their throats: employers weighing individual AI usage in performance reviews (19 November 2025)