A finished draft is only part of the work.
Imagine adding an overdue-invoice filter to an internal application. The code appears quickly, but somebody still has to check which invoices qualify, correct the edge cases and release the change. If the finance team cannot trust the results yet, the task is still open. This is an invented example.
We propose agreeing the expected behaviour before the comparison starts: an unpaid invoice past its due date appears; a paid or not-yet-due invoice does not. Apply the same requirements to comparable changes with and without AI. The reviewer records acceptance and the reason for rejection, rather than judging whether the code merely looks plausible.[S1]
The assumed 75-minute saving in creation is outweighed by 85 additional minutes of review, rework and release. The AI-assisted change uses ten more person-minutes in total. These inputs demonstrate the accounting; they do not predict what will happen in your team.[S1]
| Human effort in the invented example | Without AI | With AI |
|---|---|---|
| Creating the change | 120 min | 45 min |
| Reviewing it | 30 min | 90 min |
| Rework | 20 min | 35 min |
| Testing and releasing | 10 min | 20 min |
| Total human effort | 180 min | 190 min |
Keep effort, waiting and quality visible.
A reviewer can spend half an hour on a change that waited all afternoon. Record both quantities. In our illustration, the unassisted task runs from 09:00 to 14:30, or 330 elapsed minutes; the assisted task runs from 09:00 to 16:00, or 420. Those spans include waiting and are separate from the 180 and 190 person-minutes of active work.[S1]
A usable comparison needs a record of each task’s starting point, effort, elapsed time and outcome. Add the defect follow-up when the agreed observation window ends.
| Record for each task | What belongs in the record |
|---|---|
| Comparable starting point | Task class, expected behaviour, risk, repository familiarity, developer experience and tool/model version. |
| People’s effort | Active minutes for creation, review, rework, testing and release across everyone involved. Allocate shared setup explicitly. |
| Delivery time | Start and acceptance timestamps, including waiting. Keep machine runtime separate. |
| Useful result | Accepted, rejected or abandoned, with the reviewer and reason. Retain unsuccessful attempts. |
| Later quality | Defects, their severity and repair effort during a window agreed in advance. An unobserved window is unknown, not zero defects. |
Failed attempts change the answer. Suppose a second AI-assisted attempt takes 95 person-minutes and is rejected. The batch now consumed 285 minutes for one accepted change, not 190. With no accepted result, effort per accepted change is undefined; report the effort and zero accepted results separately. Later repair effort also stays visible.[S1]
Compare similar work before drawing a team-wide conclusion.
Group comparable tasks before assigning them to the two approaches. Do not give AI all the easy jobs or ask someone to solve the same already-understood problem twice. Record task differences and repeat the comparison. Random assignment can strengthen a study; an observational team record does not isolate AI’s causal effect.
Current research makes that caution practical. In February 2026, METR explained why its later developer experiment did not provide a reliable estimate of the effect: who participated, which tasks they submitted and concurrent agent use complicated the comparison. Its May report on a survey of 349 technical workers, conducted from February to April 2026, described perceived benefits. These were self-reports rather than observed gains.[S2][S3]
More output can also mean different work. A dashboard that is now cheap to build might help the team, but its hypothetical manual build time is not automatically saved business value. Record why the delivered change was needed and what it enabled. METR’s separate analysis explains this distinction between task substitution and value.[S4]
Start with one recurring change your team can judge.
Choose a familiar task class with the developer and the person who accepts the result. Write the acceptance cases and defect-review date before starting; use the record above for both approaches. Review the completed sample together: did useful changes arrive sooner, use less human effort and meet the same quality bar? Where the answers differ, identify the stage that needs attention before expanding AI use.
That is the operating question behind a hybrid organisation: how people and agents deliver useful work together. We suggest this record as a starting method. It does not certify code security, price the tools or establish a staffing saving.
