Introduction

Coding agents can produce a large diff in minutes. That feels fast, but generated code is not the outcome a software team needs. The useful outcome is a correct, accepted change that remains correct after review and real use.

Public reports show people delegating longer and more parallel tasks to coding agents. Other controlled research found that experienced developers using earlier tools took longer on a specific class of mature-repository tasks. These observations are not contradictory: tool capability, task selection, workflow, and measurement all differ.

The more useful question is how to measure the result without assuming that agents make every developer faster or slower.

Measure the Accepted Change

Lines of code, tokens generated, and agent runtime are easy to count. They are weak productivity measures because they reward output rather than value.

Use a task-level outcome:

total task time =
    setup and instruction time
  + agent waiting time that blocks other work
  + review time
  + correction time
  + follow-up rework inside the observation window

Track whether the change meets predefined acceptance criteria. A fast rejected change is not a partial success unless it produces reusable knowledge that you also measure.

For each task, record:

Choose Comparable Tasks Carefully

You cannot perform the identical repository change twice without the second attempt benefiting from the first. Use a set of comparable tasks rather than exact repeats.

Group work into categories such as:

• isolated bug fixes with a reproducible failure;
• small feature additions with clear acceptance tests;
• dependency or configuration updates;
• unfamiliar-code investigation;
• broad refactors with architectural judgment.

Before starting, estimate difficulty without knowing which workflow will be used. Randomly assign agent-assisted and comparison tasks where practical. If random assignment is unrealistic, report the selection process and avoid causal claims.

Do not give the agent only clean, well-specified work while assigning ambiguous incidents to humans. That measures task allocation, not tool impact.

Separate Human Time From Agent Runtime

An agent can work while a developer does something else, so a 40-minute run does not necessarily consume 40 minutes of human labor. It can still block delivery if the result is needed immediately.

from dataclasses import dataclass


@dataclass
class TaskRecord:
    active_human_minutes: float
    blocked_wall_minutes: float
    review_minutes: float
    correction_minutes: float
    accepted: bool
    reopened_within_14_days: bool

For an OpenAI-backed task, capture API usage beside human and wall-clock time. The repository pins openai==2.24.0 so the example remains reproducible. The wrapper records measurements without storing the task prompt in the metrics table:

import os
from time import perf_counter

from openai import OpenAI


client = OpenAI()


def run_measured_task(instructions: str) -> tuple[str, dict]:
    started = perf_counter()
    response = client.responses.create(
        model=os.environ["OPENAI_CODING_MODEL"],
        input=instructions,
        metadata={"experiment": "repo-productivity-v1"},
    )
    elapsed = perf_counter() - started

    metrics = {
        "response_id": response.id,
        "model": response.model,
        "elapsed_seconds": round(elapsed, 2),
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
    }
    return response.output_text, metrics

This measures one API response, not the whole engineering task. The task record must still include prompting, tool execution, review, corrections, and later rework. Keep model and SDK versions with the experiment so future runs are not silently compared as if the system were unchanged.

Report at least two time views:

1. Active human time: attention the developer could not spend elsewhere.
2. Delivery time: elapsed time until the accepted change was available.

Parallel agents may reduce delivery time while increasing review load. A team can also generate more candidate work than its reviewers can safely absorb. Measure the queue, not just each run.

Count Review and Rework

Review effort is part of the workflow, even when the agent created tests. Record findings by severity and type:

review_findings:
  correctness: 1
  security: 0
  missing_tests: 2
  unnecessary_scope: 1
  style_only: 0

Use a fixed follow-up window, such as 14 days, for reopened issues, regressions, or corrective commits. This does not capture every long-term cost, but it prevents immediate acceptance from hiding obvious rework.

Perceived productivity is still worth collecting. Ask developers before and after the task whether the agent helped. Then report perception beside measured time. Earlier research found that developers' expectations and measured outcomes could diverge, which is precisely why both belong in the dataset.

Interpret a Small Study Honestly

A single maintainer and one repository can produce a useful field note, not a universal benchmark. Show individual task results instead of only an average. A few unusually hard tasks can dominate a small sample.

Task  Workflow       Human min  Delivery min  Accepted  Reopened
A     agent-assisted 24         51            yes       no
B     unassisted     38         38            yes       no
C     agent-assisted 19         76            no        n/a

Record the model, tool version, repository revision, permissions, and instructions. Agent products change quickly; a result is a dated snapshot.

Avoid converting usage into productivity. More tokens, longer agent sessions, or more concurrent runs show adoption and capacity. They do not prove that the resulting work was valuable.

Conclusion

The right question is not whether an agent writes code quickly. It is whether the complete workflow produces accepted, durable changes with less scarce human attention or shorter delivery time.

Define success before the task, include review and rework, separate active attention from wall time, and publish the limitations. A small honest field study is more useful than a large number with an unclear denominator.

References

• Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR, 10 July 2025
• We are changing our developer productivity experiment design, METR, 24 February 2026
• How agents are transforming work, OpenAI, 25 June 2026