Measure AI Delivery Gains from Request to Reliable Release

Choose one recurring class of work, such as small API changes in an established service. Record the request, complexity, dependencies and acceptance condition before assigning a tool. Include work that stalls or is abandoned, so the scorecard does not silently count only successes.

Capture elapsed time from an agreed starting point to acceptance, human effort across implementation and review, correction cycles and post-release defects. Keep agent runtime separate from human effort: if an engineer works on another task while an agent runs, summing both as labor overstates the cost.

Use historical comparisons cautiously because task mix, staffing and release constraints change. Where feasible, compare similar work during the same period and record selection decisions. A small pilot can reveal useful bottlenecks without establishing a universal productivity rate.

Track five signals together: accepted changes, time to acceptance, human review and correction effort, escaped defects and cost per accepted change. State the observation window and defect severity rules before the pilot. Include tool charges, integration support and repeated failed runs in the cost boundary.

For an illustrative comparison, one task moves from six hours of implementation and two hours of review to three hours of implementation and five hours of review. Both require eight hours of human effort; the visible generation gain has moved work downstream. The figures describe a hypothetical example, not a Primus result.

Measure the business outcome separately. Faster reporting changes matter if users receive timely, correct reports. More pull requests, generated lines or accepted suggestions can help explain activity, but none alone proves that the service delivers more value.

METR's early-2025 randomized study found that experienced contributors took 19% longer on its selected open-source tasks when AI tools were allowed. Its February 2026 update describes selection and time-measurement problems in a later experiment and cautions that the newer data does not reliably establish the current effect. Neither result settles what a different enterprise team will achieve.

DORA's 2025 report emphasizes the underlying organizational system. Our interpretation is that local workflow evidence should guide adoption: a slow review queue may need smaller changes and clearer acceptance criteria before another generation tool can improve end-to-end delivery.

Review a sample of both successful and failed tasks with developers and maintainers. Decide whether to expand, adjust or stop the workflow, then repeat measurement when the model, permissions or task mix changes. Share uncertainty with the result instead of turning a pilot estimate into a permanent promise.