AI time savings depend on checking effort and errors, preprint argues
An arXiv author reanalyzes developer-task data and reports that reducing AI review effort can save clock time while allowing more errors through.
Measuring AI’s time savings requires accounting for errors that escape review, not just elapsed time, argues an arXiv preprint that has not been peer reviewed. The author distinguishes time recorded on a stopwatch from a counterfactual measure that weighs saved time against defects left uncorrected.
Reanalyzing public data from a developer study by METR, the author reports that the same 246 issues support estimates of a 19% or 24% slowdown, depending on the estimand. Without adjustment for a chance imbalance, the reported slowdown is 34%. The author estimates that AI helped on somewhere between 3% and 82% of the issues, showing how much the result depends on the measure used.
In a small, pre-specified experiment, AI reviewers checked AI-written briefs based on SEC filings collected after the models’ training data. When reviewers were told the drafting system made errors ten times less often than it actually did, they checked 7 fewer of 9 figures and let a wrong figure pass in 22.5% of briefs, compared with none when given the true error rate. The author attributes the faster review to reduced checking effort, alongside more errors. The paper also says review time alone limits GPT-5’s speedup to 3.6x in GDPval’s workflow, which requires review of every output.