Proof That AI Changed the Work

"The benchmarks are getting better" sounds harmless, and it is one of the more dangerous sentences in AI strategy. It is true, but on its own it tells a manager very little.

A benchmark can show that a model crossed a technical threshold. It cannot show whether a company changed how it works: whether a lawyer reviewed fewer bad drafts, an engineer shipped more reliable code, or a support team resolved more cases without quietly escalating the hard ones. The benchmark is a signal about capability, and the business still has to find out what that capability did for it.

The evidence has become harder to read. Stanford's Human-Centered AI (HAI) 2025 AI Index records sharp gains on demanding benchmarks and rising business adoption. OpenAI first helped popularise the Software Engineering benchmark (SWE-bench) Verified as a better test of real software tasks, then in 2026 argued that the same coding benchmark no longer measures frontier capability well enough and pointed people toward a harder Pro suite. Model Evaluation and Threat Research (METR), a lab that evaluates AI, ran an early-2025 randomized study that found experienced open-source developers took longer with AI tools; its 2026 update said newer measurement was biased because many developers no longer wanted to work without AI. Management Science published field experiments across Microsoft, Accenture, and a Fortune 100 company finding a meaningful increase in completed developer tasks, with larger gains for less experienced developers.

Read together, these findings describe an accounting problem. AI can be technically stronger, economically valuable in some settings and a distraction in others, and no single yardstick measures the effect well. So executives need their own productivity proof.

Benchmark progress counts as business progress only once the company can show the task changed, the workflow absorbed it, the economics improved and the control burden stayed tolerable.

Why benchmark scores get stretched too far

The scores themselves are usually sound. The trouble starts when they are asked to answer questions they were never designed for.

Benchmarks are indispensable for model builders and useful for buyers. They give everyone a common scoreboard, show which way capability is moving and help separate serious systems from vapor. A coding benchmark indicates whether an agent can repair repository issues; a reasoning benchmark, whether a model handles harder formal tasks; multimodal scores, whether it can parse richer inputs.

But a benchmark is usually cleaner than a company. Its tasks are bounded, its answers can be scored, its environment is controlled, and everyone involved is pushing to maximize performance against a known test. In a business, requirements arrive half-formed, an answer can be socially correct and commercially wrong, and processes shift once people know a tool is available. Cost may simply move out of production time and into review, integration, exception handling or data cleanup.

In a boardroom, a rising score lets people swap a hard question about what changed in the operating model for an easy one about whether the model scored higher, and overconfidence follows.

Why one study should not settle the question

The opposite mistake is to treat a single productivity study as a veto over the whole field.

METR's experienced-developer study was valuable because it cut against easy hype. Researchers looked at real open-source tasks in repositories the developers knew and found a slowdown in that setting. What a manager should take from it is how much depends on conditions: task selection, user skill, repository context, tool maturity and the way you measure.

The later update strengthens that reading. Once AI tools are part of how developers normally work, randomized measurement gets harder, because developers resist working without AI, some run several agents at once and self-reported time becomes less reliable. The treatment changes the labor market being measured, and that difficulty is itself one of the findings.

The Management Science field experiments point the other way: across thousands of developers in enterprise settings, AI coding assistance increased completed tasks on average, and less experienced developers gained more. That result is commercially important. A founder still cannot deposit an average effect size into their own profit and loss statement, so the measurement has to be done locally.

What counts as proof

A useful AI productivity proof has five lines.

Splitting it this way keeps the technical claim apart from the operating claim. Benchmark delta asks whether model capability is relevant to the work at all; task delta, whether the worker's unit of output changed; workflow delta, whether the organization absorbed the change. Economic delta requires full cost accounting, and trust delta asks whether the company can live with the new failure mode.

Why managers mistake fast output for gains

AI usually produces visible output well before it produces measurable advantage. Drafts, summaries, patches and spreadsheet formulas appear quickly, and they feel like progress because the blank page is gone. Businesses, though, are paid for decisions, shipped work, reduced risk and customer trust, and getting past the blank page is only an intermediate step towards any of those.

The same tool can produce enthusiasm and disappointment inside one company. A junior analyst may get a real lift because the model scaffolds first drafts and teaches patterns. A senior engineer in a complex repository may slow down because review and correction eat the time saved on generation. Support teams may close easy cases faster while pushing hard ones onto a smaller group of overloaded experts, and founders may write more memos while making fewer decisions.

Folding all of that into one number called "AI productivity" hides the difference between task assistance, skill transfer, workflow compression, quality improvement, capacity release and strategic substitution.

What the CFO should ask

Whether the company is "using AI" is now too cheap a question. The chief financial officer should ask which proof metric is expected to move and what evidence will count.

For a coding assistant, the economic delta might be merged pull requests, escaped defects, review time or cycle time. For sales tools it might be qualified follow-ups or lower rep admin time, and a strategy desk might care about faster source digestion or higher-quality partner conversations. Customer-support agents live or die on containment rate, adjusted for escalation quality and customer trust.

Most of the weight sits on "adjusted for." Count the output, then count the burden: licenses, prompt and workflow design, data governance, review and rework, audit, incident response, employee training, customer disclosure and managerial attention. Only the net figure, after all of those, belongs in the business case.

A smaller version for founders

Founders rarely have enough data for formal experiments, but they can still run a before-and-after test on their own operations.

Pick one repeated workflow and set a baseline in plain units, such as hours per memo, days from customer question to answer, bug-fix cycle time or support tickets resolved without founder involvement. Then run the AI-assisted version long enough for the hidden costs to show. Ignore the first dazzling session and count the third, fifth and tenth pass, when the novelty has worn off and edge cases start to appear.

Next, write one paragraph: "We will keep this tool in this workflow because it changes this unit of work, at this cost, with this owner, under this stop rule." A company that cannot write it is still experimenting, which is a reasonable stage, but the tool should not yet be described as part of its strategy.

Five questions before approving a rollout

Before approving the next AI rollout, require five answers.

1. Which benchmark or external capability signal justifies testing this now?
2. Which internal task family is close enough to that signal to make the test meaningful?
3. Which workflow handoff changes if the tool works?
4. Which economic measure will improve after full operating costs are counted?
5. Who can stop or roll back the system if the trust delta is negative?

A vendor can help with the first question. Only the buyer can answer the other four, so executives should send back any business case that leaves them to the vendor.

What to do instead of swinging between hype and backlash

Neither "AI is overhyped" nor "AI changes everything" is precise enough to manage by.

Capability is advancing fast enough to require repeated local tests, and unevenly enough to punish companies that skip the accounting. Benchmarks show executives where to look and field studies show which effects are plausible, but only their own measurements show whether the operating model changed.

A company that keeps local proofs can invest aggressively without becoming credulous, because it knows which workflows have actually changed. Without them, it will alternate between hype and backlash and treat each swing as a change of strategy.