Tokenmaxxing Is Not Efficiency: How to Measure AI Productivity

Apparently I am passionate about this topic. Before this post existed, I had already written a full essay for one of the CEOs in my peer group arguing against token utilization as a target KPI, unprompted and at length. When you find yourself writing internal memos to other people’s companies about a metric, it is probably time to make the argument in public.

Volumetric measures of AI effectiveness are the wrong measures. Token counts, lines of code, prompts sent, agents created: none of these proxy the thing we actually want, which is efficiency. Publish any one of them as a target and it stops measuring anything at all, because it is far easier to move the number than to move the outcome underneath it. Tokenmaxxing is the purest form of this failure, a team optimizing the cost line because the cost line is the only number anyone bothered to instrument.

Prukalpa, co-founder of Atlan, made a version of this argument in AI Slop Produced by a Human Is a Yellow Flag. The post lists every AI number the company can recite: licenses provisioned, weekly active users, tokens consumed, agents created, hours supposedly saved, artifacts shipped. Investors tell Atlan it sits in the top 1% of AI adopters. Her argument is that those numbers measure whether people are using AI, not whether the company is getting more capable at anything. That is correct. What I want to add is the mechanism, because once you see the mechanism you can predict which replacement metrics will fail the same way.

The bias to action manufactures the appearance of progress

Here is what we are actually finding in practice. AI is extremely helpful at moving through the bulk of the workload. The long-term planning is still up to the human, and that division has held steady across everything we have built.

The reason is that these models carry an extreme bias to action. The model wants to get something done, even when the something is not the long-term target goal. Give it an ambiguous instruction and it will produce a plausible artifact rather than stop and ask which artifact belongs in the plan. It will pick a direction because picking a direction is what it does.

That bias produces a lot of motion very quickly, and motion reads as throughput on every volumetric metric you could point at it. What the metric does not capture is that the work did not disappear, it moved. It moved onto you, in the form of time and energy spent planning how to deal with the output over a longer horizon.

Time horizon is the whole trick. Inside a one-week frame the capability looks magical, because you see the artifacts and have not yet paid for the ones that pointed the wrong way. Stretch the frame to a quarter and the planning debt is visible. This is the mechanism behind the slop Prukalpa describes: a bias to action pointed at a volumetric target produces polished output that nobody needed, which then becomes context for the next person or the next model to build on.

Goodhart’s law comes for the AI dashboard

Goodhart’s law holds that once a measure becomes a target, it ceases to be a good measure. Volumetric AI metrics are unusually exposed to this, because the distance between the metric and the deliberate act of inflating it is close to zero. An engineer measured on token utilization will utilize tokens. A marketing team measured on artifacts shipped will ship artifacts. Neither has to lie, and neither has to become more effective. Combine that with a model that already wants to produce something, anything, and tokenmaxxing is not even a decision anyone makes. It is the default trajectory of the system.

The reason companies reach for these numbers anyway is not stupidity. It is availability. Efficiency is hard to measure and counting is easy, so people count. Tokens get reported because a token is a well defined unit that a system already logs for billing purposes, and the thing anyone actually wants to know does not come with a logger attached.

That is the deeper problem. The thing you are trying to measure may not be measurable at all, and I mean that literally rather than as a complaint about tooling. Efficiency in knowledge work is a judgment about whether the right problem was chosen and whether the solution was worth its cost, and both halves of that judgment depend on a counterfactual you never get to observe. You cannot log the version of the quarter where the team built something else. Any number you produce is a proxy standing in for a comparison that never happened, which is why every proxy in this space eventually gets gamed. We treat that as a gap in our instrumentation when it is closer to a property of the work.

Software engineering never solved this either

Look at how software teams judged themselves before any of this. Most success criteria reduce to whether the release shipped on time. But estimation is one of the hardest problems in the discipline, so the criterion rests on a number that was itself a guess. Who defines on time, and who defines success?

If the estimate was generous, a team ships early and looks efficient. If a stakeholder redefined the scope in week three, the same team ships late and looks slow. The measure inherits every ambiguity in the estimate that produced it, and no amount of precision downstream repairs that.

This matters for the AI conversation because it sets a realistic baseline. We could not measure engineering efficiency cleanly before agents arrived. Adding tokens consumed to the dashboard does not close that gap, it just puts a confident number next to an unresolved question.

The incentives point away from the outcome

We want efficiency. We do not care how many tokens it takes to get there. Stated that plainly, the problem with tokenmaxxing as a success metric is obvious, because it inverts the goal. A team that reaches the same outcome on half the tokens has done better work and scores worse.

Token consumption belongs on the cost side of the ledger, next to warehouse spend and seat licenses. Reporting it as a KPI means reporting a cost as an achievement. The same inversion applies to lines of code, which has been a discredited productivity measure since long before a model could generate them, and to hours supposedly saved, which is almost always a modeled number rather than an observed one.

What a mid-market data leader should do instead

This problem bites hardest in the mid-market, because that is where the board is asking for AI ROI numbers and there is no dedicated measurement function to push back on a bad one. Someone has to put a slide together, the billing dashboard is right there, and tokenmaxxing gets institutionalized in an afternoon.

Start by giving up the clean number. There is no drop-in replacement that is both precise and meaningful, and pretending otherwise is how you end up back at tokenmaxxing with extra steps. A coarse measure that carries judgment beats a precise measure that carries none.

Three things are worth instrumenting:

• Survival. What share of AI-generated output was still in use 30 days after it shipped. The artifacts that got discarded a week later were never productivity.
• Cost per completed outcome. Dollars per resolved ticket, shipped feature, or validated model. This puts tokens where they belong, as an input you want to minimize.
• Decisions changed. Name the consequential decisions AI changed this quarter, and hold yourself to three to five real ones rather than a count.

To be clear, these three are not immune to Goodhart. Survival can be inflated by never deprecating anything, and a decision narrative can be dressed up after the fact. The difference is the cost of faking them. Inflating token counts is free, while keeping unused artifacts alive for 30 days and defending a decision story in front of people who were in the room both cost real effort, roughly as much effort as doing the work honestly. That is the actual selection criterion for a metric in this space: not whether it can be gamed, but whether gaming it costs about as much as achieving the outcome it proxies.

Budget human planning capacity as the scarce input, because it is. If you double agent throughput without expanding the attention available to direct it, you have bought motion and will pay for it in a quarter. This is the same arithmetic that produces the maintenance trap, where a team accelerates generation and falls further behind on everything downstream of it.

Finally, put a name on every artifact. For the last thing your team shipped, you should be able to say which human chose the problem, which human accepted the quality, and which human decided it deserved other people’s attention. Keep tracking the token burn, because you cannot compute cost per outcome without it. Just keep it on the cost side of the report, next to warehouse spend, where nobody has ever been tempted to celebrate a bigger number. If the achievement side of your reporting talks about survival, outcomes, and decisions, and tokens only ever appear as a denominator, you are measuring the right things.