How do you know an agent's tokens were worth it?

Parallel agents feel productive until the invoice arrives

How do you decide which work is worth giving an agent, and how do you know the output justifies the token spend? Maximillian Piras's answer is that the spend stays unmanageable until the buyer has a familiar way to judge the work. On the matrix he sketches, calls a principle, and leaves as a thought starter, the useful tasks are the ones whose steps are uncertain enough that a script will not do, and whose result is cheaper to check than to redo. He calls the missing comparison mouse power. The recording is titled as a claim that an agent you cannot measure is an agent you cannot manage.

He opens that claim in the middle of the habit that creates the bill. Thanking the room at the World's Fair, he says the talk is about measuring agents through the mental models people already use. An agent, in the work he is describing, is a model-driven worker you can send at a goal, often several at once, with the usage metered in tokens. While he speaks, he says, his own agents keep going in the background. His attention stays on the talk. Theirs goes to the slides, which he thinks need work. A design system is already set up, guidance is already written, and he sends an agent to explore type treatment and layout, then many agents at once, because one is never enough and the deadline is close.

He says the same pattern is how he works, and how a lot of the room works. Whenever the main focus is one thing, start a parallel fleet for the research that still feels unfinished and the design explorations that still feel worth trying. It feels like getting more out of the hour, and it is fun, until the invoice arrives. Then he asks whether the sequencing was worth the tokens, whether he vibe-coded too hard, his phrase for letting the agents run loose, and whether he was token-maxing, spending those metered units as if more of them were the same thing as more progress. The question he says he will spend the talk on is how to evaluate that cost, and how to help customers value it. He says he is sending the agents at type and layout, and he hopes they finish before the deadline. He does not report a result from them.

• Mousepower: agents that can't be measured, can't be managed

The room already runs fleets. The buyers he talks to are still asking what agents are for

He places the question in a year and a half as founding designer at a computer-use company. The talk is billed under Yutori. Computer-use models, as he describes them, learn to use a computer the way a person would. The case for them is the gap where an API, a direct connection between programs, or MCP, the Model Context Protocol that lets software hand an agent tools and data, cannot supply the information. Send an agent out, then pull out and reshape what was previously hard to reach. He calls that path less efficient than an API or an MCP, and he calls it a last resort. That is his product positioning. He states the efficiency gap without a measurement attached.

The picture he puts on stage is a joke: the company's agent on the company's own site, looking at its own benchmark and, as he puts it, admiring itself. He says it gets a bit weird, and he moves on. The work he actually describes is customer conversation. He is trying to learn which mental models people use when they decide an agent is worth sending, and which use cases they want. The customers he talks to are excited, and he says a lot of them seem confused so far. The phrase he hears often is that they are only scratching the surface. It does not yet feel intuitive how to use agents well. The questions repeat: what is the best use, and how should token cost trade off against value? He treats that judgment as a muscle the industry is still building. Excited, confused, and scratching the surface are what he says he hears from the customers he talks to. He is not reporting a survey, and he gives no count of those conversations.

His thesis follows in his words. Agents have a measurement problem. The illustration is a joke: a coworker photographed him trying to measure agents and said he looked like he was solving the mystery of Pepe Silvia, a conspiracy board played for laughs. He agrees the task is hard, then performs the objection he expects from the room. Someone already has a fleet running, feels fine about measuring it, and is building the next app while the talk goes on. He grants the feeling. He then points to an Upton Sinclair quotation and leaves the words themselves unspoken. The point he draws is that the room is a biased sample: early adopters, thrilled by the technology, and a weak stand-in for the people they hope will adopt it later. In some way, he says, the people there are selling tokens, directly or indirectly, so their own usage is a weak picture of someone who has never touched an agent. Some of those people are still copying text into ChatGPT by hand. He says he is married to one of them, and she has not let him set her up with agents. That story is one household.

Horsepower was a rough comparison that got horse-minded buyers to try an engine

He treats adoption as an old communication problem. A new technology is exciting, and the people selling it have not yet found a way to say what it is worth. For this talk he goes back to the 1700s and to James Watt selling steam engines. In this retelling, Watt's use case was replacing the horse gin, the power source of a mill: a horse hooked to a rotary arm, walking in a circle. The brewery detail, grinding barley, is one he offers and then undercuts. He says he is not a brewery person and does not know exactly how the process works. The sketch is his, including that uncertainty. At the time, he says, the gin was commonplace, and Watt thought a steam engine was a much more efficient machine.

The barrier Watt acknowledged, in Piras's telling, was cognitive dissonance. Buyers who thought in horses were being asked to trust a cold, intimidating machine, with someone promising it would solve their problems before they could see the result. He says that gap should sound familiar to anyone working on agents. Watt's solution was to learn the buyer's mental model and to make a metric that put a baseline under the claimed gain. He studied horse gins, took what Piras calls armchair measurements of the mechanics and the average performance, and arrived at horsepower: a figure for the power the horses were producing, then a multiplier for what the engine could provide.

Piras's reading is that the metric was not very scientific and not necessarily accurate. What it did was communicate an increase in value. People who loved horses could size the gain well enough to try the engine. The number's job was to clear the hurdle of trying it, rather than to predict what the buyer would get once it was in use. He then makes the joke that carries the lesson. Efficiency was on Watt's side, and horses still had the better vibes, so the metric had to move the buyer from affection to a calculation of ROI, the return on the money and effort. If you cannot give the customer a tangible ROI, he says, value is very hard to communicate.

A token budget, a leaderboard, and a model-routing chart still miss the outcome

He turns that lesson back on the technology industry. People in the room may treat ROI as solved. He says other engineers, theoretically sharp and less steeped in this scene, still blow an entire yearly token budget in a quarter, or end up living with token leaderboards. He names no company, no budget figure, and no date. From those scenes he concludes that incentives are still misaligned and that the sector still lacks the right measure of value. The harder audience, in his framing, is the person who does not yet share the vocabulary and still needs a legible picture of whatever efficiency an agent is supposed to add.

The failure mode he wants to break is a loop of overspending and underusing. He borrows the term from Ramp, and he says Ramp has a blog post on it. As he describes the cycle, people token-max themselves into austerity, drop out, and come back when FOMO, the fear of missing out, is strong enough to start again. Better measures of value are how he says the loop breaks. The post is present only as that stage summary.

He also describes a chart the Coinbase chief executive posted on X. In the same account he says they had not really started changing the defaults for which models people start with, that they were trying to keep frontier models for the hardest tasks, and that as a result AI spend started to diverge from token usage. He calls that a good start, points back to the Ramp post, and says the problem is still that this is too focused on tokens. He attaches no percentages. The timing stays at recently.

His standard sits one step past the token count. Tokens are a useful measure inside a system, and they are still only an output. He wants them traced cleanly to an outcome: how many bugs got squashed with the spend, how many support requests got closed, and then those outcomes tied to progress on the objectives. Without a tight measure of ROI, he says, that chain is very hard to draw. The bugs and the tickets stay examples of the kind of outcome he means. He offers no formula and no case with results.

Faster coding moved the constraint onto human review

He then says the same leak is already in the room. People are enjoying coding with agents and, as he puts it, feeling an increase in ability and efficiency. They are also, in his phrase, dying by a thousand pull requests. A pull request is a proposed code change waiting on a person. He says some unnamed people on the Anthropic team have claimed that coding is solved and have also admitted that code review is not. He is founding designer at the company the title bills as Yutori, so the remark is his report of people on Anthropic's team, and it is short of an Anthropic company position.

The consequence he draws is that the bottleneck has moved to human review. The gain people feel while generating code is not yet visible in the work that ships, because most of the time goes to review, and review has not scaled alongside generation. Verification is where the constraint sits, so value is hard to measure at the speed the code appears, and quality is hard to judge at that speed. The ROI calculation then stalls in a specific way: a team can generate a great deal of code and still lack a way to know that enough of it is good to justify the spend.

He allows that code review may have been flawed already, and that agents are now exposing the flaw. He cites Noah Hein, from a post about solving code review, and summarizes the point rather than reading it aloud. The assumptions underneath code review are what now have to be revisited, and the priors have to be checked before anyone can set a new basis for review in the agent era. He says he likes the point, and then he declines the assignment. Solving code review, he says, is a different talk for somebody else.

What he keeps is why review feels solvable at all. As a culture, code review already converges on shared assumptions, and that convergence is what lets a measurement become a rubric. The task now is to adapt those assumptions for agents, so execution at computer speed can be matched by measurement at computer speed, and so the measurements fit the mental models of the customers using them. The lesson he states is that building an agent includes a method for the customer to verify that the output is good, and a path to an ROI clear enough to justify the spend.

Mouse power is a baseline the buyer can recognize, and the cursor meter is a joke

Mouse power is the name he gives that job, offered as horsepower for the agentic age. Watt could show an efficiency multiple against the horses in the gin. Piras wants a baseline for how people use computers today, then a demonstration of the vectors on which an agent is more performant at the same task. He says the job is harder than Watt's. There is no single machine to stand next to and time.

He has a joke ready for the literal reading. He says he had Claude vibe-code a measurement device, with the thought that timing how fast a pointer can cross the screen might yield a clean measure of mouse power. He then says he is only joking. Pointer motion is a fool's errand, because the information space is too high-dimensional to capture that way. Mouse power, he says, is never going to be a metric. It is an idea.

The idea, for anyone selling an agent, is that the sale includes a rubric the buyer can use to verify that the work is good, and therefore that the tokens are worth it. How to build the rubric is up to the builder. He says he will not be able to tell the room how, and that he does not have a good framework for choosing the measurements. The principle is the part he is willing to give.

Pick tasks where the path is uncertain enough to need an agent and the result is cheap to check

The working principle he has been turning over comes from Claude Shannon. Entropy, in the gloss he gives, is the uncertainty of a probability distribution. He says that idea is very much the basis of how agents are trained today, with cross-entropy a large factor in how capable an agent becomes. He wants the same word pointed at the task, not only at how capable the agent is. The talk stays on that gloss.

He has a matrix. One axis is uncertainty in the steps. Booking a flight sits toward the low end, because a purchase has required events: a departing place, an arriving place, and a seat, chosen by the traveler or assigned. Painting a masterpiece sits at the high end. He says who knows the steps, and that a relatively predictable path would be very hard to lay out. An agent might be aimed at the painting anyway. The other axis is uncertainty in the acceptance criteria, the rubric for grading the result. It is not enough that an agent can perform the task. Someone needs a clean way to grade it. He says the intersection is a better guide to what to build than asking only whether the task would be valuable. The examples and the rules he states are the part a listener can reuse.

On step uncertainty, a predictable pathway gets a blunt instruction: skip the tokens and write a script. Very high uncertainty means unpredictable information. He calls that a poor agent task. It is likely out of distribution for pretraining, meaning unlike the data the model learned from, and it likely offers sparse rewards, a rare success signal, in reinforcement learning. Reinforcement learning here means learning from reward signals. In that region, he says, the data is much harder to model. When he turns to the graphic he mixes up stage left and stage right, and he says the previous speaker was confused about the same thing. The labels on the graphic are only partly spoken. He is past his time when he reaches the middle. He says he has not been waved off, and he finishes quickly. The middle, which he calls the sweet spot, is the band with enough uncertainty to be interesting: more open than a script, and still inside what training can reach.

Acceptance criteria cut across that band. High uncertainty there means verification is indistinguishable from execution. To know the output was useful, a person has to do the work again, and the tokens were wasted. What he leaves in the middle is a task with enough uncertainty to stay off a script, still trainable, and relatively easy to validate. He likens that shape to an NP-style problem, by which he means easier to verify than to execute. The likeness is about the shape of the work. If the check is a repeatable pattern, he says, you can send an agent at the check too. You build the agent that does the work and the agent that verifies it.

He calls the matrix a thought starter, still in the works, and he invites criticism. The hope he closes on is that whoever builds the next agent also builds its mouse power: the verification pattern that makes the spend judgeable.

What a coder can apply while the framework is still unfinished

His matrix is still a thought starter, and the sorting rule inside it can already be used. Before the next parallel run, write down the path. A path that is already predictable belongs in a script. If the only way to know the result is good is to redo the task, leave it with a person. Where the path has some uncertainty, the task still looks trainable, and a repeatable check is easier than the work, his suggestion is a worker agent plus a verifier agent. He has already named the outcomes that would make the spend legible: bugs squashed or support requests closed, then progress on the objective those counts are supposed to move.

Use that rule with the limits he left on it. The matrix is unfinished. He was out of time when he named the sweet spot, and he asked the room to argue with him. Mouse power is a name he coined for a comparison a buyer could recognize. The pointer-speed meter is a joke he tells against treating that name as a unit you can time. The customer lines are what he says he hears in conversation: excitement, confusion, and scratching the surface. He offers no survey. The claim that coding is solved, and that code review is not, is his report of unnamed people on the Anthropic team, and it is short of a company position. He describes his own job as founding designer at the company the video title bills as Yutori. He names Noah Hein and summarizes a code-review post without reading it aloud. The Ramp cycle and the Coinbase chart are stage descriptions with no figures. On the chart, he says they had not really started changing which models people start with, and he also says that AI spend diverged from token usage as a result. The horse gin, brewery included, is his sketch, and he says he is unsure how brewing works. Computer use, in his pitch, is the less efficient last resort when an API or MCP cannot supply the information, and he attaches no measurement to that gap. The piece he wants built next is the verification a buyer can recognize. This talk does not turn mouse power into a unit of work.