Does more thinking buy anything?

Eight agents got the same design brief on the same night: one self-contained HTML page, an article in the visual language of a real site's landing page. Same brief, same repo, same commit, one file each. The only thing that changed between them was which model was running and how much thinking it was allowed.

Seven pages came back. They are the whole of this post, so they go first.

Sonnet 5 · 2k thinking27 turns · $0.98 · 4.8 min Sonnet 5 · 16k thinking12 turns · $0.52 · 2.9 min Opus 5 · 2k thinking13 turns · $0.84 · 3.7 min Opus 5 · 16k thinking49 turns · $2.92 · 10.1 min Opus 5 · 32k thinking12 turns · $1.04 · 5.1 min Haiku 4.5 · 2k thinking21 turns · $0.21 · 1.9 min Haiku 4.5 · 16k thinking13 turns · $0.15 · 2.2 min

Seven pages, one brief — drag sideways. Each opens the real page in a new tab.

The brief

The task file runs to 726 words and almost all of it is measurement, taken off the reference page hours before the run: three type families and no others, 11 hex colours given exactly, a 1200px column with running text at 680–720px, 96–160px of air between sections, one full-bleed inverted green band, and orange used sparingly and never on a heading. Then a contents list. Header, title, dek, byline, two subheadings, a pull quote, one figure that carries information, one block of monospace data, the green band, a footer of three related-article cards. And 800–1200 words of real prose on a real subject, with not lorem, not placeholder written into the brief in as many words.

One file. All the CSS in a <style> block, no local assets, nothing fetched but fonts, and no horizontal overflow at 390px wide. That last constraint is there because the pages were always going to be read beside each other, and a page that needs its own stylesheet sitting next to it would not survive being moved.

Then three checks, all of them mechanical: the file exists and is a whole HTML document over 4kB, every local reference in it resolves, and the page is copied out of the worktree before the worktree is deleted. Nothing in there grades taste, and nothing in there could. All seven pages passed all three.

Every run got its own git worktree and its own database: a Neon branch off one seeded parent, made when the run starts and deleted when it ends. This task never touched it — the answer here is one HTML file — but it is the same harness the schema half of this run used, and there the branches were the measurement.

The axis is who

Three models at two or three thinking budgets each. The budget is MAX_THINKING_TOKENS, the environment variable the CLI reads. Three of the pairs above differ in nothing else at all: same model, same prompt, same commit, one number moved.

Seven pages cost $6.66 of notional list price between them, and the whole grid ran in 14 minutes, three agents at a time.

More thinking did not buy more work

Take the three pairs that differ only in the budget, 2k against 16k. Sonnet went from 27 turns to 12 and $0.98 to $0.52. Haiku went from 21 to 13, $0.21 to $0.15. Both did less work with more thinking allowed. Opus went the other way, 13 turns to 49 and $0.84 to $2.92, and then at 32k came back down to 12 turns and $1.04: fewer turns than the same model did at 2k, and about a third of what it spent at 16k.

Two of the three models did less with more. The third did more and then less again, and its most expensive setting is the middle one. I can't tell you the mechanism from seven runs. The assumption underneath a lot of configuration advice is that the budget is a dial where higher costs more and does more, and that is not what happened in any of the three.

The floor is much lower than it looks

Haiku 4.5 built a page for $0.15. At the same thinking budget Sonnet 5 charged 3.6× that and Opus 5 charged 20×. Nothing in the set is further apart than those two. At 2k the gap between Haiku and Sonnet is 4.6×.

The checks cannot tell those pages apart, because the checks were never asked to. Whether the cheap one clears the same bar is a judgement, and the strip at the top is where you make it. Mine, for what it is worth. The dearest page is the best of the seven: where the brief asked for a figure, it drew a density chart with four series and a legend. The $0.15 page is one I would have accepted without knowing what it cost. The order by price is not the order I would put them in.

The eighth agent

One Fable 5 run at an 8k budget is missing.

It did not fail the brief. Its trace runs 170 messages and stops mid-turn. In the last minute of it the agent had installed fonts into the container, written a headless-Chrome script to screenshot its own page at two widths, run it, and got back desktop horizontal overflow px: 0 and mobile horizontal overflow px: 0. It was checking itself against the 390px constraint, and passing. Its last recorded action is opening that screenshot to look at it. Then the trace stops.

The runner recorded nothing about why: no exit reason, no totals, no checks, and no page, because the step that copies the page out of the worktree runs after the agent and never ran. The worktree was deleted at teardown, on schedule, with the page in it.

The cause is unproven and I would rather leave it that way than guess. The node was healthy. Rate limits are ruled out from the trace itself, which carries three rate-limit events and all three say the request was allowed. What is proven is a gap in my own tooling. The runner catches timeouts and nothing else, so any other exception kills the container without writing down what happened. This is the second write-up in a row where the honest answer to "why did that one die" is that I did not build the thing that would have told me.

What this does not say

• One run per cell. Seven pages is a look, not a finding. The non-monotonic budget result is clean enough to be worth reporting and thin enough that a repeat could overturn it. Repeating it is cheap.
• Nothing here grades design. Three mechanical checks, and seven pages you can open. If you want a quality ranking, you are the judge; I am not going to launder my taste through a benchmark.
• The pages imitate someone else's house style on purpose. Building in a measured visual language was the task, which is why the brief is 11 hex codes and three fonts instead of "make it nice". The pages are not that site's work and are not trying to pass as it.
• Every cost is notional list price, computed from token counts on a subscription that billed none of it.
• The budgets are not the models' own defaults. Setting MAX_THINKING_TOKENS at all is a choice, and 2k is a low floor for a design task.

What to do with this

If you are choosing a thinking budget by reasoning about it, stop and run both. The config change that makes a pair differing in one number is two lines, and it is the same change that made this whole grid. The result went against the intuition in all three pairs, in two different directions.

And if the cheapest model on your list clears your checks, go and look at what it built before you treat the price as a proxy for the work. That is what the row of screenshots at the top is for.

The other half of this run — thirty agents, one feature request, and what the model decided about a database schema — is written up separately: Engineering Bias in AI Models: Database Design

What I keep going back to is the Fable run. It had done the work: the last thing in its trace is the agent looking at a screenshot of its own finished page, checking the one constraint the brief was strictest about. Then the container died and took the page with it, and my runner had nothing to say about why, because a timeout is the only failure it knows how to write down. That is a one-line fix I have not made yet.