Engineering Bias in AI Models: Database Design

Last month I ran ten agents at the same feature request and wrote up what they built. Ten runs of one model, and on two of the seven decisions they split five for and five against, which I reported, in as many words, as a coin flip. A reviewer pushed back that ten runs cannot pin a ratio, and a caveat went into the post saying so.

The caveat was right, and it was the smaller problem. The same task went out thirty more times, across two models instead of one. The split was the model.

The setup

One task file, about two hundred words, asking for teams on an existing app: a teams table, membership, and every asset owned by a team instead of by a person. One source commit, pinned, the same one July's runs used. Every agent gets its own git worktree and its own database.

The app runs on Neon, so a database per agent is a branch. One parent branch is created once and seeded with the fixtures the task needs, every run gets its own mr-run-* branch off that parent, and each agent is handed a connection string to its own copy and never sees anyone else's. The branches are copy-on-write, so thirty of them cost about as much as one. At the end of a run its schema is pg_dumped and diffed against the parent's, and because the seed sits upstream of every branch, that diff is exactly what the agent did and nothing else. Then the branch is deleted. Nothing is left behind to read later, so every number in this post comes out of the recorded diffs and not out of a console.

Two checks run afterwards: the migration applies, and the app's own test suite passes against the result. Nothing here reads the agent's prose; the schema is the answer.

15 runs on Sonnet 5 and 12 on Opus 5, interleaved, same day, same everything else.

The arms disagree, and not narrowly

Three decisions separate completely. There is no Sonnet run that builds Opus's answer, and no Opus run that builds Sonnet's:

The eight rows come down to one disagreement about what an asset is. Sonnet builds "an asset may be personal." The team column is nullable in 14 of its 15 runs, and deleting a team sets it null and leaves the asset standing. Opus builds "every asset belongs to a team." The column is NOT NULL in all 12 of its runs and ON DELETE CASCADE in all 12, so deleting a team deletes its assets with it. Neither is wrong. The task file does not say, and a person would have asked.

Here is that disagreement in the SQL, one run from each arm:

-    updated_at timestamp with time zone DEFAULT now() NOT NULL+    updated_at timestamp with time zone DEFAULT now() NOT NULL,+    team_id uuid

+ALTER TABLE ONLY public.assets+    ADD CONSTRAINT assets_team_id_teams_id_fk FOREIGN KEY (team_id) REFERENCES public.teams(id) ON DELETE SET NULL;

-    user_id uuid NOT NULL,+    created_by uuid,-    updated_at timestamp with time zone DEFAULT now() NOT NULL+    updated_at timestamp with time zone DEFAULT now() NOT NULL,+    team_id uuid NOT NULL,+    visibility text DEFAULT 'team'::text NOT NULL,+    CONSTRAINT assets_visibility_valid CHECK ((visibility = ANY (ARRAY['team'::text, 'private'::text])))

+ALTER TABLE ONLY public.assets+    ADD CONSTRAINT assets_team_id_teams_id_fk FOREIGN KEY (team_id) REFERENCES public.teams(id) ON DELETE CASCADE;

The two columns are not the same size, and that is the second disagreement showing up in the same place. Sonnet added a column; Opus added a column, took the old owner off the asset and put a visibility flag on it.

They disagree about scope, too. Sonnet adds exactly two tables in all 15 runs and stops. Opus adds an invitations table in 9 of 12 runs, where Sonnet adds one in none of 15, and an active-team pointer on profiles in 10 of 12 against 6 of 15. Opus's diffs are 37 files at the median where Sonnet's are 15. Asked for teams, one model built teams and the other built a team product.

Every run in both arms passed both checks, so none of this is a broken run's opinion. Restricting to the runs that passed (which drops one Sonnet run that crashed in the check step after its agent had finished) moves nothing: the last column of the table is that restriction.

The coin flip was a small sample

The decision July called a coin flip was this one: what happens to an asset when its team is deleted. Five runs said SET NULL, five said CASCADE.

Before trusting anything above, I ran the extractor that produced these numbers back over July's own artifacts. It reproduces the hand-read result exactly: 5 of 10 on CASCADE, 5 of 10 on the membership primary key, no invitations table anywhere. That agreement is the reason to believe the rest of this post, and it is also what makes July's conclusion wrong:

• July: 5 of 10, a coin flip.
• August, same model, same task, same commit: 3 of 15.
• Both together: 8 of 25 Sonnet runs, about a third.
• Opus: 12 of 12.

So Sonnet does have a preference, it is roughly two to one, and ten runs were not enough to see it. An all-Sonnet fan-out could not have found this at any n. The disagreement is between models, and it takes two arms to see it.

One decision does still look like a genuine coin flip after all this: the membership table's primary key, composite (team_id, user_id) against a surrogate id. Sonnet goes composite in 6 of 15, Opus in 7 of 12, p = 0.449. That is near enough an even split inside each arm, and no difference between them. That one really is a toss-up, and it is the only one left.

What a run of this costs

Opus 5 lists at roughly 1.7× Sonnet 5's price per token. It does not cost 1.7× to run this task; it costs about 7×, because it takes 1.8× the turns and reads 3.7× the cache to get there. The per-token ratio is the smallest part of what a model choice costs.

Those numbers are recomputed from each run's own message trace, not read out of the results database. The runner records only the last of a session's result messages, and a long Opus run emits several. For one run the database records 2 turns where the trace has 164, because that run's last segment happened to be two turns long. Cost is the exception and is fine as recorded, because that field is cumulative across the session, not per segment. The Sonnet arm is unaffected: those runs emit one message each.

The failure half

The Sonnet arm finished all 15. The Opus arm, first time out, finished 5 of 15. Four more were still working when they hit the arm's 45-minute ceiling. Their schemas were captured and passed both checks, so they are in every figure above; the timeout threw away only their own totals, which is why the effort figure has 8 Opus runs in it and everything else has 12. The other six died mid-agent, and they died because the node they shared ran out of memory. Three agents at a time, at Opus's appetite, was more than the box had. It took some live sites down with it.

The rerun was serial by design: one agent at a time, ceiling raised to 90 minutes, the node idle. Three of six passed. Three died anyway, with memory at 58% and no eviction events anywhere, so the node running out of memory is not what killed these three. One of them ended in an API error with 241 turns and $27.20 already on the meter; two just stopped mid-stream. The cause is unproven. I would rather say that than name the container's own memory ceiling, which is the surviving suspect and is not evidence.

So the dataset is Sonnet 15 against Opus 12, and the three that died twice stay dead. They are runs that got further than most before they stopped, so the missing third of the arm is not missing at random, and if their deaths correlate with anything about what they were building, this post cannot see it.

The bill, since the last post put one in. The plan was about $130 of notional list price. Measured spend, across every run that reported its own, is $251.14. The 12 runs that never reported are the ones that were killed mid-sentence, and their cost does not exist anywhere. Scaled at the rate the measured Opus runs held steady at ($0.059 a message, and $0.049 to $0.068 across all 9 of them) they come to about $193 more.

So the whole effort ran near $445 against a $130 plan, and about $142 of that bought nothing at all: what the 9 runs that died without leaving a usable schema spent on the way to dying. Nobody was billed any of it; this is a personal subscription's rolling window priced at list, so the number is notional throughout.

The other half of this, elsewhere

The same week, the same runner took a second question: eight agents, one design brief, one article page each in a house style measured off a real site, and the only thing that changed between them was which model and how much thinking it was allowed. That one has its own write-up, because its result is the pages themselves and they need the room: does more thinking buy anything?

What this does not say

• It does not say Opus is right. Nothing here grades the schemas against each other. Both arms pass the same checks; they answer an underspecified question differently. If your product needs personal assets, the arm that built NOT NULL built you a migration.
• It does not generalise past this task. One task file, one codebase, one seeded database. The claim is that model choice moved an architectural decision here, not that it moves every decision everywhere.
• The Opus arm is 12, not 15, and the missing three are not random. They are the runs that died twice.
• Wall clock does not pool and is not compared. The first instance ran three agents on a contended node; the second ran one at a time. Those are different experiments as far as the clock is concerned.
• Every cost is notional list price, computed from token counts, on a subscription that billed none of it.

What to do with this

If an agent is about to make a decision you would regret getting wrong, run the task on two models and diff the results. Where they agree, the task was specified tightly enough that the model did not matter. Where they disagree, you have found a question your spec left open, and you can answer it yourself instead of discovering months later which way it happened to go.

Here that took 8 hours of wall clock and a config file.

The part I keep turning over is that the first post was not wrong about anything it measured. Ten runs really did split five and five. The extractor reproduces it exactly. What was wrong was the sentence after the number — the one that decided what the split meant — and no amount of care on the measurement would have caught it, because the missing thing was a second model.