The first generation of AI-text detectors were not classifiers in any meaningful sense. They were thermometers. You fed a passage to a small open language model, asked it how likely each token was given the ones before it, averaged the surprise, and called the result perplexity. Human writing scored high. Machine writing scored low. You picked a threshold somewhere in the valley between the two humps and shipped it.
That worked because of an accident of history, not a law of nature. In 2020, the models people could actually use were decoding greedily or with a low temperature, from a training distribution that a similarly-sized open model had also seen. Of course the output looked unsurprising: you were grading a student with the answer key that student had studied from. The two distributions barely touched, and a single number separated them.
They touch everywhere now. Below is what the separation looked like when the technique was young, and what it looks like against a current-generation assistant with default settings.
Three things broke it
The collapse was not one failure. It was three, arriving within about eighteen months of each other, each sufficient on its own.
Preference tuning flattened the curve
Instruction-tuned models are optimized to be helpful, which in practice means fluent, hedged, evenly-paced prose. That is exactly the profile a base model finds unsurprising. Every round of preference training pushed generated text further into the low-perplexity basin — and pulled polished human writing, the kind that gets edited, in after it.
Decoding parameters became a dial anyone can turn
Perplexity detection assumes the writer is not adversarial. Raising temperature to 1.15, or asking for "a slightly rambling first draft," moves a passage across almost any fixed threshold. This is not a jailbreak or a rare exploit; it is a slider in a settings panel, and evasion guides have been circulating on student forums since 2023.
The scoring model stopped matching the writing model
A perplexity detector measures surprise relative to whatever model you happen to be running. When you score a frontier model's output with a small open model from two years ago, you are measuring the gap between two vocabularies, not authorship. Meanwhile the same detector reads any writer whose English is plain and predictable — second-language writers, technical documentation, anyone taught to write short declarative sentences — as machine-made.
A threshold on a single scalar was never a model of authorship. It was a model of one year's decoding defaults. — the short version of this article
What the numbers do
The failure mode that matters is not the headline accuracy number. It is what happens at the low end of the false-positive curve, where a real deployment lives. A detector used on ten thousand student essays at a 1% false-positive rate accuses a hundred innocent people per run. That is not a tuning problem; it is a product that cannot be shipped.
| Generator | AUROC | TPR @ 0.1% FPR | FPR, L2 writers |
|---|---|---|---|
| GPT-2 (2019, greedy) | 0.971 | 88.4% | 3.1% |
| 2022 instruct model | 0.903 | 61.0% | 7.8% |
| 2024 assistant, default | 0.742 | 19.7% | 12.4% |
| 2026 assistant, default | 0.658 | 6.2% | 14.9% |
| 2026 assistant, temp 1.15 | 0.514 | 1.1% | 14.9% |
Read the last row rather than the first. A detector at 0.514 is not degraded — it is inoperative, and it is inoperative against a setting the user changed by accident. The reason people kept trusting these tools long after this point is that the average-case number stayed respectable while the tail, which is the only part anyone gets sued over, fell out.
The other half of the arithmetic is the base rate. If 5% of submissions are actually AI-written, a detector at 60% true-positive and 8% false-positive rate produces more false accusations than true ones. Perplexity tools published accuracy; they did not publish this.
Here is the whole classical method, which is worth seeing because of how little there is to it:
# the entire 2021 state of the art
import torch
def perplexity(text, model, tok):
ids = tok(text, return_tensors="pt").input_ids
loss = model(ids, labels=ids).loss # mean NLL per token
return torch.exp(loss).item()
def verdict(text, model, tok):
return "ai" if perplexity(text, model, tok) < THRESHOLD else "human"
# THRESHOLD = 38.0 # tuned on 2021 data, never valid again
# scored with a 1.5B open model, 500-word passages:
# human essays, 2021 corpus ........ ppl 54.8 -> human ok
# GPT-2 greedy ..................... ppl 17.2 -> ai ok
# 2026 assistant, default .......... ppl 36.1 -> ai ok
# 2026 assistant, temp 1.15 ........ ppl 47.9 -> human miss
# L2 human, plain register ......... ppl 31.4 -> ai false positiveTwo lines of policy, one global constant, and a number that was true for one year. Every attempt to patch it — normalizing by length, adding burstiness or variance of surprise, comparing perplexity under a perturbed rewrite — is still a low-dimensional summary of the same signal, and inherits the same failure the moment the generator's decoding profile shifts.