Why perplexity-based AI detection stopped working
The metric that once separated machine text from human text at a glance now overlaps too much to trust on its own — here's what changed, and what took its place.
In 2021, catching AI-written text was almost embarrassingly easy. Run a passage through a language model, measure how surprised the model was by its own word choices, and you had your answer. Human writing is messy — full of the small, unpredictable choices that come from having a body, a mood, and a deadline to make. Early AI text was the opposite: smooth, likely, unsurprising at nearly every turn. That gap showed up cleanly in a single number: perplexity. Five years later, that number still gets computed constantly. It just doesn't do what it used to.
The simple idea that worked, briefly
Perplexity measures how well a language model predicts a sequence of text — formally, the exponentiated average negative log-likelihood the model assigns to each token given what came before it. A low perplexity means the model kept finding the actual next word unsurprising. A high perplexity means the model kept getting it wrong: the writer went somewhere the model didn't expect.
Early large language models, sampled with low temperature to stay coherent, produced text that was extremely predictable to a model of similar size — because it was, in effect, always choosing something close to the most likely next token. Human writing, by contrast, wanders. It repeats itself oddly, reaches for an unusual word, breaks a sentence it didn't need to break. Detectors from that era paired perplexity with a second signal, burstiness — the variance in perplexity from sentence to sentence — because human writing runs hot and cold in a way flat, evenly-confident machine text does not. Together the two signals drew a boundary that was, for about two years, good enough to ship.
Perplexity was never a property of the text. It was a property of the gap between the text and one particular model's expectations. Close the gap, and the signal disappears with it. — from this article
What changed
The boundary didn't erode gradually. It collapsed for four specific, identifiable reasons — and all four were downstream of decisions made by the people building generation models, not the people building detectors.
Decoding got noisier on purpose
Default sampling moved away from greedy and low-temperature decoding toward nucleus sampling, higher temperatures, and repetition penalties — tuned specifically because low-perplexity output reads as robotic. The same change that made chatbots sound more natural made them statistically harder to distinguish from people.
RLHF optimizes for approval, not likelihood
Instruction-tuned models are shaped by human preference, not by the base model's own probability distribution. A response can be highly preferred by raters while being far less predictable to the base model than the raw pretraining objective would have produced — the two objectives pull the output's statistics in different directions.
Humanizer tools target the metric directly
Once perplexity became the known detection signal, a class of tools appeared whose entire job is paraphrasing AI output to raise it — swapping common words for less common synonyms, breaking up uniform sentence lengths, reintroducing the irregularity that low-temperature generation had smoothed away.
There is no longer one reference model
Perplexity is only meaningful relative to a specific scoring model. In 2021 there were a handful of generators to check against. Today's text may come from any of dozens of model families, each with its own statistical fingerprint, plus human editing on top — so a single reference model's perplexity score answers a narrower question than it used to.
None of this means perplexity became useless. It means perplexity stopped being sufficient as a standalone decision rule. A number that used to sit on one side of a clean line now sits, for a large share of real text, somewhere in the crowded middle — and a detector that still draws a hard threshold through that middle will misclassify a predictable share of both careful human writing and lightly-edited AI output.
# a 2021-style detector — a single scalar, a single threshold def is_ai_generated(text, reference_model, threshold=28.0): score = perplexity(text, reference_model) return score < threshold # true if "too predictable" # the same rule against 2026 text, unmodified >>> is_ai_generated(sample_a, gpt_ref) # human, tired, on deadline True # false positive >>> is_ai_generated(sample_b, gpt_ref) # AI, high-temperature, paraphrased False # false negative
Both failures come from the same root cause: a threshold tuned against one generation regime, applied to text produced under a different one. The rule didn't get worse at math. The world it was measuring moved.
What replaced a single number
Modern detection doesn't retire perplexity — it demotes it to one input among many, none of which is trusted alone. The systems that hold up in 2026 combine signals that fail independently, so a text that beats one check still has to clear the rest.
Stylometric fingerprints look at structure a paraphraser rarely touches — punctuation rhythm, clause depth, transition-word habits — patterns that survive a synonym pass. Cross-model agreement scores the same passage against several reference models at once, because a single writing sample rarely fools all of them the same way. Retrieval and provenance checks ask a different question entirely: has this exact passage, or something close to it, been seen at generation time, independent of how it reads. And every score gets calibrated against known human baselines from the population actually being screened, so a non-native English speaker's careful, formal writing isn't scored against a native-speaker curve it was never going to match.
- perplexity (single reference model)~1 of 6 signals
- cross-model agreementweighted higher
- stylometric fingerprintweighted higher
- provenance / retrieval matchoverrides on hit
- population-calibrated thresholdper cohort
The part that still holds up
The intuition behind perplexity wasn't wrong. Predictability really is a useful trace of how text was produced — it's just no longer a trace one model, one score, and one hard line can read reliably on its own. Treating a single scalar as a verdict was always a simplification; it worked for a while because the gap it measured happened to be wide. The gap narrowed because it was profitable for it to narrow, on both sides — models trained to read as human, and tools built explicitly to push the score across the line.
That's the durable lesson, more than any specific technique: a detector built around defeating one known signal will eventually be defeated by someone optimizing directly against that signal. The systems that hold up are the ones that keep changing what they measure, calibrate against who they're measuring, and never let a single number stand in for a verdict.