Substack’s new AI detector is solving the wrong problem

In 1770, a Hungarian civil servant named Wolfgang von Kempelen built a chess-playing machine for the entertainment of the Austrian empress and called it the Turk.

It sat behind a cabinet full of gears, took on Benjamin Franklin and Napoleon, and won most of its games for the better part of 80 years before anyone proved what a few skeptics had long suspected: there was a man folded inside the cabinet the whole time, working the pieces by hand.

The trick wasn’t the chess but letting people believe a machine was doing something only a mind could do, and watching them marvel at the mind’s absence instead of noticing it. Substack’s chief executive, Chris Best, opened a July 21 blog post with an 18th century engraving of the Turk, which tells you what Best thinks he’s wrestling with now.

"The Turk," engraving by Joseph Racknitz, 1789. The chess-playing automaton fooled European royalty and heads of state for decades before its human operator was discovered.

His post, titled “Against Claudefishing,” announced that Substack is rolling out AI-detection scores across its app, in partnership with a company called Pangram, which builds software that estimates whether a piece of writing came from a person or a model.

Readers can scan a post, a note, a reply, a comment, anything over 100 words, and get back an estimate of the split between hand and machine. Claudefishing, the term Best coined, means reading something because you believe a person made it, then finding out it was written by AI.

He’s careful, in the post, to say the problem isn’t AI itself, and it isn’t quality, since plenty of competent prose now comes from AI and plenty of soulless prose still comes from a person typing at a keyboard.

What he’s after is the gap between what a reader expects and what they’re really getting, especially when they don’t know there’s a gap at all. He points to a line from Freddie deBoer, a writer and cultural critic who publishes his own Substack.

“I access human-made art because I know there’s a human behind it and that’s what I’m looking for, other humans, showing me in art what they hide in their selves,” DeBoer write. “Fooling me in that process is just a con.”

But the tool Substack is utilizing to catch that “con” cannot, in fact, catch it. What Pangram measures is whether a sentence’s rhythms resemble a language model’s. What Best says he’s trying to protect is something else entirely: whether someone had the idea, did the reporting, would stand behind the argument if pressed. Those two things overlap only by accident. A score can tell a reader that a tool touched the words, but it cannot tell them whether anyone was home.

The numbers Best is working from are his own company’s. Pangram said it spent this spring scanning more than a million posts across LinkedIn, Medium, Substack, X and Reddit, and the results, published as a report earlier this month, are the kind of thing a communications team either buries or leads with.

More than 40% of LinkedIn’s longform posts came back fully AI-generated, and LinkedIn alone accounted for nearly two-thirds of all the AI content Pangram found anywhere on the internet it looked. Part of that is by design. LinkedIn has its own AI writing button built into the composer, rebranded from “Write with AI” to “Enhance post,” and it nudges people toward using it every time they sit down to post.

Substack came out cleanest among the longform platforms and was the only one where a longer post wasn’t more likely to be machine-written than a short one. Still, a fifth of Substack’s own posts flagged as AI-generated or AI-assisted, which is probably the real reason a company that could have pointed to its comparatively good numbers and said nothing decided to act instead.

The two platforms are choosing opposite bets: one is building tools to make AI writing easier to produce, the other is building tools to make it easier to spot. Pangram’s own researchers, in the same report, allowed themselves a rare moment of bluntness about where all this is headed if nobody does anything: an internet fully overtaken by undisclosed AI writing, they wrote, would be “bleak.”

There is also, not incidentally, money in the answer. Substack has spent the last year courting institutions the way a magazine courts advertisers, and it now counts the State Department, Tory Burch and the venture firm a16z among the organizations publishing through it as an owned channel, according to Axios’s reporting on the launch.

The trade publication framed the detection push as a wager, essentially, that subscribers, and the sponsors who follow them, will pay a premium for the assurance that a human being sat down and wrote what they’re reading. That’s a commercial argument wearing an editorial coat, and it goes a long way toward explaining why Substack moved first, on a problem it has less of than its rivals, instead of waiting to be forced into it.

The feature itself is underwhelming. You scan something over 100 words, and a percentage comes back. Writers get a companion tool, a “How I make this” statement where they can describe their process in their own words, plus the ability to run their drafts through Pangram before publishing and to flag scans they think got it wrong.

Writers can also turn scanning off entirely for a given post or note, so readers see “AI detection unavailable” instead of a score. At launch, doing that required generating a Pangram analysis first, then disabling it from the results menu. Within days, after writers objected, Substack dropped that requirement, letting publishers disable scanning without running the scan at all. There’s still no single switch to turn it off across an entire publication; it’s decided post by post.

Best has said more is coming: letting readers set preferences about how much AI content they want recommended to them, among other things.

The trade press covering the rollout seemed to notice, more than anything, how unusually combative the announcement itself was; Engadget called it “surprisingly spicy,” pointing to the swipe Best takes at LinkedIn along the way, a company he does not name so much as gesture at with palpable contempt.

The trouble is what happens once the score exists. Substack’s own post concedes, in a clause, that Pangram can’t detect care, only syntax. That caveat sits well below the headline. What ships is a number on a screen, and a number gets read as a verdict no matter what the fine print says underneath it.

That dissonance surfaced almost immediately in the replies underneath Best’s post. A writer named Monica Hebert, who has published on Substack for a year and a half, argued that no reader had ever accused her of running “slop,” AI use included, because she still supplies the judgment, the memory, the argument, everything a detector has no way of scanning for.

Her sharpest sentence, and the one that drew the most agreement underneath it: “The presence of AI does not prove the absence of a human.”

Other critics have made a harder version of the same argument: that a detection score doesn’t just miss the point, it actively harms the people it’s pointed at. Matteo Wong, writing in The Atlantic back in May, warned that Pangram’s mistakes are more frequent than most people realize, and that the company is, in his words, “accumulating the power to end reputations and careers.”

The concern is valid. A widely cited study by the researcher Weixin Liang and colleagues found that earlier AI detectors mislabeled essays by non-native English speakers as machine-written at a far higher rate than essays by native speakers, because non-native writing tends toward the simpler, more predictable sentence patterns that a classifier, trained mostly on native writing, reads as generated.

The dispute has already had a real casualty list. Pangram’s research flagged several opinion columnists at the Wall Street Journal as likely AI users, and the paper’s editorial-page editor, James Taranto, ran the same columns back through the tool himself, got different results each time, and published a rebuttal arguing the accusations didn’t hold up under his own testing.

Asked about the odds of any individual writer being wrongly flagged, a doctoral researcher at the University of Pennsylvania who studies these detectors told Slate that for most people, the chance is close to zero, though not quite.

“For most people, they might never, ever get a false positive,” he said.

Close to zero, applied across a platform with Substack’s volume of writers, still produces a working list of people it happens to.

One Substack writer took the objection further, reaching for Arthur Miller rather than statistics. Writing under the name The Slow AI, the writer compared the new feature to Salem, Massachusetts: a system that begins from suspicion and treats a person’s inability to prove their innocence as evidence of guilt.

“A detector installs that machinery in your reading app,” the writer argued, while adding, pointedly, that they use these tools themselves and don’t think the underlying worry about trust is wrong. The objection isn’t that detection has no place at all. It’s that turning it into a button under every post recasts every writer, by default, as someone who has to prove a negative, regardless of what the accuracy numbers eventually say.

This is the trap that every version of this argument keeps falling into, on every platform that has tried it. The harm Best describes, a reader believing there’s a mind behind a piece of writing when there isn’t one, is a question about intent and authorship.

Pangram answers a different question entirely: does this text share statistical fingerprints with model output. The two correlate, but only loosely. A writer who drafts by hand and lets a model tighten one paragraph will sometimes score as AI-assisted. A writer who simply thinks in a flat, generic register will sometimes score as fully human, on a piece with no more thought in it than the machine-written one sitting next to it in the feed.

And the company handing out that number has its own reasons for wanting people to trust it.

Worth saying plainly: Pangram is not a neutral referee here. It is a company that sells AI detection, and the research Best is citing as evidence of the problem is Pangram’s own research, marketing its own product to the exact anxiety it was built to profit from. That doesn’t make the underlying numbers false, but it does mean that a firm with a financial stake in AI panic being widespread is not the same thing as an independent auditor, and Best’s post treats it, throughout, as though it were.

Readers do want to know what they’re getting, and any platform that lets fakeness pass unremarked will eventually lose the writers who make it worth paying for.

“When I want Claude’s opinion,” Best ended his post, “I’ll ask Claude.”

That is a disclosure problem, not a detection one, and Substack already built the better tool for it in the same announcement. The “How I make this” statement hands the explanation to the person who did the work, instead of outsourcing the judgment to a classifier trained to notice phrasing.

There isn’t a clean fix for any of this, and maybe there doesn’t need to be one. Nobody asks a painter which reference photo they worked from, or a novelist which earlier book handed them the plot.

What matters is the finished product, not the tools that got it there. These tools are only going to get better, and the instinct to treat every one of them as a confession waiting to happen may not survive contact with that fact.

Someone who learns to use them well has learned a skill, the same as anyone who learns to write a clean sentence by hand. Substack built a machine to catch the con it’s worried about. The bigger question is whether the con is the thing to worry about at all.