The world is familiar by now with the usual tropes of machine-generated text: overuse of the word “delve”, an excess of em dashes, and the chirpy, relentless construction of “it’s not X but Y”.
But is it about to get even worse?
Over the weekend, Anthropic released an update saying it would change how its Claude AI model generated prose. This is to comply with an EU regulation that requires all AI-generated text to be watermarked starting in December.
Anthropic said the changes would be made at the granular, random level at which its models generated text and would be undetectable to the average reader.
But at least one commentator thinks otherwise. “This entire endeavour is a perverse adulteration of what it means to write,” wrote John Gruber, a veteran tech blogger. He argued the watermark would constrain Claude, forcing it to make worse, less precise word choices overall. While it may not make the model’s writing less accurate, he suggested these limits would make it worse.
There is a stochastic element in the small choices that an AI model makes in framing a sentence: whether it chooses to call a day “grey” or “overcast”, for example, or refers to a running water as a “stream” or a “brook”.
Anthropic’s new watermark will alter these random choices, it said, leaving a pattern that will be detectable to Anthropic itself, and to those who have a key to decode it.
Steven Murdoch, a professor of computer science at University College London, said the change “probably wouldn’t have any noticeable impact”.
Gruber’s complaint appears to centre on the fact that watermarking will make a large language model less free to decide which word comes next in a sentence, and may therefore not pick the best choice.
However, LLMs already do not make the best choices. “There’s already randomness involved in any of these large language models,” said Murdoch. “It’s pretty essential to how they work. If it wasn’t for this randomness, then they’d get stuck in loops and start repeating the same thing over and over again.”
In other words, a chatbot does not necessarily decide to call running water a “stream” as opposed to a “brook” because the latter might have a more old-fashioned register. The model does not contemplate these choices: it makes them by chance.
This was evidenced recently when a paper in a leading chemistry journal had to be retracted because the authors appeared to have used an AI tool to draft part of the publication. The tool used the phrase “mass killing of an ethnic group” as an alternative for “final solution”, though the paragraph in question appeared to be about a zinc nanogel.
On the update, Murdoch said: “There’s going to be no noticeable difference. There’s the same random number generators there – it just used to be completely random, and now it’s statistically predictable, but still random.”
The regulation, which applies to all AI companies operating in the EU, means they will have to put in place these watermarks within months. This could make it harder for students, lawyers and university professors to pass off chatbot-written content as their own.
There is another reason to watermark AI content, though, Murdoch said: there’s so much of it already out there that it might damage the models themselves. Training AI on AI-written content creates “model collapse”, leading models to confuse concepts.
In other words, watermarking is not just a quietly powerful way to combat disinformation – it’s a deeply transformative system to ensure the chatbots don’t go insane. If you delve into it.
