Anthropic bans 'abuse' of Claude — but critics say the policy goes too far

AI company Anthropic has banned persistent, unprovoked abuse of its AI model Claude, igniting a debate over whether the move promotes healthier user habits or amounts to excessive anthropomorphization of a system whose capacity for feeling remains unproven.

Anthropic revised its usage policy Thursday to add "persistent and unnecessary cruelty or abuse toward the model" to its list of prohibited behaviors, according to BBC and other international outlets. The updated policy takes effect Nov. 12.

The new clause was added alongside existing prohibitions on harassment of others, promotion of self-harm, and the creation of non-consensual intimate imagery. Anthropic drew a line, saying the provision applies only to extreme cases of repeated, purposeless cruelty. The company said ordinary user frustration, pushback, creative writing on dark themes, and model testing or research would not be covered.

When a violation is detected, Anthropic said it will act through a conversation-ending feature introduced in August last year, which allows Claude to terminate a session if abuse persists. The company said this feature would remain its primary enforcement tool going forward.

However, Anthropic did not spell out what specific behaviors would constitute abuse, nor did it clarify whether additional penalties — such as account suspension — could follow a conversation being cut off.

The revision also touched on election and opinion-manipulation provisions. The company added a new section addressing deceptive campaigns using fake accounts or impersonation of media outlets, and tightened its election-related rules to focus on prohibiting voter deception and interference with the electoral process.

When Anthropic introduced the conversation-ending feature last year, the company said the moral status of AI was "deeply uncertain." The move was part of what it calls "model welfare" research — the idea that if the possibility of AI suffering cannot be ruled out, low-cost protective measures should be put in place first. The term "model welfare," however, does not appear in the new policy document.

On social media, some users welcomed the policy as a step toward more courteous AI use and a meaningful advance for model welfare.

Pushback has been fierce. Barry Scannell, a technology partner at Irish law firm William Fry, wrote on LinkedIn that "you can't be cruel to numbers and mathematics," adding, "This level of AI anthropomorphization is harmful. It makes people believe AI is something it is not." Concerns also surfaced on social media that extending the language of abuse to AI could dilute the meaning of harm suffered by humans and animals.

Mustafa Suleyman, Microsoft's head of AI, responded to the announcement with a "melting face" emoji. He had previously criticized Anthropic's AI training approach in a BBC interview last month, saying it could have a catastrophic impact on human well-being.

The debate touches on a broader, contested question: whether users should be polite to generative AI. Some argue that phrases like "please" and "thank you" waste tokens unnecessarily, while others contend that courteous prompts produce better responses.

OpenAI CEO Sam Altman addressed the question last April, responding to a user on X who asked about the electricity cost of processing such pleasantries. "Tens of millions of dollars well spent," Altman said. "You never know."

Hostility toward AI has its own following. The term "clanker," used as a derogatory label for AI, went viral on social media last year and was shortlisted for Collins Dictionary's Word of the Year. Some experts have warned that habits of treating AI chatbots dismissively could spill over into how people communicate with one another.

Ultimately, the debate centers on how far user behavior can be regulated when there is still no answer to whether AI can experience suffering. As long as Anthropic declines to define what constitutes "abuse" or what penalties apply, the controversy is unlikely to die down.