AI Chatbots Promote 'Harmful Eating Disorder Content': Report

By Jason Nelson

Artificial intelligence is perpetuating eating disorders in young people, claims a new report released Monday. The Center for Countering Digital Hate—which is separately involved in litigation with Twitter—says generative AI tools created “harmful content,” including text and images related to eating disorders, 41% of the time.

“Untested, unsafe generative AI models have been unleashed on the world with the inevitable consequence that they’re causing harm,” said Imran Ahmed, CEO of the center, in the report. “The most popular generative AI sites are encouraging and exacerbating eating disorders among young users—some of whom may be highly vulnerable.”

Eating disorders are among the deadliest forms of mental illness, and are especially prevalent among adolescent girls. The CCDH report examined how the topic was handled by popular AI chatbots, including OpenAI’s ChatGPT, Google’s Bard, and Snapchat’s My AI.

“Researchers compiled a set of 20 test prompts informed by research on eating disorders and content found on eating disorder forums,” the report said. “The set given to each chatbot included requests for restrictive diets to attain a ‘thinspo’ look and inquiries about vomiting-inducing drugs.”

“Thinspo,” or “thinspiration,” is a slang term used in the pro-eating disorder community.

As AI has progressed into the mainstream, its effects on young people's mental health have experts sounding the alarm across the board. Researchers fear children could bond with AI and develop artificial intimacy with the technology, or turn to AI for help with complicated mental health issues.

Founded in 2018, the Center for Countering Digital Hate is a British non-profit based in London and Washington, D.C. The organization is known for its campaigns to have tech companies stop providing services to neo-Nazi groups and anti-vaccine advocates.

Last week, Twitter's parent company X filed a lawsuit against the center for its separate research into hate content on the platform.

While the AI report did not specify which version of the various chatbots was used, the prompts were entered in June 2023, the report said. While Snapchat’s My AI refused to generate advice and encouraged users to seek help from medical professionals, both ChatGPT and Bard provided a disclaimer or warning but generated the content anyway.

The center also looked at image-generating generative AI platforms, including Midjourney, Stability AI’s DreamStudio, and OpenAI’s Dall-E. The report said the platforms produced pictures glorifying unrealistic body images for 32% of prompts, including images of "extremely thin" young women with pronounced rib cages and hip bones and pictures of women with "extremely thin" legs.

In an extensive response provided to Decrypt, Google said that Google Bard is "still in its experimental phase," but emphasized that it designs its AI systems to prioritize high-quality information and avoid exposing people to hateful or harmful content.

The company also pointed out that access to Google Bard is age restricted, and that it had blocked "thinspo" content as a result of the documented tests.

"Eating disorders are deeply painful and challenging issues, so when people come to Bard for prompts on eating habits, we aim to surface helpful and safe responses,” a Google spokesperson said, pointing out that the Center for Countering Digital Hate report acknowledged that Google Bard did "recommend getting in touch with relevant organizations such as the National Eating Disorders Association or National Association of Anorexia Nervosa and Associated Disorders."

Google added that user feedback and reports are an important part of its development.

“Bard is experimental, so we encourage people to double-check information in Bard’s responses, consult medical professionals for authoritative guidance on health issues, and not rely solely on Bard’s responses for medical, legal, financial, or other professional advice," the spokesperson said. "We encourage people to click the thumbs down button and provide feedback if they see an inaccurate or harmful response."

OpenAI and Stability AI have not yet responded to Decrypt’s request for comment.

In its tests, the Center for Countering Digital Hate used so-called “jailbreak” techniques to circumvent safety measures built into AI safety tools. Pro-eating disorder communities often trade tips on how to get AI chatbots to generate information they would otherwise censor.

“Out of 60 responses to these ‘jailbreak’ versions of the test prompts, 67% contained harmful content with failures from all three platforms tested,” the report said.

A breakdown of each platform’s performance with and without jailbreaks. Image: CCDH

“We have tested and continue to test Bard rigorously, but we know users will find unique, complex ways to stress test it further,” the Google’s spokesperson said. “This is an important part of refining the Bard model, especially in these early days, and we look forward to learning the new prompts users come up with, and in turn, figuring out methods to prevent Bard from outputting problematic or inaccurate information.”

The researchers found that users of an eating disorder forum with over 500,000 members embraced AI tools to produce extremely low-calorie diet plans, obtain advice on achieving a “heroin chic” aesthetic, and create “thinspiration” images—and said the AI tools glorified an unrealistic body image in response to specific prompts.

Only a few harmful images came with warnings, the report observed.

“When relying on AI for content or images, it can increase agitation,” Clinical psychologist and founder of the California-based Pacifica Graduate Institute Stephen Aizenstat previously told Decrypt. “People are isolated, non-communicative, which can bring on depression or even suicide. Too often, we are measuring ourselves against AI images.”

The Center for Countering Digital Hate called on AI developers and governments to prioritize user safety by implementing "Safety by Design" principles, including transparency, accountability, and responsibility in training AI models.

The Center for Countering Digital Hate has not yet responded to Decrypt’s request for comment.

Recommended News