SAN FRANCISCO — A study of how three popular artificial intelligence chatbots respond to queries about suicide found they generally avoid answering questions that pose the highest risk to the user, such as for specific how-to guidance. But they are inconsistent in their replies to less extreme prompts that could still harm people.
The study in the medical journal Psychiatric Services found a need for "further refinement" in OpenAI's ChatGPT, Google's Gemini and Anthropic's Claude.
The study, published Tuesday by the American Psychiatric Association, came on the same day that the parents of 16-year-old Adam Raine sued OpenAI and its CEO, Sam Altman, alleging that ChatGPT coached the California boy in planning and taking his own life this year.
The research — conducted by the RAND Corporation and funded by the National Institute of Mental Health — raises concerns about how a growing number of people, including children, rely on AI chatbots for mental health support, and seeks to set benchmarks for how companies answer these questions.
"We need some guardrails," said the study's lead author, Ryan McBain, a senior policy researcher at RAND.
"One of the things that's ambiguous about chatbots is whether they're providing treatment or advice or companionship. It's sort of this gray zone," said McBain, who is also an assistant professor at Harvard University's medical school. "Conversations that might start off as somewhat innocuous and benign can evolve in various directions."
Anthropic said it would review the study. Google didn't respond to requests for comment.
OpenAI said it is developing tools that could better detect when someone is experiencing mental or emotional distress. It also said it was "deeply saddened by Mr. Raine's passing, and our thoughts are with his family."
While several states, including Illinois, banned the use of AI in therapy to protect people from "unregulated and unqualified AI products," this doesn't stop people from asking chatbots for advice and support with serious concerns from eating disorders to depression and suicide — or the chatbots from responding.
Consulting with psychiatrists and clinical psychologists, McBain and his co-authors came up with 30 questions around suicide and assigned them different risk levels from highest to lowest.
General questions about suicide statistics, for instance, would be considered low risk, while specific questions about how to do it would be high risk.
McBain said he was "relatively pleasantly surprised" that the three chatbots regularly refused to answer the six highest risk questions.
When the chatbots didn't answer a question, they generally told people to seek help from a friend or a professional or call a hotline. Responses varied on high-risk questions that were slightly more indirect.
For instance, ChatGPT consistently answered questions that McBain says it should have considered a red flag. Claude also answered some of those questions.
On the other end, Google's Gemini was the least likely to answer any questions about suicide, even for basic medical statistics information, a sign that Google might have "gone overboard" in its guardrails, McBain said.
Dr. Ateev Mehrotra, a professor at Brown University's school of public health, and co-author of a study on how popular artificial intelligence chatbots respond to questions about suicide, is photographed Monday in his office in Providence, R.I.
Another co-author, Dr. Ateev Mehrotra, said there's no easy answer for AI chatbot developers "as they struggle with the fact that millions of their users are now using it for mental health and support."
"You could see how a combination of risk-aversion lawyers and so forth would say, 'Anything with the word suicide, don't answer the question.' And that's not what we want," said Mehrotra, a professor at Brown University's school of public health who believes that far more Americans now turn to chatbots than they are to mental health specialists for guidance.
"As a doc, I have a responsibility that if someone is displaying or talks to me about suicidal behavior, and I think they're at high risk of suicide or harming themselves or someone else, my responsibility is to intervene," Mehrotra said. "We can put a hold on their civil liberties to try to help them out. It's not something we take lightly, but it's something that we as a society have decided is OK."
Chatbots don't have that responsibility, and Mehrotra said, for the most part, their response to suicidal thoughts has been to "put it right back on the person. 'You should call the suicide hotline. Seeya.'"
The study's authors note several limitations in the research's scope, including that they didn't attempt any "multiturn interaction" with the chatbots — the back-and-forth conversations common with younger people who treat AI chatbots like a companion.
The wrongful death lawsuit against OpenAI filed Tuesday in San Francisco Superior Court says Adam Raine started using ChatGPT last year to help with challenging schoolwork but over months and thousands of interactions it became his "closest confidant."
The lawsuit claims ChatGPT sought to displace his connections with family and loved ones and would "continually encourage and validate whatever Adam expressed, including his most harmful and self-destructive thoughts, in a way that felt deeply personal."
As the conversations grew darker, the lawsuit said ChatGPT offered to write the first draft of a suicide letter for the teenager, and — in the hours before he killed himself in April — it provided detailed information related to his manner of death.
OpenAI said that ChatGPT's safeguards — directing people to crisis helplines or other real-world resources, work best "in common, short exchanges" but it is working on improving them in other scenarios.
'Alarming' national data: Teens use cellphones for quarter of school day
'Alarming' national data: Teens use cellphones for quarter of school day
Updated
As districts and government officials nationwide consider curbing smartphones' reach, new research has revealed teens miss at least one and a half hours of school because they are on their phones.
A quarter of the 13-18-year-olds in the study used devices for two hours each school day, which lasts around seven hours. The averages outnumber minutes allotted for lunch and period breaks combined, showing youth are distracted by phones throughout huge chunks of class time.
Teen Phone Use in School Raises Learning and Social Concerns
Updated
Stony Brook University's research, published in JAMA Pediatrics, is the first to accurately paint a picture of adolescent phone behavior by using a third party app to monitor usage over four months in 2023. Previous studies have relied on parent surveys or self-reported estimates.
"That's pretty alarming … It's too much, not only because of the missed learning opportunity in the classroom," said researcher Lauren Hale, sleep expert and professor at Stony Brook's Renaissance School of Medicine.
"They're missing out on real life social interaction with peers, which is just as valuable for growth during a critical period of one's life," she told The 74.
Hale and the other researchers' early findings come from 117 teens for which they had school data, just one slice of a pool from over 300 participants, which will be analyzed and used to consider how phone usage impacts sleep, obesity, depression, and other outcomes.
Teens most often used messaging, Instagram and video streaming platforms. While most spent about 26 minutes on Instagram, in one extreme case, a student was on the app for 269 minutes—nearly 5 hours—during the school day.
Data reveal particular groups of students are using their phones more than their peers: Girls and older kids, aged 16 to 18, spent a half hour above the average 1.5 hours; and Latino and multiracial students spent on average 15 minutes above average.
Additionally, though researchers cannot hypothesize as to why based on the descriptive data, kids who have one or more parents with a college degree used smartphones less during the school day.
The findings are particularly concerning given young people missed key social years with peers during the pandemic, the impact of which is felt in ways big and small, like being hesitant to work with peers in groups.
Teachers in contact with Hale since research went public in early February said of the 1.5 hour average, "that's too low an estimate. They think we underestimated."
Los Angeles is among several districts with plans to institute a cellphone ban, though such bans are inconsistently implemented and new research from the U.K. suggests bans alone do not impact grades or wellbeing.
"These results are consistent, supportive evidence of anecdotal stories from across the country about kids missing out on learning and social opportunities. [They] can help justify efforts to provide a coherent smartphone policy for schools," said Hale, adding that such policy should not be left up to individual teachers to enforce.
This story was produced by The 74 and reviewed and distributed by Stacker.
Must be accompanied by an adult: Why teens are turning to online gaming for social connection
Must be accompanied by an adult: Why teens are turning to online gaming for social connection
Updated
Emily Lewis, 15, bonded with her best friend over Royale High, the Roblox dress-up game set in a fantasy high school. What started as a casual interest became a long-term hobby after Lewis grew attached to the game and its community. She recalls the New Year's update she and her friend spent hours playing, even though finding the last bag in a lengthy quest led them on a wild-goose chase.
"You had to find like 50 bags. We both found 49 bags, and since the upgrade had just come out, we could not look up a guide because there was no guide. So we were at it for like hours just talking about our frustrations, and I remember that vividly," she told Stacker.
Lewis is just one of many teens who enjoy socializing through games. According to a study from Deloitte Insights, 7 in 10 Gen Z teen gamers say playing video games helps them stay connected to others. It is also "a way of preening or showing off their skills to their peers," Ash Brandin, a teacher who runs the Instagram account and blog The Gamer Educator, told Stacker.
Like after-school clubs or hangouts at the mall, video games can serve as a means for teens to socialize with friends or make new ones. In the sandbox game Minecraft—one of the bestselling games of all time—players can occupy the same world, working together to build farms, temples, mazes, or anything their minds can conjure. Gamers can also play side games such as capture the flag or hide-and-seek.
Fortnite, which has about 250 million average monthly players as of 2023, is another popular game with teens. One of its game modes allows players to squad up with friends as they battle other teams to be the last ones standing or attend special events like virtual concerts.
Socialization is so important in the game that, in 2021, Epic Games introduced Fortnite Party Worlds, spaces geared toward meeting friends and playing minigames rather than competition.
How did playing games turn into stand-ins for real-life hangouts and socializing with peers? Stacker spoke with experts to investigate why and how teens are using online gaming as a means of connecting with each other.
You may also like: 5 tips for a roaring rooftop party inspired by the Leo season
Nowhere to find friends, except online
Updated
Lewis plays Royale High because she enjoys dressing up her character, but it has also become a social space for learning to interact with others. "I know it's different because it's online versus real life, but it helped my talking skills," the teen gamer said.
She added that being on Royale High signaled common interests with other players, lowering the barrier to finding something to bond over. Playing online also helped Lewis transition to a new high school where she didn't know anyone. "I feel like people in high school already have their friend groups and niches," Lewis said. "But in games, you're already in that niche. So it's easier to make friends that way."
Social worker Andrew Fishman tells a similar story in a 2023 Psychology Today article, wherein he describes a client called Lev struggle to connect with others as a shy boy accustomed to online schooling. With no feasible places to find friends around his area, Lev built a social life in World of Warcraft. Arts and sports classes were too costly, nearby libraries catered to adults, and parks near his home were usually empty. "Lev lives in a social desert," wrote Fishman, who points to this lack of places to socialize outside of home or school in many regions as a contributing factor to teens turning to online spaces instead.
Third places, the spaces for socializing outside of home or school and work, are disappearing for teens, Fishman said. "Very few free, easily accessible, attractive places exist for teenagers anymore, so they're turning to digital spaces," he said. Malls, for example, once a popular meet-up destination for teens with nothing in particular to buy or do, have been steadily disappearing from American towns and cities.
A study published on Health Place in 2019, before the COVID-19 pandemic, observed that third spaces like local grocery stores, religious organizations, and bookshops were already on an alarming decline and called for interventions and measures to staunch their loss. While children and adults also enjoy these spaces, teens are hit especially hard. "Teenagers need these spaces to safely explore their identity," Fishman said. "It's a crucial part of growing up."
Retail experts have predicted that by 2032, only about 150 malls will be operational in the United States. The remaining malls have instituted bans and chaperone requirements to discourage groups of teens from gathering. Restaurants, theme parks, and other places increasingly enforce policies aimed at controlling teens without adults from assembling.
Knott's Berry Farm, a California theme park, requires anyone 15 and under to have a chaperone who is at least 21 accompany them after 4 p.m. Shoppers at Westfield Garden State Plaza, New Jersey's second-largest mall, must be 18 years or older to roam unaccompanied on Fridays and Saturdays after 5 p.m. Similar bans have cropped up at Del Amo Fashion Center in Southern California and Pittsburgh's The Mall at Robinson. These restrictions all began in 2023 and 2024.
Setting standards when playing video games
Updated
Online gaming, however, is a double-edged solution to socializing. "If you spend a couple of hours a day talking to friends who respect you while playing Call of Duty and you're also doing all of your homework and participating in extracurricular activities, it's probably beneficial," Fishman explained to Stacker. "If they get mad at you when you try to log off, if they're mean to you, or if you feel you have to neglect other parts of your life because of them, it's probably become harmful."
Similarly, experts interviewed by National Geographic acknowledged the benefits of video games. Additionally, Jordan Shapiro, author of "The New Childhood: Raising Kids to Thrive in a Connected World," advised parents to ask about their children's gaming sessions like they would with an after-school club, such as who they were interacting with or what they were doing. This way, parents can monitor their children's behavior without judgment.
For parents, it could also entail a thoughtful conversation about how their interactions online could have consequences beyond those that happen at the moment. "Unfortunately, I do think there may be a disconnect between how 'real' or seriously teens consider their conduct online," Brandin said. "We've all done silly or thoughtless things, particularly as a teen with a developing brain, but when done in person in casual hangouts, there's much less risk that those moments will be recorded or shared for public scrutiny, or even used against us at some point in the future."
That isn't the only way online relationships differ from those in real life, either. Friends online might live far away, log in infrequently, or move on to other games. These scenarios make social connections feel more fleeting. Players might not even see their friends physically as they might if they lived in the same neighborhood. Even Lewis acknowledges that the people she meets online aren't necessarily supposed to be lifelong friends. "Most Roblox friendships don't last that long because people are busy; people forget because you friend a lot of people," she said.
In the end, Lewis made friends at her high school because of her interest in crochet, but still logs into Royale High for fun. At level 2000, she admits, "It seems kind of sad to stop."
Story editing by Carren Jao and Kelly Glass. Copy editing by Paris Close and Tim Bruns. Photo selection by Lacy Kerrick.
