Character.AI has rolled out new safety features and policies for building and interacting with the AI-powered virtual personalities it hosts. The new measures aim to make the platform safer for all users, but particularly younger people. The update includes more control over how minors engage with the AI chatbot, more content moderation, and better detection of the AI discussing topics like self-harm.
Though not cited in the blog post about the update, Character AI linked to the announcement in a post on X expressing condolences to the family of a 14-year-old who spent months interacting with one of Character.AI’s chatbots before taking his own life. His family has now filed a lawsuit against Character.AI for wrongful death, citing a lack of safeguards for the AI chatbots as a contributor to his suicide.
We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family. As a company, we take the safety of our users very seriously and we are continuing to add new safety features that you can read about here:…October 23, 2024
AI chat guardrails
Character AI’s post laid out several new safety features for the platform. For instance, if the model detects keywords related to suicide or self-harm, it will display a pop-up urging the user to the National Suicide Prevention Lifeline and related resources. The AI will also be better at spotting and removing inappropriate content in a conversation, with a particular sensitivity to when users are under 18.
Presumably, minors would already have restricted content in conversations, but Character.AI may have upped that sensitivity further. In cases where that might not be enough, entire chatbots have been removed.
“We conduct proactive detection and moderation of user-created Characters, including using industry-standard and custom blocklists that are regularly updated. We proactively, and in response to user reports, remove Characters that violate our Terms of Service,” Character.AI explained in its post. “Users may notice that we’ve recently removed a group of Characters that have been flagged as violative, and these will be added to our custom blocklists moving forward.”
Other new features are more about helping ground users. So, you’ll see a notification when you have spent an hour on the platform asking if you want to keep going as a way of helping make sure you don’t lose track of time. You’ll also see more prominent disclaimers emphasizing that the AI is not a real person. There are already such disclaimers in the conversations, but Character.AI wants to make it impossible to ignore.
These safety features are the flipside of how Character.AI has made engaging with chatbots feel more like talking to a real person, including voices and the two-way voice conversations available with the Character Calls feature. Still, the company is likely keen to ensure its services are as safe as possible, and its moves could inform how others in the space shape their own AI chatbot characters.