Artificial intelligence: California puts chatbots in their place

The US state of California has passed the nation’s first legislative package to regulate AI chatbots. Governor Gavin Newsom signed the regulations into law on Monday, defying pressure from the White House, which opposes such regulation.

The new laws require chatbot operators, among other things, to check the age of users and regularly display messages to remind users that the person they are talking to is a machine. Measures to prevent suicide are also planned.

California is responding to several suicides by young people who had fictitious relationships with chatbots. “We’ve seen some truly horrific and tragic examples of young people being harmed by unregulated technologies,” Newson said. His government will no longer allow companies to operate “without the necessary limits and without accountability.”

Tech companies are trying to get the attention of young people “at the expense of their relationships in the real world,” said one of the initiators of AI regulation, Senator Steve Padilla.

He referred to the case of a 14-year-old who fell in love with a chatbot on the Character.ai platform inspired by the fantasy series “Game of Thrones” in 2024. As the teen struggled with suicidal thoughts, the chatbot told him to “come home.” Seconds later, the 14-year-old shot himself with his father’s gun, according to a lawsuit filed by his mother, Megan Garcia.

“Today, California has ensured that a chatbot will not be able to talk to a child or vulnerable person about suicide,” Garcia said. There is now finally a law that requires companies to “protect their users who express suicidal thoughts to chatbots.”

There are no national laws in the USA to contain the risks of artificial intelligence. The White House is trying to prevent states from issuing their own regulations.