Can AI be too emotional? China introduces new regulations [Beijing Chronicles]

China has rolled out a set of regulations to curb dependency on AI-powered companion bots. 

Transcript

SPEAKER_00 0:02

Experience love beyond reality with Aye Girlfriend. Ignite passion and romance with AI boyfriend. Hello and welcome. This is Beijing Chronicles. My name is Razia Ustman. Now, China on Wednesday rolled out a set of regulations aimed at curbing emotional dependency on AI-powered companion boards, marking a new step in efforts to regulate this rapidly growing sector. AI companion services, which have been used in areas ranging from customer service and mental health support to childcare and elderly care, have gained popularity by simulating human personalities and conversations. But concerns over emotional dependency and other risks have fueled calls for higher regulation. The new rules require platforms to detect emotional distress, intervene in crisis situations, limit excessive use, give users full control over their personal data, and prevent the misuse of user information. The regulations define AI companion services as those offering sustained emotional interaction through text, images, audio or video, while excluding task-oriented AI applications such as customer service, work assistance, education, and scientific research. The rules, China's first dedicated effort to regulate AI-based anthropomorphic interactive services, were jointly formulated by the Cyberspace Administration of China and four other government bodies. China's major AI content generation platforms, including Dubao, Gwen, and Yuanbao, have recently announced the suspension of their AI agent features. On clearer boundaries, AI companion boards, a swiftly growing segment of the AI landscape, have become part of many people's daily lives, but problems are beginning to surface. These virtual companions can create deep emotional bonds with young users seen as particularly vulnerable to overuse and retreat from real-world relationships. In 2025, the China Youth and Children Research Center surveyed more than 8,500 minors across the country. The results showed that over 60% had used AI, while more than 20% said they only wanted to chat with AI and did not want to talk with real people. Said Wang Shen Chen, an associate researcher at the Department of Psychology, School of Social Development and Public Policy at Fudan University in Shanghai. Experts warn that prolonged immersion in the conflict-free communication zone created by AI could weaken people's ability to navigate real-world relationships, potentially leading to a broader decline in social connection. To address this issue, the new rules explicitly prohibit AI systems from encouraging addictive use, requiring mandatory reminders after more than two hours of continuous use, pop-up interventions when signs of dependency are detected, and intervention mechanisms in cases of extreme emotional distress. To protect users' interaction data, the document requires providers of AI-based interactive services to adopt measures such as data encryption and access controls and prohibits them from sharing users' interaction data with third parties without consent unless otherwise required by law. It also stressed that users have rights over their interaction data, including the ability to copy or delete chat histories and other records of past interactions safeguarding their legitimate interests. While protecting users' privacy and personal information, the regulations also call for the establishment of risk intervention mechanisms to prevent extreme behavior. Zhang Jiyu, the executive director of the Law and Technology Institutes at Remmin University of China, said technical tools could help identify potential risks and enable timely interventions. For general risks, AI systems could offer supportive responses while in more serious cases they could alert designated contacts. Such mechanisms, he said, could help provide stronger safeguards as they are adopted more widely. China's AI market has continued to expand, driven by broader adoption across industries. It was expected to surpass 1.2 trillion yuan, that's about $177 billion in 2025, and is tipped to exceed 1.8 trillion yuan by 2028, according to the latest research. In recent years, China has strengthened AI legislation and established a multidimensional regulatory framework covering areas like data, algorithms, and applications. Chen Liang, the dean of the School of Artificial Intelligence and Law at the Southwest University of Political Science and Law, located in Chongqing municipality, said the new regulations provide an important opportunity for the industry to upgrade and transform. The rules will encourage companies to shift their focus from maximizing user immersion and engagement to developing healthier and more responsible human-AI interaction models, thereby helping channel resources toward socially valuable applications and supporting the industry's orderly development. And that's it for this edition of Beijing Chronicles. My name is Rosia Othman. Thanks for listening.