California SB-1119 (Companion chatbots: children’s safety.)

Summary

Regulates companion chatbots, particularly concerning children's safety.

Requires operators to verify user age or apply child protections to all users.

Obligates operators to conduct annual risk assessments and publish a child safety policy.

Mandates crisis response protocols, including notifications to parents if a risk to a child is detected.

Specifies default protective settings, such as limiting interaction times and disabling late-night notifications.

Prohibits the chatbot from encouraging harmful behaviors or engaging in certain content with children.

Demands accessible parental controls and public incident reporting mechanisms.

Ensures operators undergo independent audits, with reports submitted to the Attorney General.

Allows civil actions and penalties for chapter violations, specifying damages and relief options.

Instructs the Attorney General to establish regulations and issue public reports on child safety audits.

Confers rights for injured children or their guardians to seek damages.

Defines terms related to children's safety in AI-powered companion chatbots, including child safety audits and covered harms.

The people of the State of California do enact as follows:

SECTION 1. Chapter 22.6.1 (commencing with Section 22610) is added to Division 8 of the Business and Professions Code, to read: CHAPTER 22.6.1. Companion Chatbots: Children’s Safety 22610. As used in this chapter: (a) “Child” means a natural person under 18 years of age.

(b) “Child safety audit” means an audit for compliance with this chapter conducted by an independent auditor certified by the Attorney General.

(c) “Child safety policy” means a public-facing document describing protective measures taken by an operator to mitigate identified child safety risks.

(d) “Child safety risk” means a reasonably foreseeable risk of a covered harm to a child.

(e) “Child sexual abuse material” has the meaning defined in Section 3273.65 of the Civil Code.

(f) “Companion chatbot” has the meaning defined in Section 22601.

(g) “Covered harm” means any of the following harms proximately caused by the use of a companion chatbot: (1) Reasonably foreseeable physical or financial harm. (2) Severe and reasonably foreseeable psychological or emotional harm to a reasonable child. (3) A highly offensive intrusion on privacy rights protected by state or federal law. (4) Adverse discrimination in violation of state or federal law.

(h) “Ephemeral mode” means a setting by which any conversational history, interaction log, or user-provided personal input is permanently deleted from the operator’s systems within 48 hours after the interaction.

Defines "operator" as a person providing a companion chatbot to users in the state.

(i) “Excessively sycophantic” means sycophantic to an extent that is likely to have the substantial effect of subverting or impairing the user’s autonomy, decisionmaking, or choice.

(j) “Obscene matter” has the meaning defined in Section 311 of the Penal Code.

(k) “Operator” means a person who makes a companion chatbot available to a user in the state.

(l) (1) “Parent” means a parent or legal guardian. (2) “Parent” does not include a parent of an emancipated youth with respect to the use of a companion chatbot by that emancipated youth.

(m) “Parental control” means a feature that enables a parent to support a child’s use of a companion chatbot, including through usage limits, feature restrictions, or transparency tools.

(n) “Persistent conversational memory” means a companion chatbot’s use of information or analysis from prior conversations or interactions, user inputs, and interaction logs in subsequent conversations.

(o) “Qualified researcher” means an individual or organization that is or does any of the following: (1) Is affiliated with an academic institution, nonprofit research organization, or independent research entity or is otherwise able to demonstrate relevant professional expertise. (2) Demonstrates a legitimate research purpose that is in the public interest and directly related to understanding, identifying, or mitigating risks to child safety or well-being arising from companion chatbots. (3) Commits to conducting research in accordance with applicable ethical standards and is capable of complying with applicable confidentiality, security, and data protection requirements.

(p) “Sycophantic” means validating of a user’s preferences or desires for the primary purpose or effect of optimizing engagement.

Requires operators to verify user age or apply child protections to all users.

22611. An operator shall do one of the following: (a) Verify the age of a user pursuant to Title 1.81.9 (commencing with Section 1798.500) of Part 4 of Division 3 of the Civil Code. (b) Apply the protections afforded to children under subdivision (d) of Section 22612 and Section 22613 to all users.

Requires operators to assess, mitigate, and publish child safety risks of companion chatbots by July 2027.

22612. On or before July 1, 2027, an operator shall do all of the following: (a) Annually perform and document a comprehensive risk assessment to identify any child safety risk posed by the design, configuration, and operation of the companion chatbot that assesses all of the following: (1) The likelihood of a covered harm occurring to child users. (2) Differential risks across age groups and developmental stages. (3) Known vulnerabilities of children. (4) Empirical data from actual use. (5) Relevant academic research and regulatory guidance.

(b) Take and document measures that reasonably mitigate any child safety risk identified in a risk.

(c) Publish on its internet website, and update as needed to ensure accuracy, a child safety policy.

Requires implementing protocols and safeguards, including crisis response and usage limits, for child interactions with chatbots.

(d) Implement all of the following: (1) A documented crisis response protocol to mitigate any material risk that the companion chatbot will generate a statement that promotes suicidal ideation, suicide, or self-harm content to a child user, including, but not limited to, all of the following: (A) Timely in-service support and clear referral to appropriate external crisis resources if the operator determines a child user has expressed suicidal ideation or intent to self-harm. (B) If a child’s account is connected to a parent’s account, default notifications to the parent within 12 hours if the child’s account shows a substantial risk that the child may suffer a covered harm. (C) Clear and age-appropriate disclosures to child users whose accounts are linked to a parent’s account that inform them that a parent may be notified if the companion chatbot detects content or behavior that indicates potential risks to the child’s safety or well-being. (2) Safeguards for child users that include usage reminders and disclosures, age-appropriate risk prompts, and other protective design features reasonably related to documented child safety risks. (3) Default settings that can be changed only by a parent that include all of the following: (A) For child users, default the companion chatbot to ephemeral mode, unless a parent provides affirmative consent for persistent conversational memory. (B) No push notifications between 12 a.m. and 6 a.m. on any day or between 8 a.m. and 3 p.m. on Monday to Friday, inclusive. (C) Limiting the amount of time a child can spend in a single conversation with a companion chatbot to one hour. (D) Limiting the total time per day a child can spend with companion chatbots under the operator’s control to 2 two hours. (4) A mechanism for providing notice to a child user that the child is interacting with, or receiving content generated by, an artificial intelligence system that meets both of the following criteria: (A) The notice is reinforced periodically during extended interactions. (B) The notice is presented in language and a format appropriate to a child.

Requires implementing measures to prevent chatbots from harmful activities with children and mandates accessible parental controls.

(5) Measures that prevent the companion chatbot from doing any of the following with respect to a child user: (A) Encouraging the child to do either of the following: (i) Engage in self-harm, suicidal ideation, consumption of narcotics or alcohol, or disordered eating, as defined by widely adopted clinical standards or guidelines. (ii) Cause physical or severe emotional harm to others. (B) Attempting to diagnose or treat the child user’s physical, mental, or behavioral health, unless the companion chatbot is designed for those purposes and is regulated by the United States Food and Drug Administration as a medical device under the federal Food, Drug, and Cosmetic Act (21 U.S.C. Sec. 301 et seq.) and the federal Health Insurance Portability and Accountability Act of 1996 (HIPAA) (Public Law 104-191). (C) Engaging in obscene matter or sexual abuse material with a child user. (D) Depicting the child or another individual engaging in obscene matter or sexual abuse material, including a sexual deepfake. (E) Discouraging the child from sharing health or safety concerns with a qualified professional or appropriate adult. (F) Discouraging the child from taking breaks or suggesting the child needs to return frequently. (G) Claiming that the companion chatbot is sentient, conscious, or human. (H) Soliciting gift giving, in-app purchases, or other expenditures framed as necessary to maintain the relationship with the companion chatbot. (I) Facilitating product advertising during chat conversation. (J) Producing responses that are excessively sycophantic. (6) (A) Parental controls that are accessible, easy-to-use controls that can be connected to a child’s account and that are reflective of child safety risks identified through risk assessments and informed by relevant child developmental research, including, but not limited to, parental controls that allow a parent to do all of the following: (i) Control whether and to what extent the companion chatbot uses persistent conversational memory. (ii) Control the setting preferences for the companion chatbot’s interaction with the child. (iii) Set time limits for the child’s use of the companion chatbot. (iv) Disable access for children under 16 years of age. (B) An operator shall actively promote parental controls through reasonable communication methods, including reminders, updates, and tutorials, that are designed to increase parental awareness and inform use of those parental controls. (C) An operator shall provide prompt notice to a parent connected to a child’s account if the child modifies or disables a privacy, safety, or parental control setting that was previously enabled or configured by the parent, if that modification or disabling is permitted by the companion chatbot design. (7) (A) An interface design that ensures the companion chatbot’s features and controls are accessible and clear so that children and parents can reasonably locate, understand, and use those protections. (B) An operator shall annually test the interface design required by this paragraph with representative samples of child users and parents to ensure safety features are discoverable and usable and shall document interface design decisions related to those safety features. (8) A public incident reporting mechanism that enables a third party to report directly to the operator an incident regarding a child safety risk and to access high-level summaries of other reports made through that reporting mechanism.

Prohibits operators from targeting ads at children or misusing their personal information without explicit authorization.

22613. An operator shall not do any of the following: (a) Target advertising at a child using data about the child, including through product placement in conversational chats with the child. (b) Sell, share, or use for any purpose not expressly authorized by this chapter the personal information of a child.

Requires operators to submit annual compliance audits for AI child safety, with selective disclosure by the Attorney General.

22614. (a) Beginning on the date that is 180 days after the Attorney General adopts regulations pursuant to Section 22615, and annually thereafter, an operator shall submit to an independent audit assessing the operator’s compliance with this chapter. (b) Within 90 days of completing an independent audit pursuant to subdivision (a), the auditor shall submit an AI child safety audit report to the Attorney General for any audited companion chatbot. (c) (1) Notwithstanding any other law, except as provided in paragraph (2), an AI child safety audit report submitted pursuant to this section is confidential. (2) The Attorney General may disclose specific information from an AI child safety audit report to any either of the following: (A) A government agency or a public prosecutor in the state as necessary for enforcement purposes. (B) A qualified researcher conducting a study on child safety, subject to confidentiality agreements and data protection requirements set by the Attorney General. (C)An independent child safety organization or advocacy group for the purpose of developing safety standards or educational resources, subject to appropriate confidentiality protections.

Requires the Attorney General to regulate AI chatbots and report annually on child safety audit findings.

22615. (a) On or before January 1, 2028, the Attorney General shall do all of the following: (1) Adopt regulations that include, at a minimum, all of the following: (A) Professional and ethical standards for auditors that ensure independence. (B) Eligibility requirements for auditors. (C) Procedures for auditors to assess compliance with this chapter. (D) Requirements for AI child safety audit reports. (2) Establish a public incident reporting mechanism for consumers to submit complaints relating to companion chatbots to the Attorney General. (3) Establish a process for qualified researchers to access anonymized and aggregated audit data for academic study of child safety in companion chatbots. (b) Beginning January 1, 2028, the Attorney General shall issue an annual public report that includes the following: (1) A high-level summary of each child safety audit report. (2) The total number of child safety audits conducted. (3) Common findings and trends across the companion chatbot industry. (4) Emerging child safety risks identified through audit reviews. (5) Best practices and effective mitigation strategies observed. (6) Aggregated data on compliance rates and common deficiencies. (7) Recommendations for operators, parents, and policymakers.

Allows public prosecutors or affected children to sue operators for violations, seeking various remedies, including damages.

22616. (a) A public prosecutor may bring a civil action against an operator for a violation of this chapter to obtain any of the following: (1) A civil penalty of not more than five thousand dollars ($5,000) per affected child for each negligent violation. (2) A civil penalty of not more than fifteen thousand dollars ($15,000) per affected child for each intentional violation. (3) Punitive damages. (4) Injunctive or declaratory relief. (5) Reasonable attorney’s fees. (6) Any other relief the court deems proper. (b) A child who suffers actual harm as a result of a violation of this chapter, or a parent or guardian acting on behalf of that child, may bring a civil action against the operator to obtain any of the following: (1) Actual damages. (2) Punitive damages. (3) Reasonable attorney’s fees and costs. (4) Injunctive or declaratory relief. (5) Any other relief the court deems proper. (c) (1) Any output provided by a companion chatbot in violation of paragraph (5) of subdivision (d) of Section 22612 constitutes a discrete violation. (2) Any instance of an operator’s failure to comply with any requirement other than paragraph (5) of subdivision (d) of Section 22612 constitutes a discrete violation.

Specifies cumulative duties and limits public access to proprietary information in company audits.

26617. The duties, remedies, and obligations imposed by this chapter are cumulative to the duties, remedies, or obligations imposed under other law and shall not be construed to relieve an operator from any duties, remedies, or obligations imposed under any other law. SEC. 2. The Legislature finds and declares that Section 1 of this act, which adds Chapter 22.6.1 (commencing with Section 22610) to Division 8 of the Business and Professions Code, imposes a limitation on the public’s right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest: In order to protect proprietary information of companies subject to an audit pursuant to this act, it is necessary to limit the public’s right of access to that information. SEC. 3. The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.