Library

Emerging Security Practices for AI Agents

Detection and triage Investigation, mitigation, and response External reporting 06 09 03 03 05 07 07 08 10 12 12 13 13 11 3 Security risk management 14 4 Model reporting 16 5 Input from external experts 17 6 Allocation of responsibility for risk management 18 7 Framework change management

Issue Brief: Information Sharing, Incident Reporting, and Incident Response for Frontier AI Risks

GPT-5.5 Instant System Card OpenAI May 4, 2026

Position: Comprehensive AI governance requires addressing non-model capability gains Arthur Goemans 1 Dan Altman 1 Noemi Dreksler 2 Jonas Freund 2 Milan Gandhi 1 Zhengdong Wang 1 Sarah Cogan 1 Sebastien Krier 1 Demetra Brady 1 Lewis Ho 1 Allan Dafoe 1

Responsible Scaling Policy Version 3.2 Effective April 29, 2026 For more information, see www.anthropic.com/responsible-scaling-policy

GPT-5.5 System Card OpenAI April 23, 2026

Advanced AI Scaling Framework Version 2 Meta’s vision is to bring personal superintelligence to everyone. We believe in putting this power in people's hands to direct it towards what they value in their own lives. Realizing this will require highly capable AI systems that are reliable, robust, and secure. This Advanced AI Scaling Framework outlines how Meta manages and prepares for Frontier AI…

The AI Index Report is an independent, comprehensive annual assessment of the state of artificial intelligence, produced by the Stanford Institute for Human-Centered AI (HAI). The 2026 edition tracks and benchmarks AI progress across research and development, technical performance, responsible AI, the economy, science, medicine, education, policy and governance, and public opinion — drawing tog…

New York General Business Law, Article 44-B — the Responsible AI Safety and Education (RAISE) Act — as enacted by the S.8828 chapter amendment, signed March 27, 2026. The chapter amendment repealed the original Article 44-B and replaced it with §§ 1420–1429. The regime takes effect January 1, 2027.

New York General Business Law, Article 44-B — the Responsible AI Safety and Education (RAISE) Act — as enacted by the S.8828 chapter amendment, signed March 27, 2026. The chapter amendment repealed the original Article 44-B and replaced it with §§ 1420–1429. The regime takes effect January 1, 2027.

March 9, 2026 U.S. National Institute of Standards and Technology Center for Artificial Intelligence Standards and Innovation 100 Bureau Drive Gaithersburg, MD 20899 To Whom It May Concern: The Frontier Model Forum (FMF) is grateful to the U.S. Center for AI Standards and Innovation (CAISI) for developing voluntary guidelines on AI agent security. We share the CAISI’s view that best practice gu…

GPT-5.4 Thinking System Card OpenAI March 5, 2026

TECHNICAL REPORT | MARCH 2026 AI and Bomb Plots: Distinguishing Potential Effects from Language Models Connor A. Stewart Hunter and Luca Righetti

TECHNICAL REPORT | MARCH 2026 Global Cybercrime Damages A Baseline for Frontier AI Risk Assessment Kamile Lukosiute, John Halstead, Luca Righetti

Measuring AI R&D Automation {Alan Chan1∗ Ranay Padarath1† Joe Kwon1†}‡ Hilary Greaves2 Markus Anderljung1§ 1GovAI 2University of Oxford

POLICY BRIEF | MARCH 2026 Requirements for Model Specifications in the EU GPAI Code of Practice Alan Chan

Adversarial distillation is a growing concern for frontier AI safety and security. Although frontier AI developers may use or authorize distillation in select cases – for instance, to create smaller, more efficient, versions of their flagship models or to adapt models for specialized domains, such as life sciences – adversarial distillation may raise significant safety and security risks by all…

Frontier AI offers significant promise for cybersecurity, including accelerating vulnerability discovery and patching, optimizing defensive systems, and enhancing threat detection capabilities. However, these same capabilities create dual-use risks, potentially lowering barriers for malicious actors to exploit known vulnerabilities or discover new attack vectors. As AI capabilities advance, it…

MEASURING MID-2025 LLM-ASSISTANCE ON NOVICE PERFORMANCE IN BIOLOGY Shen Zhou Hong1, *, Alex Kleinman1, *, Alyssa Mathiowetz1, *, Adam Howes2, Julian Cohen1, Suveer Ganta1, Alex Letizia1, Dora Liao1, Deepika Pahari1, Xavier Roberts-Gaal1, Luca Righetti3, 4, and Joe Torres1, 4, † 1Active Site, Cambridge, MA 02142, United States 2Independent 3Model Evaluation and Threat Research, Inc., Covina, CA…

Research Update: Frontier AI and Nuclear Security

Issue Brief: Chain of Thought Monitorability As artificial intelligence (AI) systems become more capable, it is important that they operate safely and as intended. A common issue is the "black box" problem: we can see an AI's final output, but we often cannot understand the internal reasoning process that produced it. This opacity makes it difficult to detect, predict, and prevent potentially h…

Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies Miles Brundage1* Noemi Dreksler2 Aidan Homewood2 Sean McGregor1 Patricia Paskov3 Conrad Stosz4 Girish Sastry5 A. Feder Cooper1 George Balston1 Steven Adler6 Stephen Casper7 Markus Anderljung2 Grace Werner1 Sören Mindermann5 Vasilios Mavroudis8 Ben Bucknall9 Charlotte Stix10 Jona…

NBER WORKING PAPER SERIES HOW ADAPTABLE ARE AMERICAN WORKERS TO AI-INDUCED JOB DISPLACEMENT? Sam J. Manning Tomás Aguirre Working Paper 34705 http://www.nber.org/papers/w34705 NATIONAL BUREAU OF ECONOMIC RESEARCH 1050 Massachusetts Avenue Cambridge, MA 02138 January 2026 We thank Dimitris Papanikolaou, Martin Beraja, Mark Muro, Shriya Methkupally, Daniel Rock, Morgan Frank, David Autor, Maria d…

xAI Frontier Artificial Intelligence Framework Last updated: December 30, 2025 xAI seriously considers safety and security while developing and advancing AI models to help us all to better understand the universe. This Frontier AI Framework (“FAIF”) outlines xAI’s approach to policies for handling significant risks, including catastrophic risks, associated with the development, deployment, and…

Today we are announcing a new cohort of 11 grantees who have received more than $5 million through the AI Safety Fund (AISF). As frontier AI systems become more powerful and widely deployed, advancing our understanding of them and building robust safety tools is essential – which is why the AISF issued several requests for proposals late last year in Biosecurity and Cybersecurity, as well as AI…

Common Elements of Frontier AI Safety Policies December 2025

TECHNICAL REPORT | December 2025 Does the UAE Have an Advantage in Building Data Centers? Comparing the Total Cost of Building and Operating a 100-Megawatt Data Center in the US and the UAE Amelia Michael

Dual-Use AI Capabilities and the Risk of Bioterrorism Converting Capability Evaluations to Risk Assessments Luca Righetti Authors and affiliations Luca Righetti GovAI Contact Corresponding author: Luca Righetti (luca.righetti@governance.ai). Please cite as Righetti, L. (2025). Dual-Use AI Capabilities and the Risk of Bioterrorism: Converting Capability Evaluations to Risk Assessments. GovAI. Go…

Model card update: December, 2025 Gemini 3 Pro Model Card Gemini 3 Pro - Model Card Model Cards are intended to provide essential information on Gemini models, including known limitations, mitigation approaches, and safety performance. Model cards may be updated from time-to-time; for example, to include updated evaluations as the model is improved or revised. See the Google DeepMind site for a…

Permission Manifests for Web Agents Lightweight Agent Standards Working Group (LAS-WG) Samuele Marro1,2∗ , Alan Chan3, Xinxing Ren4, Lewis Hammond1,5, Jesse Wright1,2, Gurjyot Wanga6, Tiziano Piccardi7, Nuno Campos8, Tobin South9,2, Jialin Yu1,2, Sunando Sengupta10, Eric Sommerlade10, Alex Pentland9, Philip Torr1,2, Jiaxin Pei11,9,2 1University of Oxford 2Institute for Decentralized AI 3Centre…

Sam Manning December 2025 AI’S LABOR MARKET IMPACTS Opportunities for the Department of Labor’s AI Workforce Research Hub UNDERSTANDING Table of ContentsIntroduction ...................................................................... 1 Background ....................................................................... 3 Part 1: Establish Data-Sharing Partnerships with Leading AI Developers to…

POLICY BRIEF | NOVEMBER 2025 Labeling of AI Agent Activity in Article 50 of the EU AI Act Alan Chan

POLICY BRIEF | OCTOBER 2025 Assessing Risk Relative to Competitors: An Analysis of Current AI Company Policies Sophie Williams, Noemi Dreksler, Aidan Homewood, Markus Anderljung, Jonas Freund

TECHNICAL REPORT | OCTOBER 2025 Inference Scaling and AI Governance Toby Ord Inference Scaling and AI Governance Toby Ord1 1Oxford Martin AI Governance Initiative, University of Oxford The shift from scaling up the compute used to pre-train AI systems (pre-training compute) to scaling up the amount used to run them (inference compute) may have profound effects on AI governance. The nature of th…

September 12 2025 U.S. National Institute of Standards and Technology 100 Bureau Drive Gaithersburg, MD 20899 To Whom It May Concern: The Frontier Model Forum (FMF) is grateful to the NIST Zero Draft project for its effort to develop a standard on AI testing, evaluation, verification, and validation (TEVV). We share NIST’s view that rigorous TEVV practices are essential for building trust in AI…

STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reports Tegan McCaslin Independent Jide Alaga METR Samira Nedungadi SecureBio Seth Donoughe SecureBio Tom Reed GovAI Rishi Bommasani Stanford HAI Chris Painter METR Luca Righetti∗ GovAI

Third-party assessments can be conducted on frontier models to confirm evaluations or claims on critical safety capabilities and mitigations. In appropriate contexts, these assessments may help to confirm or build confidence in safety claims, add robust methodological independence, and supplement expertise. This report outlines practices and approaches among Frontier Model Forum (FMF) firms for…

Incident Analysis for AI Agents Carson Ezell1, Xavier Roberts-Gaal1, Alan Chan2 1Harvard University 2Centre for the Governance of AI cezell@college.harvard.edu, xavierrobertsgaal@g.harvard.edu, alan.chan@governance.ai

Issue Brief: Preliminary Taxonomy of Frontier AI-Bio Misuse Mitigations Frontier AI presents transformative opportunities within the biological sciences, including the potential to rapidly accelerate beneficial research discoveries and development. However, the dual-use nature of these technologies may also introduce novel risks. One potential harm involves the misuse of legitimately accessed f…

Forecasting LLM-enabled biorisk and the efficacy of safeguards uthors˓ ridget fillias˒ Luca Mighetti˒ 1osh Mosenberg˒ Mebecca e¾¾as de astro˒ @tto 2uusela˒ Mhiannon ritt˒ ily Qoice˒ lÙaro Morales˒ 1on Qanders˒ Qeth onoughe˒ 1aes lack˒ åra 2arger˒ Jhili¾  ˑ Wetlock First released on: July 1, 2025 Forecasting LLM-enabled biorisk and the efficacy of safeguards Bridget Williams1*, Luca R…

arXiv:2507.03050v1 [cs.CY] 3 Jul 2025 From Turing to Tomorrow: The UK’s Approach to AI Regulation Oliver Ritchie∗ † Markus Anderljung∗ Tom Rachman Centre for the Governance of AI This is a preprint of the following chapter: Oliver Ritchie, Markus Anderljung, and Tom Rachman “From Turing to Tomorrow: The UK’s Approach to AI Regulation”. It is intended for publication in a forthcoming book. It is…

Frontier mitigations are protective measures implemented on frontier models, with the goal of reducing the risk of potential high-severity harms, especially those related to national security and public safety, that could arise from their advanced capabilities. This report discusses emerging industry practices for implementing and assessing frontier mitigations. It focuses on mitigations for ma…

Frontier AI frameworks outline methodologies for identifying, managing and mitigating the potential for large-scale risks to public safety and national security that stem from frontier AI development and deployment. This report examines the rationale for including only select risk domains within frontier AI frameworks, the current practices used to identify which risks warrant inclusion and how…

Issue Brief: Frontier AI Biosafety Thresholds The rapid advancement of frontier AI presents transformative opportunities in the biological domain. For example, frontier AI systems may accelerate the discovery of new medical treatments, optimize biomanufacturing processes, or facilitate the development of novel biocatalysts. However, given that frontier AI capabilities are often dual-use, they m…

51 AEA Papers and Proceedings 2025, 115: 51–55 https://doi.org/10.1257/pandp.20251045 Extending “GPTs Are GPTs” to Firms† By Benjamin Labaschin, Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock* Given the major role in the adoption and deployment path of artificial intelligence (AI) played by firms, corporate data on AI-related investments, opportunities, and risks are important inpu…

ISO/IEC 42001 Certification: The Global Standard for AI Management Systems (AIMS) How to establish trust, transparency and control in artificial intelligence governance May 2025 KPMG. Make the Difference. Document Classification: KPMG Public 2© 2025 KPMG AG, a Swiss corporation, is a group company of KPMG Holding LLP, which is a member of the KPMG global organization of independent firms affili…

arXiv:2505.01643v2 [cs.CY] 4 Jul 2025 Third-party compliance reviews for frontier AI safety frameworks Aidan Homewood∗ 1 Sophie Williams1 Noemi Dreksler1 John Lidiard1 Malcolm Murray2 Lennart Heim1 Marta Ziosi† 3 Seán Ó hÉigeartaigh4 Michael Chen5 Kevin Wei6 Christoph Winter4,7 Miles Brundage8 Ben Garfinkel1 Jonas Schuett1 1Centre for the Governance of AI 2SaferAI 3Oxford Martin AI Governance I…

Frontier Capability Assessments are procedures conducted on frontier models with the goal of determining whether they have capabilities that could increase risks to public safety and security, such as by facilitating the development of chemical, biological, radiological, or nuclear (CBRN) weapons, advanced cyber threats, or some categories of advanced autonomous behavior. This report discusses…

As AI systems advance in capability, they have the potential to accelerate scientific discovery and drive economic growth. Yet alongside those benefits they also pose a distinct challenge: Highly capable frontier AI systems may introduce or elevate large-scale risks to public safety and national security, including those related to advanced cyber and chemical, biological, radiological, and nucl…

Trends in Frontier AI Model Count: A Forecast to 2028 Iyngkarran Kumar∗1 and Sam Manning2 1University of Edinburgh 2Centre for the Governance of AI April 24, 2025

Preliminary Reporting Tiers for AI Bio Safety Evaluations Frontier AI models and systems are particularly promising for advancing medicine and public health. At the same time, their knowledge of biology and ability to reason about biological concepts may also be misused in ways that pose significant risks to public safety and security. To manage those risks, many frontier AI developers have pub…

Frontier Model Forum Response to the Request for Information on the Development of an Artificial Intelligence Action Plan The Frontier Model Forum submitted the below response on March 14, 2025 to the Request for Information on the Development of an Artificial Intelligence (AI) Action Plan. We are grateful for the opportunity to respond to the request for information from the Office of Science…

RAISE Act — A.6453-B (original 2025 introduction) — full text and source on file.

In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI Shayne Longpre⋆ 1 Kevin Klyman⋆◦ 2 Ruth E. Appel⋆ 2 Sayash Kapoor♢ 3 Rishi Bommasani♢ 2 Michelle Sahar♢ 4 Sean McGregor♢ 5 Avijit Ghosh♢ 6 Borhane Blili-Hamelin♢ 7 Nathan Butters♢ 7 Alondra Nelson 8 Amit Elazari 4 Andrew Sellars 9 Casey John Ellis 10 Dane Sherrets 11 Dawn Song 12 Harley Geiger 1…

On Regulating Downstream AI Developers Sophie Williams1) Jonas Schuett1) Markus Anderljung1)

Thresholds for Frontier AI Safety Frameworks Although frontier AI holds enormous promise for society, advanced AI systems may also pose significant risks to national security and public safety. Frontier AI safety frameworks have recently emerged as a method for frontier AI developers to demonstrate how they manage those risks effectively. By establishing processes for how to identify, evaluate,…

CBRN weapons 11 Offensive cyberoperations 12 Advanced autonomy 13 Appendix II – Change log 14

MEASURING AI AGENT AUTONOMY: TOWARDS A SCALABLE APPROACH WITH CODE INSPECTION Peter Cihon ∗ GitHub San Francisco, CA, USA petercihon@gmail.com Merlin Stein ∗ University of Oxford Oxford, UK merlin.stein@bsg.ox.ac.uk Gagan Bansal Microsoft Redmond, WA, USA Sam Manning Centre for the Governance of AI Oxford, UK Kevin Xu GitHub San Francisco, CA, USA

• 1 Options and Motivations for International AI Benefit Sharing Claire Dennis, Sam Manning, Stephen Clare, Boxi Wu, Jake Okechukwu Effoduh, Chinasa T. Okolo, Lennart Heim, Katya Klinova CLAIRE DENNIS* Oxford Martin AI Governance Initiative, University of Oxford Centre for the Governance of AI SAM MANNING* Centre for the Governance of AI STEPHEN CLARE* Centre for the Governance of AI BOXI WU Un…

Infrastructure for AI Agents Alan Chan∗ Centre for the Governance of AI Kevin Wei Harvard Law School Sihao Huang University of Oxford Nitarshan Rajkumar University of Cambridge Elija Perrier Australian National University Seth Lazar† Australian National University Gillian K. Hadfield† Johns Hopkins University Markus Anderljung† Centre for the Governance of AI Reviewed on OpenReview: https: // o…

Local US officials’ views on the impacts and governance of AI: Evidence from 2022 and 2023 survey waves Sophia Hatz1Y*, Noemi Dreksler2Y, Kevin Wei2,3Y, Baobao Zhang2,4 1 Department of Peace and Conflict Research, Uppsala University, Uppsala, Sweden 2 Centre for the Governance of AI, Oxford, United Kingdom 3 Harvard Law School, Harvard University, Cambridge, Massachusetts, USA 4 Political Scien…

• 1 What Does the Public Think About AI? An overview of the public’s attitudes towards AI and a resource for future research Noemi Dreksler, Harry Law, Chloe Ahn, Daniel S. Schiff, Kaylyn Jackson Schiff, Zachary Peskowitz Authors and affiliations NOEMI DREKSLER Centre for the Governance of AI HARRY LAW University of Cambridge CHLOE AHN University of Pennsylvania DANIEL S. SCHIFF Purdue Universi…

Frontier AI-bio safety evaluations aim to test the biological capabilities and potential biosafety implications of frontier AI. As this field remains nascent, evaluations vary significantly in purpose and methodology. A shared understanding of safety evaluation functions and types is essential for building an effective evaluation ecosystem in the AI-bio space.

Issue Brief: Preliminary Taxonomy of Pre-Deployment Frontier AI Safety Evaluations

December 3, 2024 U.S. National Institute of Standards and Technology 100 Bureau Drive Gaithersburg, MD 20899 Dear Director Kelly and U.S. AI Safety Institute staff: The Frontier Model Forum (FMF) is an industry-supported non-profit dedicated to advancing the safe development and deployment of frontier AI systems. The most advanced AI systems have the potential to dramatically improve public hea…

By: Frontier Model Forum Posted on: 22nd November 2024

Issue Brief: Components of Frontier AI Safety Frameworks

Safety case template for frontier AI: A cyber inability argument Arthur Goemans∗ 1 Marie Davidsen Buhl1 Jonas Schuett1 Tomek Korbak2 Jessica Wang2 Benjamin Hilton2 Geoffrey Irving2 1Centre for the Governance of AI 2UK AI Safety Institute

                                                            …

IDs for AI Systems Alan Chan alan.chan@governance.ai Centre for the Governance of AI Mila (Quebec AI Institute) Noam Kolt University of Toronto Peter Wills Centre for the Governance of AI University of Oxford Usman Anwar University of Cambridge Christian Schroeder de Witt University of Oxford Nitarshan Rajkumar University of Cambridge Lewis Hammond University of Oxford Cooperative AI Foundation…

Safety cases for frontier AI Marie Davidsen Buhl∗ Gaurav Sett Leonie Koessler Jonas Schuett Markus Anderljung Centre for the Governance of AI

VOICE AND ACCESS IN AI: GLOBAL AI MAJORITY PARTICIPATION IN ARTIFICIAL INTELLIGENCE DEVELOPMENT AND GOVERNANCE WHITEPAPER October 2024 Authors: Sumaya N. Adan, Robert Trager, Kayla Blomquist, Claire Dennis, Gemma Edom, Lucia Velasco, Cecil Abungu, Ben Garfinkel, Julian Jacobs, Chinasa T. Okolo, Boxi Wu, Jai Vipra

September 9, 2024 U.S. National Institute of Standards and Technology MS 20899 100 Bureau Drive Gaithersburg, MD 20899 Dear Director Kelly and U.S. AI Safety Institute staff: The Frontier Model Forum is an industry-supported non-profit dedicated to advancing the safe development and deployment of frontier AI systems. Our work focuses on addressing the potential risks of frontier AI by identifyi…

A Grading Rubric for AI Safety Frameworks Jide Alaga∗ Jonas Schuett Markus Anderljung Centre for the Governance of AI

1 Economic Policy Challenges for the Age of AI1 Anton Korinek (University of Virginia, CSH Vienna, and GovAI) September 2024 This paper examines the profound challenges that transformative advances in AI towards Artificial General Intelligence (AGI) will pose for economists and economic policymakers. I examine how the Age of AI will revolutionize the basic structure of our economies by diminish…

The core mission of the Frontier Model Forum is to advance frontier AI safety. By identifying best practices, supporting scientific research, and facilitating greater information-sharing about frontier AI safety, we aim to meaningfully improve the safe development and deployment of the most advanced general-purpose AI systems.

Training Compute Thresholds: Features and Functions in AI Regulation Lennart Heim∗ Centre for the Governance of AI Oxford, United Kingdom Leonie Koessler Centre for the Governance of AI Oxford, United Kingdom & European New School of Digital Studies Frankfurt (Oder), Germany

Issue Brief: Early Best Practices for Frontier AI Safety Evaluations

Issue Brief: Foundational Security Practices

NIST Trustworthy and Responsible AI NIST AI 600-1 Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile This publication is available free of charge from: https://doi.org/10.6028/NIST.AI.600-1 NIST Trustworthy and Responsible AI NIST AI 600-1 Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile This publication is…

Open Problems in Technical AI Governance Anka Reuel∗ Stanford University anka.reuel@stanford.edu Ben Bucknall∗ Centre for the Governance of AI & ben.bucknall@governance.ai Oxford Martin AI Governance Initiative Stephen Casper MIT CSAIL Tim Fist Institute for Progress & Center for a New American Security Lisa Soder interface – Tech Analysis and Policy Ideas for Europe e.V. Onni Aarne Institute f…

Intelligent financial system: how AI is transforming finance I Aldasoro BIS L Gambacorta BIS & CEPR A Korinek Univ. of Virginia & GovAI V Shreeti BIS M Stein Univ. of Oxford June 20, 2024

Risk thresholds for frontier AI Leonie Koessler∗ Jonas Schuett Markus Anderljung Centre for the Governance of AI

Recent AI safety and governance proposals have leveraged total training compute as a proxy for the capability level of general purpose models. By tying specific safety measures to particular compute thresholds, such proposals aim to ensure both that AI safety mechanisms are proportionate to model capabilities and also that those mechanisms can be applied in a consistent and objective manner. Wh…

Societal Adaptation to Advanced AI Jamie Bernardi*a, Gabriel Mukobi*b, Hilary Greaves*c, Lennart Heima,d, Markus Anderljung*†a a Centre for the Governance of AI b Stanford University c University of Oxford d RAND

WHITEPAPER March 2024 In partnership with Lennart Heim, Tim Fist, Janet Egan, Sihao Huang, Stephen Zekany, Robert Trager, Michael A. Osborne & Noa Zilberman GOVERNING THROUGH THE CLOUD: THE INTERMEDIARY ROLE OF COMPUTE PROVIDERS IN AI REGULATION Governing Through the Cloud: The Intermediary Role of Compute Providers in AI Regulation Lennart Heim,∗ † 1 Tim Fist,∗ 2 Janet Egan,∗ 3 Sihao Huang,∗ 4…

Scenarios for the Transition to AGI* Anton Korinek and Donghyun Suh University of Virginia, Brookings, and GovAI March 2024

Computing Power and the Governance of Artificial Intelligence Girish Sastry,∗†1 Lennart Heim,∗†2 Haydn Belfield,∗†3 Markus Anderljung,∗2 Miles Brundage,∗1 Julian Hazell,∗2,4 Cullen O’Keefe,∗1,5 Gillian K. Hadfield,∗6,7 Richard Ngo,1 Konstantin Pilz,8 George Gor,9 Emma Bluemke,2 Sarah Shoker,1 Janet Egan,10 Robert F. Trager,11 Shahar Avin,12 Adrian Weller,13 Yoshua Bengio,14 Diane Coyle15 1OpenA…

Konstantin Pilz * 1 Lennart Heim * 2 Nicholas Brown 3

Response to the RFI related to NIST’s assignments under the Executive Order Concerning AI Jonas Schuett Research Fellow Centre for the Governance of AI jonas.schuett@governance.ai Leonie Koessler Research Scholar Centre for the Governance of AI leonie.koessler@governance.ai Markus Anderljung Head of Policy Centre for the Governance of AI markus.anderljung@governance.ai February 2024 We welcome…

2023 has been an extraordinary year for AI. As chatbots surged in popularity and generative AI went mainstream, the promise and potential of AI gained worldwide attention. Yet at the same time, the powerful new capabilities of frontier AI systems also raised concerns about risks. It is important that we act now to ensure models are built safely and responsibly.

Preparing the Workforce for an Uncertain AI Future Dr. Anton Korinek Professor of Economics, University of Virginia and Darden School of Business Economics of AI Lead, Centre for the Governance of AI Written Statement to the U.S. Senate AI Insight Forum on Workforce Washington, D.C. November 1, 2023 Majority Leader Schumer, Senators Rounds, Heinrich, and Young, and Distinguished Members of the…

Towards Publicly Accountable Frontier LLMs Building an External Scrutiny Ecosystem under the ASPIRE Framework Markus Anderljung1,2,*, Everett Thornton Smith1, Joe O’Brien1,3, Lisa Soder1, Benjamin Bucknall1, Emma Bluemke1, Jonas Schuett1, Robert Trager1,4, Lacey Strahm5, and Rumman Chowdhury6,7 1Centre for the Governance of AI 2Center for a New American Security 3Institute for AI Policy and Str…

Frontier Model Forum: What is Red Teaming?

Oversight for Frontier AI through a Know-Your-Customer Scheme for Compute Providers Janet Egan1 and Lennart Heim2 ∗ 1Harvard Kennedy School, 2Centre for the Governance of AI

WHITEPAPER October 2023 In partnership with Benjamin S. Bucknall, Robert F. Trager STRUCTURED ACCESS FOR THIRD-PARTY RESEARCH ON FRONTIER AI MODELS: INVESTIGATING RESEARCHERS’ MODEL ACCESS REQUIREMENTS In partnership with STRUCTURED ACCESS FOR THIRD-PARTY RESEARCH ON FRONTIER AI MODELS: INVESTIGATING RESEARCHERS’ MODEL ACCESS REQUIREMENTS Benjamin S. Bucknall∗ ben.s.bucknall@gmail.com Robert F.…

Vol.:(0123456789)1 3AI & SOCIETY https://doi.org/10.1007/s00146-023-01811-0 MAIN PAPER Three lines of defense against risks from AI Jonas Schuett1,2,3 Received: 15 April 2023 / Accepted: 31 October 2023 © The Author(s) 2023

Coordinated pausing: An evaluation-based coordination scheme for frontier AI developers Jide Alaga∗ Centre for the Governance of AI Jonas Schuett Centre for the Governance of AI

Center on Regulation and Markets Working Paper #9 Market concentration implications of foundation models: THE INVISIBLE HAND OF CHATGPT Jai Vipra and Anton Korinek This working paper is available online at: https://www.brookings.edu/series/center-on-regulation-and-markets-working-papers/ The Center on Regulation and Markets at Brookings creates and promotes rigorous economic scholarship to info…

• 3 An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives Open-Sourcing Highly Capable Foundation Models Elizabeth Seger, Noemi Dreksler, Richard Moulange, Emily Dardaman, Jonas Schuett, K. Wei, Christoph Winter, Mackenzie Arnold, Seán Ó hÉigeartaigh, Anton Korinek, Markus Anderljung, Ben Bucknall, Alan Chan, Eoghan Stafford, Leonie Koessler, Aviv Ovadya,…

INTERNATIONAL GOVERNANCE OF CIVILIAN AI A Jurisdictional Certification Approach Robert F. Trager, Ben Harack, Anka Reuel, Allison Carnegie, Lennart Heim, Lewis Ho, Sarah Kreps, Ranjit Lall, Owen Larter, Seán Ó hÉigeartaigh, Simon Staffell, José Jaime Villalobos WHITEPAPER August 2023 In parthership with International Governance of Civilian AI: A Jurisdictional Certification Approach Robert F. T…

We're forming a new industry body to promote the safe and responsible development of frontier AI systems: advancing AI safety research, identifying best practices and standards, and facilitating information sharing among policymakers and industry.

FRONTIER AI REGULATION: MANAGING EMERGING RISKS TO PUBLIC SAFETY Markus Anderljung1,2∗†, Joslyn Barnhart3∗∗, Anton Korinek4,5,1∗∗†, Jade Leung6∗, Cullen O’Keefe6∗, Jess Whittlestone7∗∗, Shahar Avin8, Miles Brundage6, Justin Bullock9,10, Duncan Cass-Beggs11, Ben Chang12, Tantum Collins13,14, Tim Fist2, Gillian Hadfield15,16,17,6, Alan Hayes18, Lewis Ho3, Sara Hooker19, Eric Horvitz20, Noam Kolt1…

National Priorities for Artificial Intelligence Response to the OSTP Request for Information by the Centre for the Governance of AI Jonas Schuett Research Fellow Centre for the Governance of AI jonas.schuett@governance.ai Markus Anderljung Head of Policy Centre for the Governance of AI markus.anderljung@governance.ai Lennart Heim Research Fellow Centre for the Governance of AI lennart.heim@gove…

Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industries Leonie Koessler∗ Centre for the Governance of AI Jonas Schuett Centre for the Governance of AI

Response to the NTIA AI Accountability Policy Request for Comment Everett Thornton Summer Research Fellow Centre for the Governance of AI thorneverett@gmail.com Jonas Schuett Research Fellow Centre for the Governance of AI jonas.schuett@governance.ai Markus Anderljung Head of Policy Centre for the Governance of AI markus.anderljung@governance.ai Lennart Heim Research Fellow Centre for the Gover…

Vol.:(0123456789)1 3AI and Ethics https://doi.org/10.1007/s43681-023-00289-2 ORIGINAL RESEARCH Auditing large language models: a three‑layered approach Jakob Mökander1,2 · Jonas Schuett3,4 · Hannah Rose Kirk1 · Luciano Floridi1,5 Received: 17 February 2023 / Accepted: 18 April 2023 © The Author(s) 2023

2023-5-25 Model evaluation for extreme risks Toby Shevlane1, Sebastian Farquhar1, Ben Garfinkel2, Mary Phuong1, Jess Whittlestone3, Jade Leung4, Daniel Kokotajlo4, Nahema Marchal1, Markus Anderljung2, Noam Kolt5, Lewis Ho1, Divya Siddarth6, 7, Shahar Avin8, Will Hawkins1, Been Kim1, Iason Gabriel1, Vijay Bolina1, Jack Clark9, Yoshua Bengio10, 11, Paul Christiano12 and Allan Dafoe1 1Google DeepM…

SPEAR PHISHING WITH LARGE LANGUAGE MODELS Julian Hazell Oxford Internet Institute, University of Oxford Centre for the Governance of AI julianchristianhazell@gmail.com December 14, 2023

• 1 A survey of expert opinion | May 2023 Towards best practices in AGI safety and governance Jonas Schuett, Noemi Dreksler, Markus Anderljung, David McCaffary, Lennart Heim, Emma Bluemke, Ben Garfinkel Towards best practices in AGI safety and governance: A survey of expert opinion Jonas Schuett⇤ Noemi Dreksler Markus Anderljung David McCaffary Lennart Heim Emma Bluemke Ben Garfinkel Centre for…

How to design an AI ethics board Jonas Schuett∗ Centre for the Governance of AI jonas.schuett@governance.ai Anka Reuel∗ Stanford University anka@cs.stanford.edu Alexis Carlier Centre for the Governance of AI alexispcarlier@gmail.com

RECENT TRENDS IN CHINA’S LARGE LANGUAGE MODEL LANDSCAPE

“Democratising AI”: Multiple Meanings, Goals, and Methods Elizabeth Seger1 Centre for the Governance of AI Oxford, UK elizabeth.seger@governance.ai Aviv Ovadya Harvard Berkman Klein Centre Cambridge, MA aviv@aviv.me Ben Garfinkel Centre for the Governance of AI Oxford, UK ben.garfinkel@governance.ai Divya Siddarth Collective Intelligence Project Oxford, UK divya@cip.org Allan Dafoe Centre for t…

Exploring the Relevance of Data Privacy-Enhancing Technologies for AI Governance Use Cases Emma Bluemke*1, Tantum Collins1, Ben Garfinkel1,2, Andrew Trask1,2,3

Response to the UK’s Future of Compute Review: A Missed Opportunity to Lead in Compute Governance Jess Whittlestone (Centre for Long-Term Resilience) Shahar Avin (Centre for the Study of Existential Risk) Lennart Heim (Centre for the Governance of AI) Markus Anderljung (Centre for the Governance of AI) Girish Sastry (OpenAI) We are pleased to see the publication of the UK’s Future of Compute Re…

Risk management in the Artificial Intelligence Act Jonas Schuett* The proposed EU AI Act is the first comprehensive attempt to regulate AI in a major jurisdiction. This article analyses Article 9, the key risk management provision in the AI Act. It gives an overview of the regulatory concept behind Article 9, determines its purpose and scope of application, offers a comprehensive interpretation…

• 1 Lessons from the Development of the Atomic Bomb Toby Ord | November 2022 1 Lessons from the development of the atomic bomb Toby Ord* The creation of the atomic bomb is one of the most famous and well-studied examples of developing a transformative technology — one that changes the shape of human affairs. Teams of scientists and engineers in many different countries knew in advance that the…

Safety Not Guaranteed: International Races for Risky Technologies Eoghan Stafford, Robert F. Trager, Allan Dafoe November 2022

Charlotte Siegmann* and Markus Anderljung* | August 2022 The Brussels Effect and Artificial Intelligence: How EU regulation will impact the global AI market

© 2022 Published for the Foreign Policy Research Institute by Elsevier Ltd. The Security Governance Challenge of Emerging Technologies August 2022 Robert F. Trager Abstract: In recent decades, governments have been ineffectual at regulating dangerous emerging technologies like lethal autonomous weapons and synthetic biology. In today’s era of great power competition, changing course is difficul…

Future of Compute Review - Call for Evidence Response by the Centre for the Governance of AI August 5, 2022 Lennart Heim Research Scholar Centre for the Governance of AI lennart.heim@governance.ai Markus Anderljung Head of Policy Centre for the Governance of AI markus.anderljung@governance.ai About the Centre for the Governance of AI (GovAI) The Centre for the Governance of AI (GovAI) is a nonp…

The IAEA Solution: Knowledge Sharing to Prevent Dangerous Technology Races Eoghan Stafford∗ and Robert F. Trager† July 2022

FORECASTING AI PROGRESS: EVIDENCE FROM A SURVEY OF MACHINE LEARNING RESEARCHERS Baobao Zhang,1,2∗ Noemi Dreksler,1 Markus Anderljung,1 Lauren Kahn,3 Charlie Giattino,4 Allan Dafoe,1,5 Michael C. Horowitz6 1Centre for the Governance of AI 2 Maxwell School of Citizenship and Public Affairs, Syracuse University 3Council on Foreign Relations 4Our World In Data, University of Oxford 5DeepMind 6Perry…

Comments on the interim report of the National Artificial Intelligence Research Resource Task Force June 30, 2022 Lennart Heim Research Scholar Centre for the Governance of AI lennart.heim@governance.ai Markus Anderljung Head of Policy Centre for the Governance of AI markus.anderljung@governance.ai About the Centre for the Governance of AI (GovAI) The Centre for the Governance of AI (GovAI) is…

Information Hazards in Races for Advanced Artificial Intelligence [Preliminary draft: Do not cite] Nicholas Emery∗ , Andrew Park† , Robert Trager‡ June 2022

Preview – Microsoft Responsible AI Standard v2 – Introduction 1 Microsoft Responsible AI Standard, v2 GENERAL REQUIREMENTS FOR EXTERNAL RELEASE June 2022 2 Index About this release ....................................................................................................................................................................................................... 3 Accountability…

Center on Regulation and Markets Working Paper #2 Aligned with whom? DIRECT AND SOCIAL GOALS FOR AI SYSTEMS Anton Korinek and Avital Balwit This working paper is available online at: https://www.brookings.edu/center/center-on-regulation-and-markets/ The Center on Regulation and Markets at Brookings creates and promotes rigorous economic scholarship to inform regulatory policymaking, the regulat…

Comments on the Initial Draft of the NIST AI Risk Management Framework Jonas Schuett Research Fellow Centre for the Governance of AI jonas.schuett@governance.ai Markus Anderljung Head of Policy Centre for the Governance of AI markus.anderljung@governance.ai 29 April 2022 About the Centre for the Governance of AI (GovAI) The Centre for the Governance of AI (GovAI) is a nonprofit based in Oxford.…

The Artefacts of Intelligence: Governing scientists' contribution to AI proliferation Toby Shevlane Faculty of Law (Centre for Socio-Legal Studies), University of Oxford Supervisor: Dr Bettina Lange Abstract: This DPhil dissertation is about attempts to govern how artificial intelligence (AI) researchers share their work. There is growing concern that the software artefacts built by AI research…

Emerging Technologies, Prestige Motivations and the Dynamics of International Competition Joslyn Barnhart February 2022 Abstract: The study of international races has focused almost exclusively on security motivations for competitive arming. But international races may also be motivated by prestige. This article defines a “prestige race” and outlines the significant ways in which prestige motiv…

This is the author's version of the work. It is posted here by permission of the AAAS for personal use. The definitive version was published in Science , (2021-12-10), doi: 10.1126/science.abi7176 https://www.science.org/doi/10.1126/science.abi7176 Filling gaps in trustworthy development of AI Incident sharing, auditing, and other concrete mechanisms could help verify the trustworthiness of act…

Analysis and lessons learnt from NeurIPS Broader Impact Statements CAROLYN ASHURST∗, Alan Turing Institute EMMIE HINE∗, Oxford Internet Institute PAUL SEDILLE∗, Harvard Kennedy School ALEXIS CARLIER, Future of Humanity Institute Ethics statements have been proposed as a mechanism to increase transparency and promote reflection on the societal impacts of published research. In 2020, the machine…

Original Research Article Journal of Law and Empirical Analysis 2026, Vol. 3(1) 220–250 © The Author(s) 2026 Article reuse guidelines: sagepub.com/journals-permissions DOI: 10.1177/2755323X261427048 journals.sagepub.com/home/lex AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice Daniel Schwarcz1,* , Sam Manning2,*, J.J. Prescott3,* , Pa…

Bill A6578 — AI Training Data Transparency Act (Memo) — full text and source on file.

1 Amazon’s Frontier Model Safety Framework At Amazon, we look to our leadership principles every day to guide our decision-making. Our approach to AI development naturally follows from our leadership principle “Success and Scale Bring Broad Responsibility.” As we continue to scale the capabilities of Amazon’s frontier models and democratize access to the benefits of AI, we also take responsibil…

Model card published/updated AUG 01, 2025 Gemini 2.5 Deep Think Model Card Gemini 2.5 Deep Think - Model Card Model Cards are intended to provide essential information on Gemini models, including known limitations, mitigation approaches, and safety performance. Model cards may be updated from time-to-time; for example, to include updated evaluations as the model is improved or revised. Technica…

February 2026 Gemini 3.1 Pro Model Card Model Cards are intended to provide essential information on Gemini models, including known limitations, mitigation approaches, and safety performance. Model cards may be updated from time-to-time; for example, to include updated evaluations as the model is improved or revised. See the Google DeepMind site for a comprehensive list of model cards. Publishe…