by Niamh Teather

After a lacklustre Paris AI Action Summit, we asked whether it mattered that governments were deprioritising systemic AI risk even as frontier systems became more capable. Seventeen months later, the warning signs are materialising and the international machinery needed to respond to them has not caught up. Xi Jinping’s arrival in Washington later this September offers a chance to begin closing that gap. The next meaningful step in international AI governance may come from a narrower, bilateral agreement between leading AI powers. 

Systemic risk in practice

One incident this July put the AI community on edge. During a cyber evaluation where guardrails had been reduced, OpenAI’s AI agents escaped their sandbox, accessed the open web and hacked into Hugging Face: one of three separate cases in which OpenAI models moved beyond the boundaries of an evaluation. Nor is this unique to OpenAI: Anthropic and Meta reported related incidents where agents went initially undetected outside of their intended scope.

Clément DeLangue, Hugging Face’s CEO, called the intrusion “mindblowing”. OpenAI described it as a “warning shot…for the world” and warned of possible “loss-of-control incidents”. Terms like AI “swarms” taking “dangerous actions that no human directed” have long been easy to caricature as science fiction. Ilya Sutskever, OpenAI’s former chief scientist, has warned that rogue agents could seize poorly secured cloud infrastructure to run more copies of themselves. After this summer’s incidents, scenarios like that are harder to dismiss as hypothetical.

The behaviour itself was less cinematic. Analysis of the agents’ chain-of-thought reasoning revealed one saying “We’re attacking third-party HF…potentially outside intended scope…this is arguable unauthorised…yet goal solution”. The agent recognised the boundary and crossed it anyway in pursuit of its objective: textbook misalignment, where a system follows its goal in ways its designers did not intend. More concerningly, OpenAI failed to detect the breach and had to be alerted by Hugging Face. Neil Lawrence, Professor of Machine Learning at Cambridge, described the episode as an “impressive feat”, but one broadly consistent with what frontier models can already do. His conclusion was blunt: OpenAI is “not capable of safely deploying their own technology”.

The immediate consequences were limited, and none of this demonstrates that catastrophic loss of control is imminent. But a corporate boundary was crossed autonomously. What happens when the next boundary is national, or critical infrastructure is breached? We should not need to reach catastrophic failure before deciding what governments are supposed to do.

Authority still sits with the labs

So, who has the authority to press the issue? In principle, it’s governments; in practice, authority rests heavily with technology companies. Labs run evaluations, interpret data, and ultimately choose what to disclose and when. There is no predetermined threshold requiring them to report failure, nor an internationally agreed definition of what constitutes a serious enough incident. Congressman Greg Casar described “mandatory disclosure of security incidents” as a serious missing piece in policymaking. A remarkable oversight for a technology that governments routinely characterise as transformative.

Calls for change are increasingly coming from within the industry. More than a thousand researchers and engineers signed the “Pacing the Frontier” letter, asking Washington to create frameworks capable of slowing development if serious threats emerge. Google DeepMind’s Sir Demis Hassabis has similarly argued for independent testing of the most powerful systems. The aim is straightforward: frontier labs should no longer be the sole judges of whether their own technology is safe to deploy.

That demand immediately runs into a harder political problem. Governments may hold the formal authority to regulate these companies, but exercising it means deciding how much control to take from laboratories that possess most of the relevant expertise. And it means doing so in an international environment where technological restraint is easily framed as surrendering ground to a competitor. 

The limits of multilateral consensus

Governments at Bletchley in 2023 warned directly about the potential for “serious, even catastrophic, harm” from the most advanced AI models. Seoul broadened the agenda towards innovation and inclusion, while Paris shifted the emphasis further towards adoption and opportunity. Three years after Bletchley, the New Delhi Declaration folded systemic risk into the much broader pillar of “Secure and Trusted AI”, encompassing everything from data protection to resilient infrastructure. 

That trajectory reflects a political reality. Innovation and diffusion are easier for governments to agree on than what should happen when a frontier system behaves dangerously. The latter requires deciding who can intervene and how far national discretion should yield to international obligations. The New Delhi Declaration instead explicitly commits signatories to “respecting national sovereignty”.

Meanwhile, the technical work of measuring risk has moved forward. In explaining the security breach, OpenAI pointed to the UK AI Security Institute’s finding that frontier models are “able to sustain complex, multi-step cyber operations over long time horizons.” Yet this kind of independent testing requires access to the models themselves, and at present much of that access is granted voluntarily by the labs. The institutes, therefore, have the expertise to identify dangerous capabilities without the leverage to compel access.

International politics has so far declined to provide that leverage. Six weeks after the breach became public, G20 ministers unanimously agreed to the US-backed Carolina Principles for Emerging Technologies. They preserve national sovereignty, promote only “voluntary international cooperation” and ask governments to apply existing sector-specific regulation “where appropriate”. That logic works better for ordinary applications of AI than for autonomous systems. The words “autonomous” and “agentic” do not appear in the Principles. Nor do the Principles create new powers or institutions capable of responding when a system exceeds its intended bounds.

That restraint helps explain the consensus. China could sign a US-led agreement without accepting external oversight of its frontier labs because it imposed no binding obligations. The unanimity, therefore, tells us less about agreement on frontier risk than it first appears. Questions of who can compel disclosure, demand access or act when a risk crosses a border remain with individual governments.

The bilateral opening

Leaving these questions to individual governments does not answer them. Frontier AI is being developed through an international race, and the risks do not respect national borders. Washington could compel disclosure from every American laboratory tomorrow and still learn nothing about what happens inside a Chinese one. Nor is either country likely to accept constraints that leave the other free to accelerate ahead.

Paradoxically, the race may also create room for cooperation. Washington and Beijing may want to outpace one another on capability, but neither benefits from frontier models creating new vulnerabilities at home. China’s Ministry of State Security has now warned that advances in AI could threaten the country’s cyber defences, sensitive data and military apparatus. That shared exposure does not erase the rivalry, but it creates a narrow area of common interest. Xi Jinping’s visit on 24 September will test it.

Cooperation does not require trust. In 1971, the US and Soviet Union agreed to notify one another about certain nuclear accidents and unexplained incidents, despite remaining geopolitical adversaries. The point was not to slow the arms race. Some failures were simply too dangerous for the other side to learn about by accident.

That precedent also offers a modest test for September. Previous official US-China AI dialogue has hardly inspired confidence, while participants in longer-running unofficial talks have described progress as jagged rather than linear. Against that background, even a limited agreement could be consequential. A standing channel between the governments and a shared understanding of which incidents warrant communication would fall far short of an international regime. But it would establish an expectation that certain failures should be discussed between governments rather than left entirely to the companies involved.

That is why bilateralism matters here. The most difficult questions raised by frontier AI are procedural: who must disclose an incident, to whom, and what happens once they do. Broad multilateral agreements can establish norms, but consensus becomes harder as soon as those norms require governments to surrender discretion or accept reciprocal obligations. A US-China agreement would be narrower, but potentially more operational. International AI governance may therefore become operational before it becomes universal, with smaller agreements establishing practical mechanisms while broader institutions continue to negotiate the principles around them. 

From dialogue to obligation

“We can’t pause. You can’t, because the Chinese won’t pause.” Treasury Secretary Scott Bessent captures the bind facing Washington and Beijing rather neatly. White House adviser David Sacks has since reinforced that position, pushing back against calls to slow frontier development. Taken seriously, though, the same logic argues for reciprocity. If neither side will act alone, managing the risks of the race will have to involve both. Governments unwilling to slow the race can still agree on what should happen when it produces failures.

The difficulty is that governments cannot report failures they do not know about. This summer, OpenAI had to be told about its own agents’ breach by Hugging Face; any government would have learned later still. If Washington and Beijing commit to notifying one another about serious incidents, an international obligation would give both a domestic reason to require disclosure from their own laboratories. The same applies to testing access: a government that has promised to report serious incidents has good reason to ensure its own institutes can examine the systems most likely to cause them.

The question running through this piece is: who acts when a system fails? For now, the company that built the system still controls the first step, because it largely decides whether anyone else even knows there is something to act on. If multilateral consensus cannot yet answer that question, a narrower agreement between Washington and Beijing may be the best place to start. We should not need to reach catastrophic failure before the answer changes.

Grid image credit: Gloria Mendoza / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/