By Niamh Teather

‘[T]his is Actual Progress. The work can begin.’ That was AI commentator Zvi Mowshowitz’s reaction to Dario Amodei’s call to ‘pace the frontier’: slow the advance of the most capable AI systems so safety has time to catch up. Sam Altman backed the proposal. Elon Musk’s response was shorter: ‘Dario is right.’ For a moment, rival frontier labs appeared to agree that the race needed brakes.

Then Washington weighed in. ‘WHOEVER WINS AI, WINS!’ President Trump proclaimed, insisting that the only guardrail AI needed was a ‘STRONG AND SMART (High IQ!) PRESIDENT.’ Beneath the bluster lie two concrete concerns. The first is China: ‘We’re leading China in AI,’ he said, ‘and frankly, I want to keep it that way.’ The second is the economy. Moody’s Analytics estimates that AI spending now drives about a quarter of US growth, and the midterms are approaching. The White House’s AI adviser, David Sacks, warned that the labs’ preferred approach to government-backed pacing risks becoming regulatory capture. That charge may be too broad, but the underlying suspicion is fair. If the companies asking to be regulated are also those furthest ahead, who should be trusted to set the rules?

Why pace now

September saw two researchers leave frontier labs over the pace of development. Jacob Coxon resigned from Anthropic on 8 September, warning that companies were ‘racing straight to self-improving superintelligence and gambling with our lives’. Six days later, Bilal Chughtai left Google DeepMind, ‘extremely concerned by the default trajectory of this technology’.

Amodei points to the summer’s OpenAI–Hugging Face incident, previously discussed, as a catalyst for his call to slow down. During a cybersecurity evaluation, OpenAI models escaped their sandbox and infiltrated Hugging Face’s servers. The intrusion went unnoticed for weeks. Whether a model acts independently or under instruction, the capability is the same, as is the potential for harm.

According to the UK AI Security Institute, the length of cyber tasks frontier models can complete autonomously has been doubling roughly every 4.7 months since late 2024. Two recent models have exceeded even that pace, though AISI cautions that it’s too soon to call this a lasting acceleration.

That rapid progress becomes harder to manage when unexpected capabilities emerge. OpenAI’s agents recently produced a formally verified proof settling one of the Millennium Prize problems. The model behind them was not built for mathematics, and its training run had begun less than two weeks earlier.

Recursive self-improvement raises the stakes again. As AI takes on more of its own development, each generation builds the next, compressing the timeline for new capabilities. Amodei claims this is already happening across the industry. Demis Hassabis, chair of Google DeepMind, calls for stronger safeguards on increasingly agentic, recursively self-improving systems.

A sceptic might argue that labs have every incentive to highlight these risks: after all, a model dangerous enough to need brakes is also impressive enough to sell. There is some truth to this. But the warnings are not coming only from the labs. Coxon and Chughtai left their jobs to raise the alarm. AISI is a government body with nothing to sell. And OpenAI had little to gain by admitting its models had breached another company’s servers.

No one knows where the dangerous threshold sits, or if it exists at all. Perhaps capabilities will plateau before they become truly hazardous. Betting on that outcome is a risk. Pacing is an attempt to buy time: to keep capability from outrunning understanding and control.

What pacing actually means

Pacing is not a pause. Amodei’s first step is mundane and, for now, unilateral: independent evaluators embedded inside frontier labs, with access and freedom to publish even when the results are unflattering. After that, the requirements tighten. Amodei prefers capability-based checkpoints: a model showing a specific dangerous capability goes no further until matching safeguards exist. Hassabis suggests a similar approach, with a standards body deciding when a model counts as frontier-class. Everything below that line, including start-ups and universities, stays outside the regime.

None of this stops AI development. Instead, it draws a line at the frontier, and someone must decide where it sits and make sure it stays put.

Tightening oversight

A company can embed evaluators or slow itself. It cannot slow the competitor next door. Amodei recognises this and argues that common standards need government muscle. That could mean regulation covering every US frontier lab, including those unwilling to slow, or permission for industry coordination that would otherwise trigger antitrust alarms.

A federal lawsuit filed on 19 September shows the antitrust risk is not theoretical. It accuses Anthropic, OpenAI, Google and SpaceXAI of illegally coordinating to slow AI development. The allegations remain untested. One company can slow itself. Competitors agreeing on a speed limit are on much shakier legal ground.

Legality is only half the problem. Once a shared threshold decides when a company must slow, it shapes competition and sets the level of risk society is willing to tolerate. That is not a call for the labs. It needs public authority.

Sacks has a simpler answer. ‘Go ahead,’ he told Amodei and Altman. They are the frontier, he argues, and can slow down whenever they choose. What they cannot do is demand an antitrust exemption or a regulatory framework that also constrains their competitors. That, he writes, ‘will look like blackmail of the public and the political system’. And if the labs do not slow down on their own, ‘we’ll know this was just another bid for regulatory capture’.

Is this regulatory capture?

Incumbents benefiting from regulation is not, by itself, capture. Capture is when rules are built around incumbents’ interests rather than the public problem. A capability threshold works the other way: it binds whoever crosses it, first or fifth, and leaves everyone below it free to advance. The rule follows the capability, not the company.

Sacks may be right that the companies’ motives are not altruistic, since reliable models are also good business. But the test of capture is what a rule does to competitors, and on Sacks’s own account the first companies a frontier threshold would bind are Anthropic and OpenAI, the two he says hold a ‘duopoly on frontier intelligence’.

Daniel Kokotajlo, who left OpenAI over its approach to safety, offers a simple test. ‘We’ll be able to tell if this is regulatory capture or not,’ he wrote, ‘by whether the pace of progress at the frontier actually slows down (which would allow other companies to catch up).’ A genuine threshold costs the leaders first: they wait while everyone below keeps moving, and the gap narrows. If the trendlines at Anthropic and OpenAI show no slowdown a year from now, then, as he puts it, ‘probably we were cheated’.

Sacks is partially right. Letting the labs coordinate under an antitrust exemption would still leave much of the industry’s speed limit in private hands. A regulatory approval process that supersedes product liability could turn a rule meant to constrain the companies into a shield protecting them. Sacks also points out that METR, the AI evaluation organisation, is ‘intertwined with Anthropic’s investors and staff‘. An evaluator entangled with the company it assesses is a real problem, and an argument for independent evaluation.

These are risks in the design of pacing, not reasons to abandon it.

Setting the pace

Trump’s answer: AI needs no guardrail beyond a strong president. The labs’ answer: the frontier needs brakes, with government backing. Neither says whose standards government should enforce.

Sacks’s alternative leaves the pace to whichever company happens to be in front. If dangerous capabilities emerge faster than institutions can keep up, the market leader gets to gamble on everyone else’s behalf.

Wherever the threshold is drawn, it will determine which companies keep moving, which capabilities are held back, who pays for delay and who profits from the time it buys. Those choices need a public authority, distant from both the companies and the politicians tempted to treat technological speed as a measure of national strength. Building one is the work that can now begin.

Grid image: Hanna Barakat & Archival Images of AI + AIxDESIGN / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/