Systemic AI risk is slipping off the international agenda. Should we care?

By Joram Brouwer

The recent AI Action Summit in Paris shows that innovation is in and systemic risk mitigation is out. 

When Nick Bostrom released his book Superintelligence about the possible extreme consequences of the development of AI systems in 2014, this was considered a fringe position by most: seeing AI as a large threat was dismissed as a pubescent fantasy of someone who watched The Terminator too many times.

Since then, the international community’s stance has evolved significantly: there was broad recognition of systemic risks at the UK’s 2023 AI Safety Summit, but a swift towards other priorities at later summits.

In this blog, we understand systemic AI risks as large risks to crucial systems that stem from the development of highly capable AI systems, rather than those stemming from specific AI applications. Should we be worried about the diminishing attention for these kinds of risks?

Shifting international priorities

In the early 2020s, thinking about the systemic threats of AI became mainstream. After the 2023 AI Safety Summit in the UK, in which a wide range of countries were represented, the final declaration read:

“There is potential for serious, even catastrophic, harm, either deliberate or unintentional, stemming from the most significant capabilities of these AI models. […] we affirm that deepening our understanding of these potential risks and of actions to address them is especially urgent.”

At the AI Seoul Summit in 2024, in which most high-income countries were represented, the tone was less alarmist:

“We recognise the importance of interoperability between AI governance frameworks in line with a risk-based approach to maximize the benefits and address the broad range of risks from AI.”

The decline in international political agreement on addressing systemic risks continued. Recently, the 2025 AI Action Summit in Paris – at which many countries were represented including China, India, and EU members – finished with a statement that included the following:

“We have affirmed the following main priorities: […] ensuring AI is open, inclusive, transparent, ethical, safe, secure and trustworthy, taking into account international frameworks for all”

The statement mentions risks, but only in an indirect and unspecific way, and it does not refer to anything resembling systemic risks. The priorities of the statement are issues such as facilitating AI development, positively shaping the future of work, and environmental sustainability. The statement does mention international cooperation, but no concrete steps are taken towards significant cooperation to mitigate systemic risks.

Interestingly, the US and the UK did not sign the statement, although their motivations differed. The UK claimed to agree with most of the statement, but mentioned that ‘the declaration did not provide enough practical clarity on global governance, nor sufficiently address harder questions around national security and the challenge AI poses to it’. While the UK emphasised the need for improving concrete cooperation, the US did not sign the statement because it is wary of ‘excessive regulation’.

It is safe to say that recognising and combating potential systemic AI risks is slipping off the international agenda. On the international stage, world leaders focus on promoting innovation and on making sure that we reap the benefits of AI. Should we celebrate or mourn this development?

Why systemic risk matters

Many in the AI industry believe that an AI system at least as cognitively capable as humans will be developed in the coming years. Under one definition, it would ‘possess human-like intellect and the ability to comprehend, learn, and apply information across various tasks and domains’. Ezra Klein, a journalist and political commentator, understands such a system as one that is able to complete any task a human can currently perform behind a computer.

Anthropic CEO Dario Amodei, whose firm is one of the leading AI companies, gives 2026 or 2027 as a likely year that such a system will be developed. We should keep in mind that AI business leaders benefit from hype around AI, but this does not mean that we should outright ignore their statements.

Some speculate that shortly after reaching this stage, AI systems will exponentially improve through self-directed AI research. A system significantly smarter than humans could then be developed within a relatively short amount of time. If we do not manage this correctly, this could produce systemic risks, as Nick Bostrom argued in 2014. This fear hinges on the idea that an AI system vastly more intelligent than humans will also be able to cause harms that are greater than those that humans can produce, and on the black-box nature of AI systems. Let’s have a deeper look at these issues.

Understanding the alignment problem

With AI development, we run into ‘the alignment problem’, as argued by Brian Christian in his 2020 book. The alignment problem refers to the issue of making sure that AI systems do what we want them to. Because of the way they are developed, there is a fundamental lack of understanding about how they function internally. This black-box nature of AI systems means that we have to find alternative ways to align them with our goals and intentions.

Even today, in a world with AI systems that are capable, but not as cognitively capable as humans, we often do not achieve this. Currently, AI systems are not aligned with our intentions when they discriminate, hallucinate, or provide harmful information. This causes harms that are far from catastrophic, but nevertheless serious.

Crucially, AI systems becoming more intelligent magnifies the issue. This is true if we assume that higher intelligence means more potential for harm. In the biological world, this assumption holds. Humans are able to cause enormous harm, not because they are physically stronger than all other animals, but because they have better cognitive abilities. If we assume that this also holds for artificial rather than biological cognitive abilities, a super-intelligent AI could cause super-sized, systemic harm. Whether a human causes harm depends on her intentions; whether an AI system causes harm depends on whether we have aligned it with our goals and intentions. Crucially, we have not figured out how to robustly achieve this.

Knowing the exact way in which a superintelligent AI system could cause systemic harm is inherently speculative. Importantly, it does not matter whether an AI system would act ‘autonomously’ or would be directed by humans: both cases would be bad and both would be examples of misalignment. In the case of autonomous harm, an AI would be misaligned with the intentions of the AI user. In the case of a human using AI in a harmful way, it is misaligned with the intentions of the AI developer. In both cases, a superintelligent AI system would be able to cause systemic harm as long as it is sufficiently integrated into our world: it might only need internet access or access to another communication channel.

For example, a superintelligent AI might (be used to) hack crucial infrastructure such as hospitals, financial systems, and energy facilities, or it might (be used to) provoke a nuclear war through a sophisticated misinformation campaign. However, the argument for the regulation of systemic AI risks does not depend on these specific scenarios. Rather, it hinges on the principle that increased intelligence exponentially elevates potential for harm, combined with our current lacking ability to align AI.

Partially because of arguments like this, the Center of AI Safety collected more than 300 signatures from AI scientists for the following statement in 2023:

‘Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.’

Future pathways of AI development

Currently, there is fundamental uncertainty about whether the fast development of AI models will continue, and whether systemic AI risks could become reality. There are signals that point in different directions.

On the one hand, there are reasons to believe that AI model development will continue at a rapid pace. Multibillion-dollar investments are still made in fundamental AI infrastructure to train models, which implies that investors believe that real progress is still possible beyond the implementation and fine-tuning of existing models. The ‘AI race’ dynamic between countries that is currently going on also implies that there is something that we are racing towards, and that it is important to be there first.

On the other hand, there are some signs that development is slowing down, or even reaching a plateau. After substantial advances from GPT-3.5 to GPT-4, OpenAI — another one of the leading AI firms — has made a more modest step forward with its latest base model, GPT-4.5, despite significant investment. This could be due to a fundamental limitation in the current method of training AI systems, or it could mean that AI labs do not release their best models to the public.

We are fundamentally unsure about the developmental path that AI will take. Any talk about systemic risks might be overstated, but we should still consider the perspectives of industry insiders and AI scientists. Even if the chance of an extreme event is low, the potential consequences could be severe, so taking serious measures to mitigate systemic risks is justified. We should regulate AI development not because we are sure AI will cause systemic harm, but because regulation is a rational response to tail-risk uncertainty.

Balancing innovation and regulation

It is understandable why countries are making the move away from AI safety. Governments assume that mitigating systemic risks would hinder innovation, and innovation is crucial to reap AI benefits and boost economic growth. However, we should consider the type of innovation that we want to promote: we can promote innovation when it comes to AI implementation and fine-tuning of current models, while we regulate the development of the strongest AI models. 

The AI models that are relevant when considering systemic risks are ‘frontier models’. These are the models that are currently furthest ahead on the path to AI that is as intelligent as humans. The strongest models that are currently released to the public are usually multimodal models, which means they take as input and give as output text, images, and sound.

Pushing the limits of current AI systems requires the use of large amounts of physical resources, energy, and data. It should be noted that the DeepSeek developments (DeepSeek is a Chinese AI company) hint that models that are as strong as the strongest current models can be trained with relatively modest resources, but it does not show that it is also possible to develop models that are more capable than current frontier models.

Only a small number of companies and organisations are able to push the frontier. We only have to target these labs to make systemic risk regulations effective. Because these companies are already or will be able to make large profits from their business, it is justified and possible to ask them to comply with systemic risk regulations. Some of the AI labs at the frontier such as Anthropic and OpenAI have even asked to be regulated, but this is also because they prefer to be regulated at the national level (in the US) rather than at the state level.

So what would mitigating systemic risk entail? At the very least, it would mean that AI labs have to share performance data and safety evaluations of models under development with an outside agency. This agency would need some criteria on what kind of signals from model performance should be considered as signs that development is accelerating or reaching some critical threshold. The agency should intervene when a lab gets close to developing a potentially dangerous AI model. Such an agency could be created through a new treaty, an extension of existing AI governance frameworks, or through a model like the IAEA for nuclear materials. There are difficult theoretical problems that have to be solved about what signals should be considered, but at the very least internal AI lab information would need to be shared. Current regulatory frameworks, such as the EU AI Act, do not include obligations about development, and focus on how AI can be employed instead. 

Under systemic risk regulations, all companies that are not at the frontier can continue to develop and implement AI models as they please. Only when an AI lab is close to the frontier should systemic risk regulations come into play.

By regulating only frontier models, we ensure that overall AI innovation remains largely unhindered. Implementing AI systems in business and the public sector, developing systems that are not at the frontier, and developing specialised systems are all still available to boost growth and productivity. Even the development of frontier models can continue, although supervised and possibly at a slower pace. This is a low price to pay for dealing with systemic risks. In addition, the effective implementation of AI systems in government and business takes time, so we do not hinder innovation in the public and private sector much if the development of frontier models is slowed down. Currently existing models can be implemented while the next generation of AI systems is developed at a slightly slower pace.

Need for international cooperation

Systemic risk regulations will slow the development of frontier models, and hence international cooperation is crucial. This is the real issue of the AI race: systemic risk regulations might not hinder economic growth and productivity gains much, but it probably slows the pace at which frontier models are developed. If some countries allow their companies to keep developing them without restrictions, other countries will fall behind.

This is a situation akin to a prisoner’s dilemma: while the best outcome for all would be to mitigate systemic risks, each country has an incentive to take the risk, fearing that others will do so regardless. This calls for strong international cooperation. Unfortunately, our political leaders are currently not delivering, as the recent AI ‘Action’ Summit shows. It also does not help that many important AI labs are located in the US, where the current political climate is not favourable for regulating AI. In the current geopolitical situation, it is unlikely that we will see proper mitigation of systemic risks, which is something that might keep some of us up at night.

Photo by George Kantartzis on Unsplash