Existential risk is not something static which lives outside of humanity. It is something humans shape. This can happen directly, when it comes to things which are created by humans, like nuclear weapons or indirectly, like if we sufficiently prepare for major volcanic eruptions or not. This means actors can increase the hazards we face or worsen our vulnerability to existing hazards. But not everybody has the same influence on this. My neighbours do not have any influence on the question if there will be a nuclear war between Pakistan and India. The power to influence such events is usually concentrated in very few hands (1). But whose hands? What kind of organisation? What kind of people? Let’s explore this a bit here.

Individual bad actors

If you are an individual who is set out to destroy the world, chances are high that you aren’t an average joe. What kind of people might do this, is explored in Torres, (2018a) and Torres (2018b). Torres tried to catalog all the kinds of individual actors that could have the potential to go on a rampage to destroy the world. They highlight that we have to take such actors more in consideration now, as dual use technologies with potentially high destructive potential like AI or biotech are having lower and lower barriers to access. Meaning people have an easier time getting their hands on technology that could end civilization as we know it. Also you never know what future scientific breakthroughs have in store, there is always the chance that one day a technology might pop up which is super easily accessible and super destructive (2). The only chance we might have in such a case is to try to understand the actors who might want to use such weapons.

Torres (2018a) first develops a framework in which we can group these kinds of actors. The idea here is that the weapons which bring destruction are interchangeable, but if we understand the motivations for them to be used, there is a chance we can stop them before they are employed. The first insight here is that this means that we can exclude many groups which typically engage in terroristic behaviour. For example, rebels fighting against an authoritarian regime in their home country do not really have much of an incentive to use global weapons of mass destruction. If your goal is to free the people in your lands, it does not really make sense to kill them. This is a bit abstract, but Torres suggests a simple thought experiment to determine who we should consider to be those apocalyptic agents: Imagine there is a button you could press, which would instantly kill all humans. Every person who would push this button is an apocalyptic agent as explained in the paper.

This means we are left with people who think that the current state of Earth (or even the universe) is so irredeemably bad that the only reasonable thing to do is to kill everybody. While this is certainly a very extreme position, we know from history that such people exist. Torres thinks there are six kinds of individuals. While Torres (2018a) gives the framework, Torres (2018b) provides the examples to fill the framework, so I will combine the two here:

Apocalyptic terrorists

These are people who believe in some kind of coming apocalypse, would like to see it sooner and contribute to this end if they can. This often has a religious motivation.

An example of this would be James Ellison. He was the leader of “The Covenant, the Sword, and the Arm of the Lord”, a far right, anti-government militia in the United States. While they started as a “normal” Christian fundamentalist cult, Ellison formed them into an active terrorist group. He had the idea the Second Coming of Christ would only happen after an apocalyptic race war between white Europeans and everybody else. This means if he would be able to start a world destroying race war, Christ would come again sooner. Ultimately, they weren’t really successful in attaining any of their goals, but one could easily imagine them pushing the “end of the world” button.

Misguided moral actors

Misguided in the context of Torres papers refers to actors who do not subscribe to the long-term transhumanist consequences of classical utilitarianism. This means especially forms like negative utilitarianism, who subscribe to the idea that only the prevention of negative experience matters. Therefore, any world which includes suffering should be avoided. As all worlds contain suffering, none should exist. Or said more simply, dead people cannot feel pain.

Fortunately, this quite extreme view is only held by some philosophers tightly locked away in academia, so we do not really have an individual to point to who tried to destroy the world using this kind of reasoning.

Radical ecoterrorists

These are people who think that Earth/Gaia would be better off without humans. We had our chance to live in peace with this planet, we spoiled it, so we should just be extinct and let this planet find its way back to equilibrium again.

Examples of such ideas in the wild are the fringes of the deep ecology movement. One of the more well known groups who subscribe to this idea is the Gaia Liberation Front. They state that their mission is to extinct all humans, as we cannot be trusted with Earth. To bring this about they think that a pandemic that can only target humans would be most well suited, because other methods are too impractical (like mass suicide) or too destructive (like nuclear war).

Idiosyncratic actors

This group is essentially everybody who wants to destroy the world, but does not really fit any major category. Meaning these are likely people who suffer from mental health problems like a psychosis, which implants the idea in them that the world would be better off if nobody existed on it. Or traumatized actors, who feel like the world has wronged them and want to take revenge.

We can find many such actors. A more famous example is Eric Harris, who was the shooter at the 1999 Columbine High School massacre. In his diaries he explained that he wanted to see the world burn and kill mankind. Fortunately, he did not have the means to do so, but we cannot rest assured that future idiosyncratic actors will not have more powerful tools at their disposal.

Non-human actors

Besides these actors where we can give historical examples, Torres also argues that there are two further, more hypothetical groups: value-misaligned machine superintelligence and belligerent extraterrestrials. But for obvious reasons we can only speculate about the ideas and incentives for such actors.

Institutional bad actors

While it seems far-fetched that an individual could destroy the world today, even if they have motivations as outlined above, institutions which can endanger the world already exist. This is explored in a short overview article by Kemp (2021) (3). In it Kemp argues that existential risk is mainly driven by big, institutional actors, as everybody else has too few resources to meaningfully move the needle.

The pathways that have been identified so far to end the world are things like artificial general intelligence, catastrophic biological threats, climate change, lethal autonomous weapons and nuclear weapons. All of these pathways share that they include a lot of effort. You cannot build nuclear weapons in your backyard, neither can the lifestyle of an individual meaningfully influence how many ppm carbon dioxide are in the atmosphere. To do these things you have to have lots of power and lots of resources. Something only very large actors like the military-industrial complex in different countries, the fossil fuel industry or Big Tech can accomplish. In many (or maybe all?) of these institutions we can see that their profits are privatized, while the potential downsides are public. Let’s look at some examples:

• Artificial general intelligence: There are very few AI companies who actually build new models. Presently the biggest ones are OpenAI, xAI, Anthropic and Deepseek. It is unclear if their models even could lead to AI, but what is clear is that it is extremely expensive to work on them, because they need large amounts of electricity and compute to be build and run.
• Climate change: Globally around 100 companies are responsible for the vast amount of carbon dioxide emissions. Similarly, more than half of the global cumulative emissions fall to just three main actors: the US, EU-27 and China. Finally, only six countries hold 80 % of the global fossil fuel reserves.
• Nuclear weapons: Almost 90 % of all nuclear weapons are held by two countries: the United States and Russia. And for the remaining 10 % you just have to add 7 other countries. The maintenance and production of American weapons is bundled up into just 28 companies. The command chain to use those weapons are quite short. In the United States the president has sole authority to order a nuclear strike. There are four key advisors, but their consent is not required.

All this means that human-made catastrophic hazards are highly concentrated. The United States military could easily be considered as the largest contributor to global risk. They emit gigantic amounts of carbon dioxide, they control the most advanced nuclear arsenal, are at the forefront of developing autonomous weapons and surveillance technologies and have by far the largest military in the world.

What makes such actors so tricky to reign in is that they often use their power and resources to distribute misinformation, so public opinion is swayed and they can continue their mission. Clear examples here would be NATO trying to discredit nuclear winter research (4) or the fossil fuel industry financing large misinformation campaigns for decades. Similarly, these powerful organizations often lack efficient oversight, which allows them to keep risks secret. Like the fossil fuel industry knowing about the fact of global warming for decades.

In the future, some of those pathways to global catastrophes might get into the reach of individual actors, but for now it seems pretty obvious who creates global risk and we should focus our efforts on those actors. The motivation of these actors is likely not to ruin the world, but we cannot get around the fact that they do endanger us all. They are driven to accept these risks because they have a profit motive and face competitive dynamics.

Policy around global risk

This concentration of global catastrophic risk in the hands of relatively few powerful institutions raises a critical question: how do those responsible for governing these risks actually perceive the challenge? Understanding how policymakers navigate this landscape of concentrated power is essential, as they represent one of the few potential checks on these institutional actors. However, policymakers themselves face significant constraints. Research by Nathan & Hyams (2022) provides direct insight into how those closest to global catastrophic risk governance perceive their own capacity to address these challenges, revealing both the opportunities and structural barriers that shape policy responses to our most dangerous institutional actors. They interviewed 16 civil servants, civil society group representatives and individuals from the private sector in the UK, US or Europe and tried to distill these interviews. This brought up four themes on how these people think about global risk:

• Scepticism: They realize that current solutions are often lacking and that the government is too often just in a reactive role. New systems are needed to plan before disaster happens. This skepticism directly reflects the challenge of regulating the powerful institutional actors identified earlier. As one UN policy lead put it regarding pandemics: “if a crisis hits tomorrow, it’s too late.”
• Realism: Global catastrophic risks are just a very complex problem, which often requires long times to implement solutions. This is just really hard in a policy world which is focussed on the short term. As one health security expert noted, there’s “realpolitik because you can have as many agreements as you want, but if you’ve got the UK and the United States producing something, politically, their first obligation is clearly their own citizens.” This reflects the challenge of governing institutional actors whose influence spans national boundaries while policymakers remain constrained by domestic political pressures.
• Influence: Individual policymakers feel like they can be a force for good when they do good work on global catastrophic risk. For one to implement potential solutions, but also just simply by making the topic something that can get more easily discussed. One interviewee emphasized the importance of formative conversations with “people a lot brighter than me who are maybe going to end up in a sovereign wealth fund, maybe advising the government of Saudi Arabia or Abu Dhabi… If their framing of the world is just really, really slightly different and then they’re making a decision that relates to tens of billions of dollars being allocated in a particular area rather than on another.”
• Governance outside government: The idea here is that governance also happens a lot outside of government. This means we should not only rely on the policy world, but also work on creating more accountable governance structures in other places like academia or companies. As one expert on emerging biotechnology policy stated: “where I find the most optimism is where I dream of creating supplementary spaces that create bridges between regulation and scientists that sit just under regulatory decision-making that augment the process.” This reflects recognition that traditional state-based governance may be insufficient to control the institutional actors driving global catastrophic risk.

The research reveals a fundamental tension: policymakers understand the risks posed by concentrated institutional power, but feel constrained by the very power asymmetries this concentration creates. They see opportunities for influence through language, agenda-setting, and non-governmental governance mechanisms, yet remain skeptical about their ability to fundamentally alter the incentive structures that drive institutional bad actors.

Conclusion

My takeaway from all this is that large risks from individual actors is something we should be concerned with and prepare for, but the ultimate problem is not them, but some of the institutions which have been built to shape and control parts of society. Their incentives are often misaligned with society and we need more accountability and control over them. Policy makers are thinking about this, but it is just hard. The research shown here highlights that the actors posing the most risk globally are often held unaccountable and even the policy makers who think most about them, are unsure how they can make a difference. This means we need to restructure our system, so accountability exists again. And I might just sound like a broken record, but I think this is again an area where we can see that we have to make our society more democratic, that we have to experiment with new ways to distribute power. It might not fix all of our problems, but more decentralized decision making does lend itself less to unaccountable power centers by definition (5).

Endnotes

(1) This is especially the case for nuclear weapons, as the short time span to decide if you want to launch the weapons or not, makes it impractical to include a lot of shareholders. Kurzgesagt has a nice video which highlights this urgency: https://youtu.be/wmP3MBjsx20

(2) A more formal explanation of how this might come about can be found in Nick Bostrom’s vulnerable world hypothesis.

(3) Yes, I know that this is a newspaper article and not a paper, but I think it is a pretty good source and also includes lots of references to underline its arguments. You can also find a more detailed account of this in chapter 20 of Luke Kemp’s Goliath’s Curse book, which I summarized in another post.

(4) I have written about this here.

(5) See here for a more detailed argument why I think democracies are essential.

References

• Kemp, L. (2021). Agents of Doom: Who is creating the apocalypse and why. https://www.bbc.com/future/article/20211014-agents-of-doom-who-is-hastening-the-apocalypse-and-why
• Nathan, C., & Hyams, K. (2022). Global policymakers and catastrophic risk. Policy Sciences, 55(1), 3–21. https://doi.org/10.1007/s11077-021-09444-0
• Torres, P. (2018a). Agential risks and information hazards: An unavoidable but dangerous topic? Futures, 95, 86–97. https://doi.org/10.1016/j.futures.2017.10.004
• Torres, P. (2018b). Who would destroy the world? Omnicidal agents and related phenomena. Aggression and Violent Behavior, 39, 129–138. https://doi.org/10.1016/j.avb.2018.02.002