How not to think about risk

25 Jun, 2026

Here’s a story that is both totally absurd and a lesson on how not to think about risk.

In the early nineties, the US Environmental Protection Agency got into a fight with a company over a toxic waste dump in New Hampshire.

The company had already spent a fortune cleaning the site up. They’d removed enough toxic chemicals that a child could safely play there and even eat small amounts of dirt 70 days a year.

But the EPA weren’t satisfied. They wanted the chemical levels lower. So low that it would have cost another $9.3 million, and left the site clean enough that a child could safely eat the dirt 245 days a year.

But the site was a swamp, there were no children, and no one lived anywhere near it.¹

This craziness comes about because we often treat risk as if it’s a physical property of an object, like its weight. Toxic chemicals are risky, so we must reduce the chemicals.

But that’s not quite right. The sun is a giant nuclear furnace; it’s objectively pretty dangerous. But you aren’t currently being incinerated because you’re 93 million miles away and protected by the atmosphere.

The danger is a property of the sun, but the risk is a property of the situation. And that’s a function of several things happening at once.

For something to be a risk, the underlying thing has to be dangerous, you have to come into contact with it, directly or indirectly, and if you do, it has to affect you.

Here’s one way to think about it:²

Risk = Hazard × Exposure × Vulnerability

If the EPA had thought about risk like this, they might’ve reconsidered. The hazard (toxicity) and vulnerability (biological susceptibility) were understood. But the exposure, aka the chance of a local kid eating swamp dirt, was effectively zero. By obsessing over the hazard, they missed that the lack of exposure had already solved the problem.

Exact numbers are hard to come by, but we don’t need to be precise. Just using this as a mental model gives a practical way to see where to focus and how to reduce the overall threat.

And this way of thinking about risk turns out to be a useful way to look at all sorts of problems, including AI safety.

Whether you’re an engineer working on technical alignment or a policymaker worried about the economy, the equation works the same way.

You could try to reduce the possible hazards by making the models more aligned. But you could also limit exposure by building monitoring tools to catch problems before they spread, or lower vulnerability by making the models more robust to adversarial attacks³ and our institutions and societies harder to break.

I wrote recently about the potential for AI to disrupt the labour market. The hazard there is simply that these models keep getting better at doing useful things, superintelligent AI not required. Exposure is growing because AI adoption in business is growing. And since the UK economy is heavily based on white-collar service jobs, our vulnerability is high.

When you multiply a rising hazard, growing exposure, and high vulnerability together, the overall risk level gets pretty high.

But can this new mental model help us?

Let’s assume UK policymakers can’t easily stop labs from making the models more capable,⁴ meaning the hazard is largely out of their hands.

They could try to tackle exposure by passing laws that restrict AI in the workplace. But if companies aren’t allowed to use these tools, they might become less competitive with their international peers.

So if we can’t stop the models from getting smarter, and we might not want to stop companies from using them, the only practical lever left to pull is making our society less vulnerable to the fallout.

We can debate what those policies might look like,⁵ but what matters most is having a mental model that helps us understand risk, and maybe even decide what to do next.

Thinking this way won’t solve risk or AI safety, but it might stop us from spending $9.3 million cleaning up swamp dirt.

Footnotes

Originally posted on my Substack, which I no longer use.

#ai #safety