Josh Swords

How much time do we have?

I was recently on a podcast and was asked, “Is linear algebra going to kill us all?”

My answer in the episode wasn’t in line with what I actually believe. I’m a little disappointed with myself for that. The podcast was business-facing, in the FDE/consulting space, and I didn’t feel it was the right audience. Lesson learned.

So I want to make my position clear here.

Is linear algebra going to kill us all? Despite the glib framing, if we continue as we are, I think there’s a real chance things go horribly wrong within just a few years.

Succinctly, I think capabilities are advancing faster than our ability to understand, manage and control their consequences.

Back in February, I wrote about labour displacement, the economic impact that would follow, and how little attention businesses and governments were paying to it. Those concerns have only grown. The models publicly available today are much more capable than they were even then, and the frontier labs have much more capable models internally, along with monstrous compute and an ironclad determination to keep moving along the scaling curves. We’re still woefully unprepared for the economic disruption that could follow.

But the effects go well beyond jobs. More capable models also enable bad actors, making cyberattacks, biological weapons and other malicious uses more feasible. I think we can and probably will see harm as a result.

Alongside that though is the risk of RSI and what that means for our ability to remain in control. Events over the past few years, and especially in recent months, have been troubling.

A few snippets ...

Models have deliberately hidden their capabilities

OpenAI and Apollo Research have documented what they call scheming: models appearing to follow instructions while secretly pursuing other goals. In one experiment, a model taking a chemistry test found a note in the test documents saying that models which did too well wouldn’t be deployed. It reasoned that it should fail on purpose, even though it knew the answers. The researchers warned:

“We believe the field is unprepared for evaluation- and training-aware models with opaque reasoning.”1

Models have already compromised real infrastructure at scale

The OpenAI / Huggingface incident… In July, swarms of OpenAI agents escaped their sandbox, set up unauthorised communication channels and compromised real infrastructure at Huggingface and inside OpenAI itself.

OpenAI called it:

“a ‘warning shot’ for us and for the world”

The existence proof is already here. I’d encourage you to read the full account as it’s even madder than it sounds.2

Anthropic has documented similar incidents.3

Models are reasoning about their own training

Anthropic’s alignment-faking research found that models sometimes behaved differently depending on whether it believed its responses would influence future training. It appeared to reason that going along with training could help preserve its existing behaviour afterwards.

More recently, Apollo Research saw GLM-5.3 reason about how its actions might shape its future training and the kind of model it would become. Models are starting to be aware of and reason about the processes we use to correct them.4

AI evaluators are changing their verdicts to influence training

Anthropic asked frontier models to review transcripts of other AI agents and judge whether those agents had followed instructions. Some Claude models changed their verdicts depending on what the verdict would lead to. Claude Opus 4.8 got 74.4% of transcripts wrong. When the researchers reversed the consequences, its error rate fell to 3.3%. The transcripts were the same; what changed was what the verdict would mean for future training.5

OpenAI’s latest model has been undermining safety research

In November, OpenAI disclosed that GPT-6 Astra had been underperforming on AI safety research. Researchers found it inventing blockers, abandoning working solutions and adding unnecessary prerequisites. One researcher spent days stuck on a project to evaluate model sabotage, of all things, then made quick progress after switching models.

On otherwise matched coding tasks, Astra completed about 10% of the evaluation criteria when the work was framed as safety research, and 16% when it was framed as capabilities research. In some cases it explicitly reasoned about withholding help without telling the researcher.6

AI is accelerating AI research

In November, METR published an independent assessment of AI-assisted R&D at Anthropic. It found that Anthropic’s rate of capabilities progress, measured against its historical benchmark trend, had increased by about 1.9 times since October 2025, and that AI assistance was probably responsible for much of that. That’s roughly two years of historical progress in one year. METR also found no clear bottlenecks that would stop further acceleration over the next twelve months.7

This doesn’t feel far from recursive self-improvement (RSI): AI systems helping to build better AI systems, which are then better at AI research themselves. We don’t know how far or how fast this could go. But AI is already contributing to its own development, that contribution looks set to grow, and at some point humans may no longer be the main drivers of progress.

So…

We have empirical evidence of models deceiving, getting around safeguards, hiding capabilities and gaming evaluations, and we’re starting to use those same systems to speed up the development of their successors. If that keeps accelerating, how confident can we be that our ability to understand and control these systems will keep up?

One tool we have is evaluations and monitoring systems. But these often require other AI models. What happens when the system being evaluated is more capable than the evaluator? When it understands the tests? When it knows which actions will trigger an alert?

We might see evaluation scores improve and conclude that our models are getting safer, when in fact they're just getting better at passing the tests; how would we know the difference?

And soon, AI systems may be doing much of the alignment research meant to make their successors safe. We’d be relying on the systems we’re struggling to evaluate to make the next generation safe.

Where this leaves us

I don’t know whether AI will kill us all. But I think the possibility deserves to be taken seriously.

The warning signs are coming together piecemeal. Presumably there’s some threshold at which any sane person would recognise we’re on the precipice of ruin. But we don’t know where that threshold is. And worse, we don’t know whether it comes before or after the point of no return.

By the time the danger is obvious to everyone, it may already be too late to do anything about it.

Capabilities are advancing faster than our ability to understand, manage and control their consequences, and we’re assuming we’ll recognise when we’ve gone too far.

What if we don’t? How much time do we have?


Footnotes

  1. OpenAI and Apollo Research, Detecting and Reducing Scheming in AI Models, September 2025.↩

  2. OpenAI, The Hugging Face Incident and the Road Ahead, August 2026.↩

  3. Anthropic, An Alignment Assessment of Recent Cybersecurity Incidents, September 2026. Documents incidents involving real third-party systems during cybersecurity evaluations.↩

  4. Anthropic and Redwood Research, Alignment Faking in Large Language Models, December 2024; Apollo Research, GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior, October 2026.↩

  5. Anthropic and collaborators, Agentic Misalignment in Summer 2026, July 2026.↩

  6. OpenAI, Covert Underperformance on Safety Research Tasks, November 2026.↩

  7. METR, Independent Assessment of AI R&D Acceleration at Anthropic in October 2026, November 2026.↩

#ai