Table of Contents
[hide]
Artificial intelligence is improving faster than many people expected. AI models can already write code, analyze information, create images, solve difficult problems, and help with scientific research.
But as AI becomes more powerful, an important question is becoming harder to ignore:
Are we building AI faster than we can understand and control it?
Jacob Coxon, a researcher who recently resigned from Anthropic, believes the answer is yes.
Coxon says he spent the last three years doing AI pretraining research at both OpenAI and Anthropic. After leaving Anthropic, he shared a series of posts explaining why he believes the AI industry is taking dangerous risks.
His comments are serious, but they also raise important questions about where AI development is heading.
Who is Jacob Coxon?
Jacob Coxon is an AI researcher who has worked on pretraining, the process used to teach large AI models from huge amounts of data.
He says he has worked at both OpenAI and Anthropic, two of the world's leading AI companies.
After leaving Anthropic, he publicly criticized the direction of the AI industry.
His main concern is simple:
AI companies may be entering a race to build extremely powerful, potentially self-improving AI before society understands how to keep those systems safe.
What does "superintelligence" mean?
Before understanding Coxon's concerns, it helps to understand the term superintelligence.
A superintelligent AI would be a system that performs intellectual tasks far better than humans across many areas.
Imagine an AI that could:
- Write and improve software extremely quickly
- Conduct scientific research
- Discover new technologies
- Analyze huge amounts of information
- Solve complex engineering problems
- Potentially improve its own capabilities
This is very different from today's ordinary AI assistants.
Today's AI systems can be extremely capable, but they still have significant limitations. Superintelligence is a hypothetical future level of capability.
Coxon believes AI development could eventually reach this point much sooner than many people expect.
His biggest concern: the AI race
According to Coxon, the biggest problem is not simply that AI is becoming powerful.
The problem is competition.
AI companies are competing with each other to build better and more capable models.
One company does not want to slow down because it fears another company will continue developing AI and gain an advantage.
This can create a dangerous situation:
"If we don't build it, someone else will."
Coxon argues that this mindset could push companies to take risks they otherwise might avoid.
Instead of asking:
"Are we ready for this technology?"
the industry may increasingly ask:
"How quickly can we build it before our competitors do?"
Why does he think advanced AI could be dangerous?
Coxon argues that future AI systems could become capable of gaining access to computer systems, developing new technologies, influencing people, and acquiring resources.
He describes a future where AI could potentially become much more capable than humans in many important areas.
That does not mean such an outcome is guaranteed.
It is a risk scenario.
The concern is that if an AI system becomes significantly more capable than its creators, humans may not always be able to predict or control what it does.
This is one of the major questions studied by AI safety researchers.
The difficult question: If AI is dangerous, why build it?
This is probably the most interesting part of Coxon's argument.
If researchers genuinely believe advanced AI could be extremely dangerous, why are they continuing to develop it?
Coxon gives different explanations for OpenAI and Anthropic.
He argues that at OpenAI, some people may not fully appreciate the potential consequences.
He describes Anthropic differently. According to him, many people there understand the risks but believe that someone else will continue the race if they stop.
So the thinking becomes:
"We need to build it responsibly because someone else will build it anyway."
This creates a difficult cycle.
Company A worries that Company B will move ahead.
Company B worries that Company A will move ahead.
Both continue developing increasingly powerful AI.
The result could be a race where nobody wants to slow down.
What is the "alignment problem"?
Another important idea behind these concerns is AI alignment.
In simple terms, alignment means making sure an AI system's behavior remains consistent with human goals and safety requirements.
For example, imagine asking a very powerful AI to accomplish a particular objective.
If the system interprets your objective differently from what you intended, it could potentially take actions that technically achieve the goal but create serious problems.
The more capable the system becomes, the more important this problem could become.
The challenge is therefore not simply:
"Can we make AI smarter?"
It is also:
"Can we make AI smarter while remaining confident about how it will behave?"
Coxon argues that the industry may be moving toward extremely capable systems without having enough confidence about how their internal reasoning and behavior work.
What does he mean by "self-improving AI"?
One of the most serious scenarios discussed in AI safety is a system that can contribute to improving its own capabilities.
For example, a highly capable AI might help researchers:
- Design better AI algorithms
- Write training software
- Improve model architecture
- Optimize computing systems
- Develop better versions of itself
If AI becomes extremely effective at AI research, development could potentially accelerate.
This is sometimes called an AI feedback loop.
It is important to emphasize that this does not mean today's AI systems can simply redesign themselves and become superintelligent overnight.
The concern is about what could happen if future systems become capable enough to substantially automate AI research and development.
Coxon wants the industry to slow down
Coxon does not argue that humanity should permanently abandon AI.
Instead, he calls for greater coordination and caution.
He believes major AI companies and governments may need to work together rather than allowing an uncontrolled race between competing organizations.
He even mentions the possibility of temporarily limiting improvements in model capabilities if that becomes necessary to prevent a dangerous global race.
That is an extreme proposal, and it would be very difficult to implement.
But his broader point is easier to understand:
Some technologies may require society to slow down before continuing forward.
Can AI companies coordinate?
This is where the situation becomes complicated.
If every major AI company agrees to slow down, coordination could potentially reduce the pressure to race.
But what happens if one company or country refuses?
The others may fear falling behind.
This is not only an AI problem. Similar challenges have appeared throughout history whenever powerful technologies created competition between countries or organizations.
The difference is that AI development is increasingly international, and software can spread very quickly.
Why researchers are worried about the next few years
Coxon's message to AI researchers is particularly direct.
He asks researchers to think about what the next few years could actually look like.
Instead of thinking about AI as a distant science-fiction problem, he wants people working in AI to consider what happens when increasingly powerful systems are being trained and deployed today.
His concern can be summarized like this:
We should not wait until AI becomes extremely powerful before asking whether we know how to control it.
Safety research needs to happen alongside capability research.
But there is another side to the debate
Coxon's argument represents one side of a much larger debate.
There are also researchers and technology leaders who believe that powerful AI can be developed safely through better testing, alignment research, monitoring, security measures, regulation, and controlled deployment.
Others argue that slowing down too much could also create problems.
AI can potentially help with medicine, scientific discovery, education, cybersecurity, software development, climate research and many other areas.
So the debate is not simply:
AI = good versus AI = bad.
The real question is:
How do we capture the benefits of increasingly powerful AI without taking unnecessary risks?
What should ordinary people take from this?
You do not need to believe that AI will destroy humanity to take these concerns seriously.
There are already practical reasons to care about AI safety.
AI systems can make mistakes.
They can generate misleading information.
They can be abused for fraud, cyberattacks, manipulation and other harmful activities.
And future AI systems may become much more capable than today's models.
The important thing is to avoid two extremes.
One extreme is:
"AI will definitely destroy the world."
The other is:
"AI is just another piece of software, so there is nothing to worry about."
Reality is likely to be more complicated.
The bigger question for the AI industry
The most important part of Coxon's message may not be his prediction about superintelligence.
It is his question about how humanity should make decisions when a technology is advancing extremely quickly.
If AI becomes capable of dramatically changing science, software, business and society, then decisions about its development cannot be treated as ordinary product decisions.
They involve governments, researchers, companies and the public.
And if the technology becomes powerful enough, the consequences of making a mistake could be much larger than simply releasing a buggy application.
Final thoughts
Jacob Coxon's resignation and his criticism of Anthropic and OpenAI are a reminder that there is significant disagreement even among people who work directly on advanced AI.
Some believe rapid progress is necessary.
Others believe the industry is moving too quickly.
Many people are somewhere in the middle: AI development should continue, but safety and governance need to keep pace with capability.
Nobody knows exactly when—or even whether—human-level or superhuman AI will arrive.
But one thing is becoming increasingly clear:
The more powerful AI becomes, the more important it is to think about safety before, not after, that power arrives.
Coxon's message is ultimately a call to pause and ask a difficult question:
Just because we can build increasingly powerful AI, does that mean we are ready to build it?
