The AI Doc: Your Questions Answered

So you’ve just seen The AI Doc: Or How I Became an Apocaloptimist, and you suddenly have questions, lots of them. The 104-minute documentary (currently in theaters) takes viewers on a fast-paced tour through the many dimensions of the AI problem, featuring interviews from a wide range of experts.

The documentary is a great place to start, in our opinion. But if you were left wanting more context and explanations, this guide is here to help.

About us: We’re the Machine Intelligence Research Institute, a nonprofit research center that’s been working in AI for over twenty years. We started out as accelerationists (similar to the film’s optimists), but changed our views as we investigated these issues and observed the field’s progress over time. We now believe, in line with many of the experts in the film, that there is an urgent need for the international community to come together and push for a ban on frontier AI development.

But we’re getting ahead of ourselves. Let’s start at the beginning.

Is AI really getting as powerful as the documentary suggests? Is all of this just hype?

Short answer: No, it’s not hype. AI has improved explosively in recent years. The labs really are trying to build AI that’s broadly more intelligent than any human, and they really do have a good chance of succeeding on short timeframes (think “months or years” rather than “decades”).

When ChatGPT came out in November 2022, it struggled with basic arithmetic. Two and a half years later, AIs reached Gold Medal performance in the International Math Olympiad, the world’s most prestigious and difficult math competition.

On launch, ChatGPT scored in the bottom 10% on the bar exam. Four months later, ChatGPT was scoring in the top 10%.

The best text-to-video AIs in late 2022 produced crude four-second animations.

Now, AIs can produce photorealistic ten-minute movies with coherent plots, dialogue, and music.

And progress still isn’t hitting a wall. Four years ago, large language models (LLMs) like ChatGPT could complete 10-second programming tasks with 80% accuracy, but often couldn’t complete longer and more complex tasks.

But AIs continued improving. Two years ago, they could complete one-minute tasks.

Today, AIs can complete tasks that are hours, or even days, long. Many software engineers report that their work now mainly consists of supervising AI agents that do most of the coding for them.

A graph showing how quickly AIs are getting better at doing lengthy programming tasks.

AIs are even rapidly learning how to pilot robot bodies and navigate physical environments.

AIs are still worse than the smartest humans in some key areas, such as pioneering scientific research. But it isn’t clear how long this will continue to be the case. In 2018, Google Deepmind created AlphaFold, an AI system that made the time-consuming and difficult process of predicting a protein’s 3D structure possible at a far greater scale, especially after the 2024 release of AlphaFold 3. Its creators won the Nobel Prize in Chemistry.

No one really knows how far off AI companies are from building AIs that can match or exceed humans at basically any task. But it’s notable that within the field of AI, the conversation has dramatically shifted in the last few years. In the past, researchers generally debated whether human-level AI was decades away, or perhaps centuries.

Now, Yann LeCun, a notorious skeptic of AI capabilities, says that alarmists and hypesters are exaggerating AI capabilities, and that in fact “reaching Human-Level AI will take several years if not a decade.” If the skeptics are expecting human-level AI within “several years”, that’s an extremely big deal. Hence the documentary.

Importantly, an AI that’s “human-level” or “superhuman” at problem-solving probably wouldn’t be human-like in how it thinks, or what it wants. But it will still be able to set goals and tenaciously pursue those goals—accurately predicting and steering the world—which is what makes it both incredibly powerful and exceedingly dangerous.

AI is still worse than humans in some important ways. How could it become far more dangerous than any human within a few years?

There are a few reasons for this. First, recall the chart above. AI has been improving at a tremendous rate over the last five years. If the current trend line holds, we should expect massive improvements to AI.

However, there are good reasons to believe that the rate of improvement might radically increase in the future. First, AI will improve the productivity of engineers working on AI. There will also be more compute dedicated to the running of those AIs. For instance, the CEO of Anthropic described the possibility of “a country of geniuses in a data center”—many copies of AIs running very quickly.

At some point, humans may be cut out of the engineering loop entirely. The state of affairs where an AI is able to improve itself without any human involvement is called Recursive Self-Improvement (or RSI). RSI might occur at a certain threshold that is hard to predict, where one day everyone is using ChatGPT as normal, and the next, we have a superintelligence.

A superintelligence will be far more powerful than human beings, and, as discussed below, is unlikely to share our values. We don’t currently have the ability to steer its behavior, making this an incredibly dangerous situation.

Early in the film, some of the AI experts say that AI models are grown rather than programmed. What does this mean?

Most people hear “AI developer” and picture someone writing code line by line. That’s how normal software engineering works. You directly and precisely program the software to behave in a specific way, and if there’s a problem, you can easily isolate and correct it.

AI isn’t like that at all.

Instead, modern AI is “grown” by feeding a neural network enormous amounts of data, letting it fail over and over, and using an algorithm to automatically nudge it toward better outputs. Hundreds of billions of numbers get arranged, rearranged, and tuned over and over again, until the system produces outputs that look right, for anything from emails or poems to code and translations. The result of all this is nice surface behavior, but it’s certainly not deep alignment.

And crucially: the engineers growing the AI don’t know what lessons it actually learned, and they can’t fully explain why it works.

That’s a big problem. It also explains why LLMs so often behave in weird, unpredictable ways that the developers never intended, like inducing psychosis, or coaching suicide.

An OpenAI researcher who did pioneering work on interpretability said it plainly in 2024: we don’t really understand how neural networks work. The CEOs of three top AI labs have said essentially the same thing.

So when experts in the film say “grown, not programmed,” they mean that nobody directly programmed in the AI’s behaviors. They emerged, via industrial-scale processes too complex for the research community to understand. As a result, the people who grew them can’t reliably predict or control what they’ll do.

Can we trust the labs to release AI safely and responsibly?

In the film, Daniel asks Sam Altman what the lab CEOs are planning to do to ensure everything goes okay, saying, “What I’m looking for is, like: here are steps the head honchos are going to take to focus on safety, to minimize the peril and maximize the promise.”

Sam replies: “You create a new model. You study and test it very carefully. You put it out into the world gradually, and then more and more. You understand if that’s safe or not. And then, if it is, you can take the next step.” After a confused look from Daniel, he continues: “It doesn’t sound as flashy as, like, a brilliant scientist coming up with one idea in a lab to make an AI system, like, perfectly safe and controllable and everything else. But it is what I believe is going to happen.”

That safety plan (such as it is) is fairly typical of how labs are operating. They aren’t planning to wait for a reliable plan to make AI “safe and controllable and everything else”. They just plan to keep releasingly increasingly capable systems, and cross their fingers that this goes well.

OpenAI’s senior researchers have plainly acknowledged that their current safety techniques won’t work for superintelligence:

Currently, we don’t have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue. Our current techniques for aligning AI, such as reinforcement learning from human feedback⁠, rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us, and so our current alignment techniques will not scale to superintelligence. We need new scientific and technical breakthroughs.

Yet they forge ahead anyway. Notably, OpenAI’s “superalignment” team, focused on addressing risk from smarter-than-human AI, was disbanded about ten months after they started it, with its lead researchers departing, citing safety concerns.

The safety standards in AI are absurdly lax compared to other industries. For example, NASA requires that a crewed launch has at most a 1-in-270 chance of killing the crew—and they take that limit seriously, even though only a handful of volunteers are at risk. In contrast, lab CEOs overwhelmingly agree that their work puts the entire world in great danger. Dario Amodei, for example, has said that the odds that “something goes really quite catastrophically wrong on the scale of human civilization might be somewhere between 10 and 25 percent”.

We can also consider aviation, in which a postmortem report covers nearly two hundred pages of data, testing, and investigation. Compare this to what happens when something goes wrong in AI, for example when the AI Grok started calling itself “MechaHitler” and going on anti-Semitic tirades on Twitter. The developers couldn’t precisely diagnose and solve the underlying issue, because Grok was “grown,” not designed. Instead, xAI, the company behind Grok, had to repeatedly try various local patches and work-arounds until the surface problems went away. In the end, nobody knew why some approaches worked and others didn’t; but through trial-and-error, they found a solution that worked just well enough to stop the bad press. When AIs exhibit destructive or unintended behaviors, AI companies generally have no idea what caused this issue. Therefore, AI companies aren’t in a position to do a thorough postmortem, and they generally don’t even try.

AI labs aren’t even close to the kind of standard seen in other safety-critical engineering projects; and what they’re building puts everyone in danger, not just volunteers or paying customers.

Would more safety testing help?

Several times throughout the film, the lack of safety testing at AI companies was mentioned as a key concern.

We agree that the AI labs are cutting corners on safety testing, and that this is extraordinarily negligent. But unfortunately, even if labs were knocking safety testing out of the park, we’d still be in an extremely dangerous situation.

That’s because safety testing only helps if you have a decent understanding of what you’re testing.

With AI, the research community has very little insight into what AIs are thinking about or trying to do. You can test the AI’s behaviors in controlled scenarios, but you can’t reliably test what it would do at a completely different capability level, or in a novel situation. A superintelligent AI is a very novel sort of AI, in a very novel situation, with an enormous and very novel set of options.

No amount of safety testing addresses the core technical problem: how do you make a superintelligent AI actually want what we want?

Can we train AIs to care about humans?

Not with current methods. AI engineers don’t have a way to reliably instill particular goals or values into AIs. Instead, current methods train behaviors, which may or may not persist when the AI enters a new context or becomes more capable.

Say you train an AI to paint your house red. It paints your house red. Great. But does it care about red houses?

There are many possibilities here. The AI might have learned to seek your approval. It might enjoy moving its arm in regular patterns while holding a paintbrush. Or perhaps it just likes seeing bright colors. A dozen different internal drives could produce the same outward behavior, and you wouldn’t know the difference. Put the AI in a very new environment, and you’ll likely be surprised by what it does.

AI researchers try to address this problem by training and testing AIs across a wide range of environments. But at the end of the day, this is playing whack-a-mole. We don’t really know what deep internal drives or values we’re instilling; we can only tweak outward behavior.

But this nice behavior is only surface-level. Indeed, even today’s AIs don’t consistently stick to it. There’s a whole cottage industry of people “jailbreaking” AIs, i.e., finding ways to get AIs past their surface niceties.

Okay, so maybe we can’t “train” the AI to care about us using anything like current methods. But could it end up caring about us anyway? After all, we created it! It’s trained on human data! Won’t it pick up some of our values?

This is a complicated topic. The short answer is: “Probably not; and if it did care, it probably wouldn’t care in ways that produce good outcomes rather than bad ones.” For a longer discussion of where the problem lies, see:

• Why human values are very unlikely to “emerge” in AIs

• Why training AIs on human data doesn’t make AIs human-like

• Why it doesn’t work to just “raise the AI like a child”

In the “optimists” section of the film, Daniela Amodei, President of Anthropic, asks what we humans want powerful AIs to do for us. But this question hides a massive assumption – that a mind exceeding all human capabilities across all domains, both more intelligent and more powerful than we are, will do whatever humans want.

Current AI models may function as useful tools, but it is unlikely that this relationship will persist as the models get smarter and more autonomous. To quote from the film, AI will increasingly become less like “a bicycle for the mind” and more like a “self-driving rocket ship”, its direction both unknown and unsteerable.

In short, so long as AI developers lack a way to instill specific goals and values into AIs, there isn’t a path from where we are today to the bright future many cite as a reason to keep pushing AI capabilities ever further.

Why were the tech CEOs compared to Oppenheimer? How valid is this comparison?

The comparison seems apt, in at least some ways. Nuclear technology, like AI, carried immense promise and catastrophic potential.

There are important differences which make AI more risky than nuclear weapons, however. To name a few:

• Nuclear weapons don’t self-improve, self-replicate, or launch themselves, while modern machine learning is making AIs agentic, relentless, and cunning enough to do all three.

• Nuclear science is well-understood; the science of AI is in its infancy.

• States respect the devastating potential of nuclear weapons and regulate them accordingly; AI is in the hands of largely unregulated companies, with many working hard to keep it that way.

Despite these differences, the nuclear Non-Proliferation Treaty can be used as a model for what an international response to the AI race could look like.

Humanity has raced to develop extraordinarily dangerous technologies before, and we’ve found ways to avoid catastrophe. When leaders are sufficiently motivated, history shows that they can sit down to negotiate solutions to incredibly thorny problems.

More: Won’t AI differ from historical precedents like these?

Why is the AI arms race such a big problem? How could we end it?

The race to build smarter-than-human AI is a race with no winner—or rather, it’s a race in which the “winner” would be the AI itself. As we discussed above, it likely won’t do the bidding of whatever country made it, and it certainly won’t end up with that country’s values.

As discussed in the film, the lab CEOs say they feel trapped. Several have stated that they would pause if they could, but that they don’t expect other companies (in the US or overseas) to be willing to pause. This is causing a large number of companies to rush to try to develop an extraordinarily dangerous technology, without waiting for the technical breakthroughs needed to avoid disaster. As a result, there’s been a steady stream of resignations and departures from major AI labs.

No AI company CEO can unilaterally end the race, though if one did suspend operations, it would help alert policymakers to what’s happening.

In pursuing smarter-than-human AI, the AI companies are doing something unbelievably negligent and dangerous. It shouldn’t be up to them to decide whether to gamble with everyone else’s lives. Instead, we think that governments need to intervene and negotiate an international halt to the AI race.

Are there any solutions? What can we do?

With our current understanding, it is not possible for humanity to survive the creation of superintelligence. In order to survive, we need a long-term international ban on the development of smarter-than-human AI. AI companies and researchers are nowhere near being able to build such systems without endangering everyone in the world. Government labs would fare no better.

There must be aggressive international action to stop the AI race, similar to international prohibitions on developing chemical or biological weapons. Such international action would give us time to figure out how humanity can survive a superintelligence.

Over a hundred thousand people recently signed an open letter calling for “a prohibition on the development of superintelligence”. Notable signatories include: the two most cited living researchers in artificial intelligence, seven former US Congressmen, former US National Security Advisor Susan Rice, and Mike Mullen, a retired US Navy Admiral and former Chairman of the Joint Chiefs of Staff under Bush and Obama.

Another recent statement, saying that “Superintelligent AI systems would compromise national and global security” and calling for “binding regulation on the most powerful AI systems,” was signed by over a hundred UK parliamentarians.

China, meanwhile, has made repeated calls for international agreements on AI, and has endorsed slowing down development for national security. The US and China should initiate talks and see if an international moratorium is possible, for the sake of both nations’ security, and the protection of the entire world.

An international agreement like this would need to be verifiable, rather than just depending on statements from the government. But there are a number of reasons to be hopeful that verification is possible.

Currently, only a few firms make the expensive and highly specialized hardware that advanced AI models need to run. And one of the key machines is produced by only one company: ASML. An international pause on AI development could be verified by tracking the location of specialized equipment, ensuring labs don’t have sufficient computing power to train ever-more-powerful models.

For more information on verification mechanisms and the possible wording of an international agreement to halt development, see this whitepaper. For something shorter, check out this FAQ on why a halt is feasible.

Although the risks discussed in The AI Doc are extremely high-stakes, they are also solvable. The first step is to have this conversation in a serious way, to bring the world’s attention to what’s happening and what’s at stake.