Omniscience is Not Omnipotence

AI risk and the hard constraints of biology

You don’t have to listen to me about AI risk in general - I dabble, I’m absolutely not an expert - but I’m tired of feeling Gell-man irritation whenever I see AI risk people talking about biology. This isn’t the first time I’ve covered this topic, and I doubt it will be the last. AI is unlikely to be the dominant factor in most plausible bio risk scenarios, and even less likely to make biology one of the primary AI doom pathways.

This follows from some broader principles I consider too often discounted in AI prediction discussions, especially among doomers. Knowing everything there is to know is not the same thing as being capable of doing anything one can imagine. Intelligence, even at extremes, is not equivalent to knowing everything there possibly is to know. You can’t build or direct an agent to become so intelligent as to be functionally omnipotent or omniscient based on intelligence alone.

Bio Risk vs. Other Risks

It’s considerably more plausible to me that AI is a cyber risk than that it could build a bioweapon, especially autonomously. The road blocks to AI writing and deploying catastrophic malware are much lower than those for creating and deploying dangerous viruses, toxins, or bacteria. I’m skeptical that even with 10 years of additional AI development, it would ever make equal or more sense for an AI to pursue bioweapons than it would to pursue native digital weapons. AI’s directed training suggests that the path of least evolutionary resistance is rarely, if ever going to turn an agent toward biology over alternatives, even in a catastrophic unaligned scenario.

Whether we’re speaking of nefarious AI agents or bad human actors, AI can indeed narrow the gap between cyber risk and bio risk scenarios, but that’s a massive friction gap right now. Parts of it are bound by physics that I don’t anticipate AI will be able to undo. I doubt that AI is going to speed up cell division in the next 10 years. Again, intelligence is not knowledge and knowledge is not magic.

Scenario Possibilities

When it comes to bio risk, publicly available arguments are messy, with multiple scenarios often confused for one another and layered on top of each other as potential dangers from current or future AIs. Some will claim the greatest risk is AI agents making bioweapons or super-viruses by themselves. Others, that AI agents will make bioweapons with human help, possibly agent coercing human, possibly human coercing agent. Some seem to believe that AI will empower individuals to make exotic strains of super-tuberculosis in their own basements.

If I were to try to break out the primary scenarios that laypeople seem to consider potential problem scenarios for AI agents and biology, they are:

• a random person could produce a bioweapon given instructions on how to do so (and no lab access),

• a random person could use lab access to produce a bioweapon without meaningful risk of getting caught, or

• an AI could produce a bioweapon with or without human intervention without meaningful risk of getting caught before deploying.

It takes a lot of time, equipment, and resources to do this for any of these actors, all of which introduce significant risk of being apprehended before completion.

Information and Reasoning Are Not Enough

A major problem here is that people have a big black box in their heads where “biological manufacturing, research, and discovery” lives, and they more or less assume that the inside of that box looks a whole lot like the inside of every other factory. Inside are robots and a conveyor belt and a relatively friction-free process that produces The Desired Product at least 90% of the time. This is an absolutely terrible model of what’s required to make these things.

It’s hard enough to make a meaningful amount of biologically effective proteins or cells with all the resources, infrastructure, and education in the world. Do you think scientists are just hiding functional off-the-shelf CAR-T cell therapies that can perfectly cure cancer for fun, or something? Biology is genuinely very hard. It’s so much harder without the lab, the resources, and the education that usually accompany it. Furthermore, AI is not an education; it doesn’t know how to tell you things you don’t know how to ask, and there are a lot of answers in biology you would never think to ask questions about.

Many bottlenecks in biology are physical rather than informational, tied to resource, equipment, and time constraints and not education. Biological systems are intrinsically noisy, fragile, and context-dependent. Anyone trying to produce proteins, viruses, toxins, etc., will have to get materials, or find a way to derive them themselves. Retailers for scientific reagents are not like Home Depot, they often want to know which lab you’re attached to, which grant you’re using to purchase things. They’re also expensive. If you want an antibody attached to a bead to do a single form of analysis on your product, it alone will set you back over a hundred dollars for around 100 micrograms. An economics question: why does the market for antibodies clear at 100s of dollars per sub milligram if these things are easy and cheap to make, requiring minimal expensive reagents, time, and equipment?

Every step of the process for creating proteins, culturing cells, and more, introduces a point where outside help is needed: reagents, equipment, etc. Every one of these steps, if undertaken by someone who didn’t want people to know what they were doing, could attract attention or get them caught. There are a lot of steps in these processes even supposing that they’re perfectly carried out. In reality, multiple steps have a failure rate above 10%. That - lots of steps, meaningful failure rate - compounds very quickly in time, costs, and risk of exposure.

AI, Alpha Fold, And Genome Design

The primary step people seem to have some familiarity with is protein and DNA design. “AI can design proteins!” and “AI can model dangerous viral genomes,” are both common concerns from educated laypeople regarding biology and AI risk.

To begin with, AI is not nearly so good at modeling these things as you might have been told. Claus Wilke is a professor at UT Austin, working in statistics and integrative biology. Here’s some of his writing on LLMs for biology:

AlphaFold is pretty good. But, in practice, what happens more often than not, is not what I would call the AlphaFold experience.

This is my usual experience: I read about some new computational method that appears to work exceptionally well, I get excited because it’s exactly what I need for one of my projects, I try the method, and results are disappointing. Most of the time things don’t work, or at least not as well as expected. I have seen this play out so many times my default assumption is nothing is going to work. And anything that does actually work is a bit of a miracle. The only successful strategy is volume. If you’re trying sufficiently many things, some do in fact work, and those you can publish.

But even after we’ve accounted for the fact that AI is actually quite a bit worse at designing biological substrate than you may have been told, another frequently ignored reality at this time: AI can model DNA and RNA and proteins. It can’t make them. Making DNA sequences requires synthesis and assembly, expensive reagents and time. It also doesn’t always work the way you imagine - you realistically need to check your construct with some form of sequencing, which gets more expensive the longer your sequence is. Small RNA constructs can be made in the lab right now, but RNA is very fragile, subject to degradation by a large number of environmental agents. Our existing bench science doesn’t allow us to trivially make large quantities of accurate DNA sequences, and it would be extremely difficult if not impossible to do so with really lengthy RNA sequences, due to their speed of degradation.

Suppose you want to make a protein, instead of a viral genetic sequence? More annoying, with far more steps.

Supposing we could do that already, however: I could pour you a glass full of the genetic material of the most dangerous viruses known to man suspended in buffered water and have you drink it and more than 99% of the time, it wouldn’t do anything to you at all. I don’t think that people understand this. DNA is fragile when it’s just out there helixing aimlessly in buffer. RNA is even more fragile. Your saliva alone is armed with enzymes that can make quick work of these things, and that’s before we get to the rest of your digestive system. We can go further: I could inject you with that DNA or RNA and it wouldn’t hurt you in the vast majority of cases. Your cells don’t like unadorned DNA or RNA anymore than those constructs like to exist unprotected; cells have a bunch of pathways designed to rip them apart because either one in the bulk of the cell signals something has gone wrong.

Basement Wet Lab Scenario

So let’s imagine what it would actually take to produce a viral product at home, with no special equipment. Suppose you somehow successfully make a viral genome using, what, your at home PCR machine? That you also got for very cheap or made yourself? You need to actually get it into a format that will allow it to exist in the human body. You could make lipid nanoparticles and use your Kitchenaid to blend them together. But if you want it to work, you’re probably not going to get away with not culturing cells in some fashion. You need to order cells from a bank, or you need to figure out how to take cells in your environment and turn them into cells that grow on their own in a dish.

I’ll grant that you could probably build a makeshift BSL2 hood without that much difficulty. If you could build your own -80 deep freeze, you are going to need some weird crap and awareness of how to build two refrigeration cycles with different refrigerants. Purchasing those that might raise some eyebrows and bring some inconvenient attention your way. Otherwise that’s going to set you back a cool ten thousand dollars or so and again, more scrutiny from vendors and neighbors alike.

You’d also need to build, repurpose, or buy an incubator in which to keep your cells, one that can maintain not only temperature but oxygen and carbon dioxide ratios within a very narrow bandwidth. It’s also going to be ideal if the doors seal particularly well, to reduce your odds of contamination. DNA is fragile under the best of circumstances, as are cell cultures. This equipment is specialized, it’s not expensive for no reason, and it’s not easy to produce yourself or to do any kind of biological work without.

Existing Lab Access Scenario

Let’s make it harder and say you already work in a lab. You can’t just use the lab’s money to buy things without explaining it to anyone, at least not in a bare minimum functional lab. You can’t just use lab equipment for any random task, especially not for months at a time, in part because if your lab isn’t literally already in the business of making functional viral particles or proteins or specific dangerous bacteria, you will need to order weird things that you shouldn’t need to do your job! People will notice, and they will ask questions, because these materials are extremely expensive, for the most part.

But even supposing you do that correctly, you then have to grow up your cells, make your DNA, transfect the cells with the DNA, expand them, test them for efficacy, all without suffering a catastrophic contamination event that wipes out all of your progress, and get all of these steps and iterations and trouble shooting done over a time scale and with oversight that is designed to catch much more mundane forms of rebellion than this. AI doesn’t really make avoiding detection so much easier as to remove this as a major limitation.

Notice something else: the sorts of people who are in any way positioned to be a risk based on lab access are rare to begin with, and they’re embedded in systems where they’re unlikely to be able to get away with it. Furthermore, people who have access to a lab generally could already try to do this if they felt like it, with or without AI. AI is unlikely to be the dominant bottleneck here, because information and training are not the bottleneck.

Consider, for example, this scenario presented this week by Noah Smith, a typically very sharp commenter on economics, geopolitics, and tech:

So Eric gets a jailbroken version of Claude Code, and tells it to design a version of Covid that’s very lethal and has a long incubation period (so that it spreads far and wide before attacking). He tells his jailbroken Claude Code agent to find a lab to make him that virus and mail him a sample of it.

Now Eric, the angry teenager, has an actual supervirus in his bedroom, with the capability to kill far more people than any nuclear weapon could.

Ideally, having gotten this far, you already know what I think is implausible about this scenario. To begin with, civilians don’t just call up labs like some kind of make-your-own-virus delivery service. Secondly, if they could, it wouldn’t be cheap, and it wouldn’t be fast. Recall that a single traditionally sourced antibody costs over $100 for less than one milliliter of substance. Even if you could find a lab to make you a particular virus modeled by current AI agents without asking questions, it would cost quite a bit more than even a reasonably wealthy teenager could afford to have this done, due to the man hours, reagents, and specialization required. If you assume that the lab knows/is capable of figuring out that this is a mega virus, you’re talking about even more specialized equipment to contain risk for those who work there - BSL3 hoods, expanded protocols, masks, and more.

Supposing that Eric manages to scrape together both the lab that would take this order and the unlikely amount of cash necessary to buy this product, he’ll still have to wait at minimum a month or two for this to come together, and possibly longer. Creating a novel virus involves the steps we’ve described so far - sequence creation, verification, cell culture - and some additional steps we haven’t discussed, all of which take time, risk considerable failure, and require verification. Teenagers who want to destroy the world are famously patient. Once it arrives, I don’t know how Eric is storing that supervirus in his bedroom without liquid nitrogen tanks or a deep freeze, neither of which tend to be particularly low profile purchases or easy to keep around without attracting attention. Eric won’t know what he doesn’t know when it comes to storage, thawing, or deployment, and he won’t know to ask. Even if he does, competence barriers remain: is Eric going to successfully thaw this virus the first time around without killing so much of it that it becomes functionally useless?

But most of all, if we assume anyone can just call up labs and order viruses according to their own specifications, Eric the angry teenager can already do that without AI. AI did not make that kind of lab spring into existence, and it will only marginally reduce the barriers to finding one and securing its services, if it exists. In short, Noah Smith’s fear scenario assumes a lot about lab work, viruses, and AI capability that have some meaningful holes. And his is one of the better ones I’ve heard!

AI Laboratory Scenario

What about the AI run lab idea?

You need one of two things for an AI run lab to exist at all. 1) Humans who take orders from an AI. If the AI is going to get away with something humans don’t want it to, these humans must either not have oversight and never realize that there’s no oversight, or they must themselves avoid ever getting caught by other humans. This version of the scenario requires an extremely implausible expectation of human competence and secrecy. It’s not literally impossible, but it’s not likely, either, for all the reasons we’ve discussed. There are too many resources, equipment, and time involved to make it probable to pull off end-to-end success with humans in the mix without getting caught.

2) The AI does everything with robotics. The reagent and equipment problems are still major issues here, as are the failure steps. Removing human intervention from biological work flows will not make necessary iteration disappear. Zooming out, however, the oversight question applies here too: AI can do a lot of things, but I would be very impressed if it were able to construct or take over an entire plant without human support or awareness, and then conduct experiments in that plant also without human intervention or awareness. These things require so much time and resources that it’s hard to do anything you’re not supposed to be doing without someone realizing it. Biology has physical constraints, not just knowledge ones, and AI, either alone or with human help, won’t make those dissolve no matter how smart it is.

A Large Product of Numbers Less Than One

Both the human and robotics lab scenario, like the rogue human scenario, require accepting a lengthy product of improbable events multiplied together. It’s not like you truly cannot imagine any of these entities doing all of this without getting caught. It’s just that once you start suggesting probabilities for every single step of this process -resource acquisition, equipment acquisition, culturing, transfection, expansion, and so on - and multiply all of that by the probability of doing it for the amount of time it actually takes to do every one of these tasks, including iteration and reorders, all without getting caught, you’re looking at a very low probability indeed. Biology is not an easy problem that falls short of its goals for lack of compute. The thinking is really not the hard part, even if extra thinking would help on the margins.

Doom Scenarios

For the absolute worst case scenarios - the case in which we imagine that AI on its own decides to harm humanity and/or cause major destruction or death - I am most skeptical of all about biology as the chosen vector. Humans, being irrational, love the drama of a pandemic. An AI, at least if left to its own devices, has a different rationality, and is less likely to bother with something so messy, inefficient, and failure-prone as biology when there are far easier and cheaper and faster ways to kill people and sow panic or do both at the same time. A human, being failure-prone and slow, is not likely to be able to do it solely because AI access made it possible, not without high risk of getting caught and low risk of producing something dangerous to large numbers of people.

It’s Not AI That Makes Biology Scary

I think biology has the lowest probability of true AI Doom approaches of any of the ones I’m aware of, and I think there’s some distance between it and the next least likely. For cases outside of doom, I think AI risk from biology is entirely prudent to build guard rails around (and/or to strengthen the ones that already exist), but hardly reasonable as a top of mind threat posed by the technology. It’s technically possible that a rogue dude could do some damage making a virus or toxin in his basement. It’s just also theoretically possible that a dude could acquire uranium, build a centrifuge for enrichment, and then successfully build and deploy a nuclear bomb. You’re assuming that a lot of nontrivial resource acquisition - much of it highly controlled - followed by production quality that’s difficult to attain without formal equipment are going to be surmountable problems. In reality, the size of this risk is incredibly small. Worth making some rules, regulations, and modeling scenarios for, sure. But not the most likely negative outcome from AI.

This is in part because AI is not at all likely to be the key difference in any biological scenario that poses a risk to other people. Lone wolves have already existed, and we’ve built safeguards to prevent, say, another anthrax incident. AI may increase the number of people who know how to do this on the margin, but it on its own does not erode these guard rails or make the actual biological work easier. Another risk scenario that is concerning involves a country intentionally using its own facilities to create and deploy a bioweapon. This too is not likely to be prevented by changing AI: it’s already possible right now. This is also true of anyone with lab access writ large: AI can make it easier in multiple ways, and that will matter to some degree, but not enough to majorly overhaul how we think about risk of bioweapons right now. Most of the biologically related risk scenarios people talk about with AI that have some plausible likelihood of happening are about equally plausible right now, and were about equally plausible five years ago.

What AI Does Change

From where I’m sitting, there are some things AI could make easier that have relevance to biological risk factors. It can help troubleshoot experiments, explain protocols, and lower some informational barriers to entry and procedures. Anyone who works in biology or has should know that being given a protocol, reagents, and equipment is the beginning of a sometimes very long, error-prone process, which sometimes doesn’t work out at all. The people making these arguments seem to think the protocol is all anyone needs to make smallpox from scratch. I can’t tell you how many graduate students and scientists wish that this was true.

Maybe there are better arguments than these, people in secret rooms discussing more pressing and likely scenarios than these. I’m writing this post because all the AI bio risk arguments I’ve ever seen seem to come from people who don’t.

I suspect that AI bio-related risk is meaningfully less likely than the next most likely form of harm from AI. While resources should be expended on preventing AI bio risk, I think returns will mostly come from keeping existing institutional barriers in place. I’m glad that AI companies themselves are thinking about and planning ahead for biotechnology related risk scenarios, but I’m extremely skeptical of lay people who consider AI bio risk to be one of the most serious risks posed by AI developments specifically. Fundamentally, biology is an experimental problem, and that means resources, equipment, and time. AI makes experiments slightly easier to do and troubleshoot. It doesn’t change the complexity of cellular life.