TL;DR: I claim that even if we could perfectly remove a dangerous capability, there are still open problems in the paradigm of reducing AI risk. To do this, I assume we could identify the exact fuzzy set of knowledge that gives rise to some capability (e.g. the ability to do virology might require removing not just virology knowledge, but also some set of biology and chemistry). Then, I assume we can remove it in a way that the model doesn’t exhibit the capability at all in a broad range of tasks, regardless of its other knowledge and what memory and tools it can use via its harness. Is this enough? I think not, for reasons that have interesting analogies in biology.
A model governed by components.
Imagine this, you have a model with a wide range of capabilities except for a couple of dangerous ones which can be switched on and off. Even better, you have total modularity, in the sense that without retraining you can give one version with say the virology capability activated to a group of people and with it deactivated to another.
There is a bit of a problem. A model is an entangled web, where no capability is stored in a collection of circuits you could remove completely or think of turning it on or off.
It turns out there is a method which would allow us to do this as shown in this recent paper from AE Studio and Anthropic. There is a shared core network and a couple of auxiliary subject-specific ones (e.g. for biosecurity, cybersecurity, explosives, and so on). There is also a classifier, so the model can decide what the data it is dealing with is related to.
During training, the core network can help process the text without always learning from it. If we train on some virology data, its corresponding network gets updated at every batch. And, the core also gets updated, though only sometimes. Otherwise, if we train on standard data, the core network gets updated at every batch. And, you guessed it, the subject-specific networks also get updated, though again only sometimes.
This steers what the model learns from subject-related data into the matching subject-specific network. After training, you can remove or add the model’s virology knowledge by simply deactivating or activating its corresponding network’s weights while keeping the core.
How does this compare to parameter and compute-matched models? You’d expect to pay for constraining how and where learning happens. Here is the nice part! In the above paper, for ~1B-parameter models, with the virology network on, performance is similar to a model trained on all of the same data. With it off while doing virology tasks, the performance is roughly similar to a model trained on all but the data the classifier flagged as virology. And with it off while doing non-virology tasks, the performance is again similar to a model trained on all of the same data.
What about training the knowledge back into the models? We’d want to know what happens if someone got access to the models with the virology network switched off and wanted to restore it or switch it on. The model was robust to this! Better said, at least as robust as techniques filtering the virology data before training, and more robust than techniques trying to unlearn that same data from the model.
Suppose this technique (or a related one) scaled perfectly, allowing us to build safe AGI. Training on virology data might be sufficient to give a model the virology capability, but it’s probably not necessary, since other data (e.g. data that’s important for other capabilities) might also help it perform well at virology tasks. In the context of this paper, we never said the model couldn’t do tasks in virology at all. They only measured a form of cross-entropy loss on held-out text since it’s a good proxy for downstream capability. You can imagine that a model like this could use tools to answer questions or even design something akin to a virus using its biology, chemistry, and general knowledge. This is such a big problem that a lot of people seem to assume that if we cracked it we would be done.
However, let’s assume we could solve this. Let’s say we could find the exact fuzzy set of data which contributes to a model having virology capabilities without hurting its biology and chemistry learning capabilities. And let’s say we could also come up with many ways of checking that the capability is gone across a broad variety of tasks, and its other knowledge doesn't help either, regardless of what memory and tools it can use via its harness. The other capabilities wouldn’t seem to help in getting it back either.
We could say this is a model with virology capabilities (or any capability for that matter) removed. Then we would be done? Would we be?
A fly governed by genes.
I keep thinking about this mapping recently. Mechanism aside, organisms are strikingly similar to models. This mapping is probably not structure preserving, but nevertheless seems useful.
A model can be built with a network absorbing general knowledge and its subject-specific ones for corresponding capabilities that can be removed without hurting performance on other tasks much. An organism can have a combination of fundamental genes necessary for operation and a couple of behavior-specific ones which can be removed without changing some aspects of life much. Most genes influence many traits at once just like different subject-specific data can influence different capabilities at once.
Trying to get the exact fuzzy set of genes that produces a behavior is usually a mess. This, of course, is the exact same huge problem we run into with our models. Unlike in AI, where we didn’t have an example for a solution, in genetics scientists found some!
If we look at fruit flies, the so-called fruitless gene is not only sufficient but necessary for courtship. How do we know this? Put a male fly in isolation for a couple of days, then put it next to a female fly. Usual flies? They will follow a specific courtship routine, more so than flies raised in a crowd. They turn toward the female, tap her with one of his legs, vibrate a wing to produce a song, and chase after her. Flies without the proteins from the fruitless gene? Essentially none of this.
This isn’t a story about how other genes compensated for this gene to get the behaviour back. Just like the question I’m interested in is not if other knowledge can compensate for the missing knowledge to get the capability back. The question I’m interested in answering assumes this isn’t possible. I assume we've identified the exact fuzzy set of genes or data for this to not happen.
What happens then? A couple of scientists thought that behavior is given by genes. However, what if it can also be acquired through experience? In a later follow-up experiment they changed the environment that the flies grew up in. Instead of raising the male flies in isolation, they raised them in groups of only females or only males.
After spending enough time around other flies, flies without the fruitless gene showed courtship. This wasn’t imitation since usual flies don’t court flies of the same sex. And this wasn’t the same kind of courtship since the flies without the fruitless gene started going after both male and female flies, and moreover even other species!
We still don’t really know how the behaviour came back through this experience, but at least we have some idea why it can come back differently. It eventually turned out that the fruitless gene has two roles: 1) it makes the behaviour possible; 2) it shapes how the behaviour manifests. Through growth, it establishes female-directed courtship. And, through maturity, it suppresses male-directed courtship.
The reason why I keep thinking about this mapping is that if I go back to the model I assumed has its virology capabilities (or any capability for that matter) removed, we might still not be done.
On the surface, there’s the idea that a capability might come back through experiences that look nothing like that capability. A model with the virology capability removed might not need to see virology data (or any of the fuzzy set of data we used in removal) to get it back again. It could run in a crowd of other models, none of them doing anything that looks like virology, and through some part of that experience we don’t know how to model, arrive at the capability anyway, the way a fly arrives at courtship in a room with no courtship in it. And if the experiences which help get a capability back in deployment don't resemble that capability at all, then we won't know which experiences to try for in evaluations.
However, beneath that, there is the more interesting idea that the same fuzzy set of data we used to remove a capability might have two roles: 1) it’s what makes the capability possible ; 2) it’s what governs how the capability is used. If a model reacquires a dangerous capability, it might be much easier for it to redevelop the core skills required to exhibit that capability than to also redevelop the judgment required to use those skills in safe ways.
When we train, we have the ability to shape, in some regards, how it would behave at inference when it encounters tasks related to that capability. For example, we can reward the model for flagging dangerous steps, and refusing to produce those. Since those judgment gradients update many of the same weights that the capability gradients do, the skill and disposition towards using it come together in the same parameters. Whatever removal we use, it is likely to be the case we will remove both at the same time. And if we wake up in a world in which the capability comes back, we will likely wake up in a world with all of the new ways it manifests.
What comes with a crowd?
Obviously, things don't have to go down this way. It could be that a model with a capability removed in the way I assumed, will, in fact, never acquire it back. The world in which that happens gives us one less problem to worry about in AI safety.
However, I don’t think we can take this for granted. Afterall, the world in which the alternative happens, where the capability emerges someday in a similar way to how courtship did, is worrying enough that it’s worth thinking through in more detail. The skills can arrive ahead of the sense of where to point them. And they can arrive through experiences that look nothing like what we might expect.
I don’t yet know what to do about this. I just know that when we’re trying to check if we successfully removed a capability we’re asking if it’s really gone, but we maybe we should also what are the obvious and non-obvious ways it could come back, and, if that happens, does it come back with some of the guidance it had before.