What is the likelihood of deceptive misalignment?

We can build upon this intuitive example by exploring the technical details in more depth. In trying to analyze which one of these three cases is most likely, we make some assumptions - first that we are working with a relatively complex base objective such as human values, and second that we have diverse enough training environments and different enough situations to get the models to robustly understand what we humans want them to want, i.e. we are concerned with deceptive models, not with dishonest ones.^[2] Lastly, we also assume that all models are roughly made up of three parts: mesa objectives, world models, and an optimization process. The model will always start with a random proxy objective which we are trying to refine into the ‘true’ (base) objective that we want the model to have. The objective of the optimization process (Stochastic Gradient Descent (SGD)) is to improve performance, which can be done in two ways:

• Better Proxies: The model can improve its understanding of the objective

• Better World Models: The model can improve its ability to act on its existing understanding of the objective

Since ML models might result in different outcomes even when trained on the same data (path dependence), we explore if we end up with deceptive alignment in both low and high path dependance training environments. This gives us an estimate of the types of models that we expect SGD to produce.

High Path Dependence (HPD) alignment scenarios

There are three scenarios in the high path dependence case. In the HPD internal alignment scenario, the proxy is perfected to match the base objective before the world model is perfected. This means that the model is now successfully trying to optimize for the base objective. In the HPD corrigible alignment scenario, the world model is perfected before the proxy objective matches the base objective. This means that the world model is perfect enough to simulate SGD. Additionally, the best representation of the base objective is the actual base objective that SGD has. So SGD just ‘throws out’ the proxy objective and replaces it with a pointer to its own objective. In the HPD deceptive alignment scenario, the world model is again perfected before the proxy objective matches the base objective. However, this time SGD makes the current proxy a long-term goal (crystallizes it), which means that the inner optimizer is now free to maximize the base objective. In either of the latter two cases, SGD successfully trains an inner optimizer to optimize for the base objective during training.

Low Path Dependence (LPD) alignment scenarios

In a low path dependence world, the model will just try to pick the simplest algorithm out of all possible algorithms that solve the given problem. We focus on simplicity and compute inductive biases to answer the question - What is the simple, unique solution that you end up with every time you run your training process?

The simplicity inductive bias is an analysis of the algorithm complexity. The world models are exactly the same across all the different modes. The main difference between the models is the mesa objective, i.e. the complexity of the objective that we give the model which it has to learn and then optimize. The simplest internally aligned mesa-objective is the base objective. So for internal alignment, the complexity class is the full complexity of the base objective. The simplest corrigibly aligned mesa-objective is a specification for how to point to the base objective. The simplest deceptively aligned mesa-objective is literally any long term objective. So as far as Pascal is concerned the most complexity cost you have to pay is just the simplest long-term objective you can think of. In other words, the deceptively aligned one has the lowest complexity cost out of the three alignment scenarios, as long as we have a relatively complex training objective. All of the other model classes scale with the complexity of the training objective, whereas the complexity of the deceptively aligned model is essentially fixed. Therefore the more complex the base objective (e.g. human values) the more deceptive alignment wins out in the complexity race. Overall, the analysis results in the conclusion that #(Christs) < #(Luthers) < #(Pascals). So since there are more Pascals than the other two, we should expect deceptive alignment to be the default case as analyzed by the model simplicity inductive bias.

The speed inductive bias thankfully falls in favor of corrigibly and internally aligned models. The deceptively aligned model has to perform extra steps of ‘thinking’. It has to realize that caring about the mesa-objective in the long term means that it should optimize the training objective temporarily to get a higher reward on your internal objective eventually. The internal and corrigibly aligned models however don’t have to do this extra thinking and can just directly optimize what they care for (the base objective). So in terms of time taken for each model class to output the correct answer: t(internally aligned) ~= t(corrigibly aligned) < t(deceptively aligned)

This indicates both algorithmic simplicity and complexity inductive biases point towards deceptively aligned models being the most likely outcome to arise in a low path dependence machine learning training process.

Overall, we can see that gradient descent is by default going to take us toward the deceptively aligned model in both the low and the high path dependence cases. If we accept the above argument then we have to come up with techniques to better monitor models during training, as well as, undertake some intervention that changes the current ML training dynamics.

Alternative Phrasing

• How likely is deception?

• Might an agent be deceptively aligned?