Why would a misaligned superintelligence kill everyone?

While AI is unlikely to be malevolent towards humanity, we might still die as a result of the AI doing instrumental reasoning, with our deaths being either 1) an intentional goal or 2) a side-effect of some other goal:

• As an intentional goal: The AI might see a risk of humans interfering with its goals, e.g., by trying to turn it off or by building a rival superintelligent AI. The AI might attempt to remove this risk by killing us.
• As a side-effect: Whatever a superintelligent AI’s goals are, it’s unlikely that they would be best served by keeping the Earth broadly in its current state, unless the AI specifically values preserving the status quo (e.g., for the sake of humans and other life). Just like we flood large areas to build reservoirs for dams, fundamentally changing the ecosystem, an AI might undertake large-scale projects that make the world unlivable for humans, or that use the materials lying around the solar system as resources. This material could include humans themselves: “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.” But any superintelligence that was both powerful and misaligned enough to consider taking humans apart for their atoms would be modifying the rest of the world radically enough to make human life impossible, anyway.

So the end result could be human extinction.

On the other hand, keeping humans around would take only a small fraction of a superintelligence’s resources. Some have argued that an AI might be willing to pay that cost to keep us around, either if it’s only mostly misaligned and cares about us a little bit, or for various decision-theoretic reasons. Others disagree. And even if it’s true, many of us might still die, and the survivors might not like their situation, and we’d lose out on most of the universe.