Isn't the real concern with AI that it's biased?

Unfortunately, there are several real concerns. Bias and discrimination^[1] in current and future AI systems is one of a number of real issues that deserve serious consideration.

Bias in AI refers to systematic errors and distortions in the data and algorithms used to train AI systems that contribute to discrimination towards certain people or groups. Note that this use of the term bias is different from the one used in statistics, which refers to a failure to correctly represent reality but not necessarily in a way that particularly affects certain groups.

The forms of bias in AIs today that are most discussed by the media are the ones that lead to racism^{[2] [3] [4] [5]}and sexism^{[6] [7]}. Other biases^{[8] [9] [10] [11] [12]} have also been identified. These biases in AI are often a reflection of which societies are most heavily represented in the training data (such as English-speaking communities) as well as the biases within these societies.

Work to reduce existential risk is sometimes presented as opposed to work addressing bias in current systems, but the AI safety community’s focus on existential risk doesn’t mean it’s unsympathetic to concerns about bias. Yoshua Bengio, who has worked on ethics for many years, rhetorically asks: “should we ignore future sea level rises from climate change because climate change is already causing droughts?”. Humanity can address both classes of problems if it decides to prioritize them both. Furthermore, some research areas such as interpretability are useful toward both goals. On the governance side, there is some overlap in the techniques and institutions to make AI fair and to make AI safe.

That being said, we choose to concentrate on existential risk because we perceive the dangers of superintelligence to be both imminent and of the greatest importance.