Promotionalism, orthogonality, and instrumental convergence

Abstract

Suppose there are no in-principle restrictions on the contents of arbitrarily intelligent agents’ goals. According to “instrumental convergence” arguments, potentially scary things follow. I do two things in this paper. First, focusing on the influential version of the instrumental convergence argument due to Nick Bostrom, I explain why such arguments require an account of “promotion”, i.e., an account of what it is to “promote” a goal. Then, I consider whether extant accounts of promotion in the literature—in particular, probabilistic and fit-based views of promotion—can be used to support dangerous instrumental convergence. I argue that neither account of promotion can do the work. The opposite is true: accepting either account of promotion undermines support for instrumental convergence arguments’ existentially worrying conclusions. The conclusion is that we needn’t be scared—at least not because of arguments concerning instrumental convergence.