AI safety: a climb to Armageddon?

Abstract

This paper presents an argument that certain AI safety measures, rather thanmitigating existential risk, may instead exacerbate it. Under certain key assumptions -the inevitability of AI failure, the expected correlation between an AI system's power atthe point of failure and the severity of the resulting harm, and the tendency of safetymeasures to enable AI systems to become more powerful before failing - safety effortshave negative expected utility. The paper examines three response strategies:Optimism, Mitigation, and Holism. Each faces challenges stemming from intrinsicfeatures of the AI safety landscape that we term Bottlenecking, the Perfection Barrier,and Equilibrium Fluctuation. The surprising robustness of the argument forces a reexaminationof core assumptions around AI safety and points to several avenues forfurther research.