DriveLLM: Charting the Path Toward Full Autonomous Driving with Large Language Models

Abstract

Human drivers instinctively reason with commonsense knowledge to predict hazards in unfamiliar scenarios and to understand the intentions of other road users. However, this essential capability is entirely missing from traditional decision-making systems in autonomous driving. In response, this paper presents DriveLLM, a decision-making framework that integrates large language models (LLMs) with existing autonomous driving stacks. This integration allows for commonsense reasoning in decision-making. DriveLLM also features a unique cyber-physical feedback system, allowing it to learn and improve from its mistakes. In real-world case studies, the proposed framework outperforms traditional decision-making methods in complex scenarios, including difficult edge cases. Furthermore, we propose a novel approach that allows the decision-making system to interact with human inputs while guarding against adversarial attacks. Empirical evaluations demonstrate that this framework responds correctly to complex human instructions.