Generalization in autonomous driving systems

Abstract

Deep Learning (DL)-based autonomous driving (AD) develops rapidly. However, AD models generally face poor generalization. Specifically, on the one hand, AD models trained from data in a particular geographical region perform poorly when applied in other regions due to data domain-shift; on the other hand, AD models generally underperform in adverse weather conditions owing to invisibility or lack of structural information. To tackle the geographically related generalization problem, Hierarchical Federated Learning (HFL) offers a potential solution by collaborative privacy-preserving training over distributed datasets from different regions. Unfortunately, existing HFL suffers from three major challenges: (I) slow convergence because data from different cities are with disparate statistical properties; (II) inefficient use of communication resource owing to HFL's traditional static resource scheduling policy; (III) disregard of limited communication resource in practice. This thesis will introduce strategies to solve these challenges. In addition, to improve the AD model generalization in diverse weather conditions, a generalizable scheme without relying on clear reference images is proposed.

To solve the HFL challenges, this thesis firstly proposes FedGau to accelerate convergence. FedGau models the statistical distribution of both single RGB image and RGB datasets as Gaussian distributions, allowing for aggregation weights to be designed based on data statistical properties rather than purely based on volume. Alongside FedGau, this thesis introduces AdapRS, which dynamically adjusts the number of model exchanges based on real-time performance metrics of the global model. Comprehensive experiments show that FedGau accelerates the HFL model convergence and AdapRS significantly reduces unnecessary communication overhead. On the other hand, to address the communication resource constraint in HFL, this thesis proposes CRCHFL framework. CRCHFL prioritizes different stages of the HFL process to minimize generalization errors. Extensive experiments demonstrate CRCHFL outperforms existing FL methods under strict communication resource constraints.

After above HFL challenges are tackled, AD models can enhance generalization via training on large-scale and diverse data from various regions. As time progresses, the data involved in the federated AD system expands. Such ever-expanding data may eventually result in the model under-fitting, leading to poor generalization. To handle this under-fitting limitation, this thesis explores the potential of Large Vision Models (LVMs) to rectify the problem. Precisely, this thesis proposes a pFedLVM framework, which incorporates LVMs into federated AD systems, and exchanges latent features instead of entire LVM parameters between the central server and participating vehicles. This significantly reduces communication demands while allowing for personalized model updates tailored to local driving conditions.

Finally, this thesis improves AD model generalization in adverse weather conditions through a novel strategy termed AdvImmu. This approach, which integrates techniques such as Locally Sequential Mechanism (LSM), Globally Shuffled Mechanism (GSM), and Unfolded Regularizers (URs), enhances model robustness across various weather conditions without relying on clear weather references. Additionally, to address the challenge of lacking frame-wise annotations in adverse conditions, AdvImmu incorporates the foundation model SAM with a clustering algorithm SBICAC to generate pseudo-labels, thereby enabling effective training in complex scenarios.