Generalization analysis and regularization in over-parameterized models

Abstract

We study the success of over parameterized models in both regression and classification tasks. In the regression task, we uncover the phenomenon of multiple descent in random feature models, where the test accuracy follows a curve with multiple descents as the number of model parameters increases. In the classification task, we theoretically establish the capability of two-layer ReLU convolutional neural networks to learn complex XOR data. We find that these networks can achieve the Bayes optimal test accuracy when the data signal-to-noise ratio (SNR) is high. Through our theoretical investigations, we discover that benign overfitting only occurs when the data set has a high SNR. Models trained on low SNR data consistently exhibit poor test performance, indicating harmful overfitting of the training data set.

We also explore two regularization techniques which can address the issue of harmful overfitting in low SNR data sets for over parameterized models. Firstly, we investigate gradient regularization and its role during the training process. Our theoretical analysis reveals that gradient regularization can effectively suppress the memorization of noise within the model. Consequently, the models with gradient regularization exhibit improved performance in signal learning compared to models without this regularization technique. Secondly, we explore the use of early stopping as a regularization technique. By observing the spectra of weight matrices during the training procedure, researchers identify deviations from the Marchenko-Pastur law. We found that these deviations indicate the presence of sufficient training information or potential issues. As a result, we propose a spectra criterion that can guide the early stopping process during training.

Overall, this thesis highlights our investigations into the success of over parameterized models in various learning tasks. We provide insights into the conditions under which these models perform well, and investigate several regularization techniques which can mitigate the harmful overfitting.