Improving deep learning generalization from the perspective of dual-branch capsule network, contrastive ACE, and edge-GNN

Abstract

The quality of representations in a machine learning system has a direct impact on its performance. In the early day of machine learning, much effort has been devoted to designing data transformation and preprocessing pipelines, so that desirable representation from human perspective can be obtained. Recently, owing to the fast development of computation devices and easy access to large datasets, it has become possible for a paradigm to shift from hand-crafted feature extractors to data-driven feature extractors.

Though some representations are known to be effective in specific setups, their performance on a test dataset with even slightly different characteristics than the training dataset might degrade significantly. It is desirable that networks trained on datasets from certain viewpoints, domains or problem sizes will perform equally well on unseen viewpoints, domains and scales. To achieve this generalization ability, prior knowledge needs to be incorporated into the network architecture or regularization to the loss function, so that the neural network would not overly adapt to the training dataset. Successful examples on including prior knowledge to improve generalization ability include capsule networks with translation-equivariance property for generalizing to new viewpoints, graph neural networks for generalizing to new problem scales, matching representations for generalizing across domains, contrastive learning and adversarial fine-tuning for generalizing to different styles. This thesis is in pursuit of this rapidly developing trend, where three research projects with different emphases are presented.

The first work focuses on the emerging capsule networks, in which the learned representations possess an important property referred to as translational equivariance. As capsule networks often encounter difficulty in extracting hierarchical representations from real-world images, a unique dual-branch design is proposed to improve feature extraction capability while maintaining the property of equivariance under linear deformations.

In the second work, we aims at addressing the problem of domain generalization through improving the interpretability and generalization ability of representations. Previous works on domain generalization treat object representation and domain representation as independent information, and just focus on locating common representations of datasets from different domains. However, this may cause significant information loss, and poor performance. Instead we propose to learn the common feature causality from datasets across domains. Experiments on several benchmark datasets demonstrate that the proposed causality based method achieves improvement in classification accuracy.

Finally, we turn our attention to representation learning of graph-structured data mapped from communication networks. By incorporating the graph structure specific to communication topology into deep learning architectures, the neural networks are more expressive with capability of generalizing to inputs of various configurations. The proposed graph neural network is different from previous works in the sense that the variables are defined on edges rather than nodes of the graph. With a novel edge representation update mechanism, the proposed model is more flexible and is adopted to solve a cooperative beamforming problem in wireless communications for the first time.