Complex-valued neural network : theory, implementation and applications

Abstract

Complex-valued Neural Network (CVNN), a special branch of neural networks, is a model specifically designed to handle complex-valued data with complex-valued functions. Unfortunately, not much attention has been paid to this model despite the success of deep learning. This might be due to a common belief that a well-developed real-valued neural network (RVNN) could imitate and provide the same function approximation as a CVNN. Additionally, inadequate support of complex-valued operations in popular neural network frameworks also hinders the development of CVNN. Due to this misbelief and implementation barrier, the research community tends to convert complex-valued data into the form of images or spectrograms and abandon the phase information to fit data into an RVNN, or implement a mixed form of neural network where part of the operations are complex-valued while the remainder is based on real-valued functions.

In this thesis, we provide an all-round study about CVNN with a self-designed framework and demonstrate how CVNN outperforms traditional RVNN and its variants in various applications. In particular, many functions that originate from the real-valued domain could not be substituted with complex numbers directly, including sigmoid activation functions and softmax functions. Furthermore, many existing commonly used neural network functions such as dropout and real-imaginary part concatenation would decouple the unity of a complex number. We address this issue by proposing new complex-valued functions and complex-valued layers of CVNN. To preserve the probabilistic representation at the output of classification tasks, we introduce novel complex-to-real (C2R) layers. This makes CVNN workable for both regression and classification tasks.

With a developed truly CVNN, we test the model and compare it with real-valued counterparts in three different tasks: multi-label classification in music transcription, multi-class classification in EEG dataset and a regression task in channel estimation. Experimental results show that CVNN outperforms all its variants and real-valued counterparts. Even a double-sized RVNN model does not outperform CVNN. This thesis represents the first step towards showing CVNN could reach promising results that cannot be achieved by real-valued neural networks.