Tensor decompositions : from optimization to Bayesian model

Abstract

Tensor decompositions are widely used to tackle high-order tensor data in signal processing and machine learning. Tucker decomposition and canonical polyadic decomposition (CPD) are two most commonly used tensor decomposition schemes. As they are based on low-rank assumptions, determining appropriate tensor rank is an important task in data fitting. For Tucker decomposition, existing optimization methods minimize the trace norms of various unfoldings of the tensor data, which unfortunately do not directly minimize the tensor multilinear rank, and this leads to overfitting of noise in tensor data. On the other hand, while existing Bayesian Tucker methods could learn the multilinear rank automatically, they model the core tensor elements as independently and identically distributed and with the same variance, thus restricting the modeling capabilities in real-world data analytic tasks. To overcome the shortcomings of existing Tucker modelings, this thesis unveils an equivalent form of Tucker decomposition as a CPD with low-rank factor matrices. Based on this newly established relationship, we develop a new trace norm problem formulation such that direct minimization on the multilinear rank could be achieved. To avoid hyperparameter tuning, a Bayesian method is further developed and it gives a more flexible core tensor than existing methods. Both the proposed Tucker algorithms exhibit significantly improved performance in terms of multilinear rank learning and tensor signal recovery accuracy. On the other front, vanilla CPD has been relatively well-studied both from optimization and Bayesian perspectives. However, if nonnegativity is added as constraints on the factor matrices, only optimization based method exist. To extend Bayesian method to this regime, we develop a new sparsity enhancing prior distribution that restricts the values to be nonnegative. Then, a probabilistic nonnegative CPD model and the corresponding inference algorithms are developed. For the first time, this gives an algorithm to learn the nonnegative CPD factors from the tensor data, along with an integrated feature of automatic rank determination.