SUNNet: A novel framework for simultaneous human parsing and pose estimation

Abstract

This paper presents a novel Separation-and-UnioN Network (SUNNet) for simultaneous human parsing and pose estimation. Our SUNNet consists of two stages: a feature separation stage and a feature union stage. In feature separation stage, we leverage a common feature extractor to implicitly encode the correlation between human parsing and pose estimation, meanwhile, two task-specific feature extractors are designed to extract the features for both tasks. By combining the task-specific features and common features with a feature consolidation module in a coarse-to-fine manner, we can get an initial prediction for parsing and pose estimation; In feature union stage, we refine the initial prediction by explicitly leveraging the features from parallel task to predict the kernels’ receptive fields in a convolutional neural network. We further propose to leverage a 3D human body reconstructed from the image to facilitate these tasks, and a novel Gated Feature Fusion (GFF) block is designed to automatically decide whether to use or skip the priors from the reconstructed 3D human body. Extensive experiments demonstrate the effectiveness of our SUNNet model for human body configuration analysis.