New methods for human arm pose modeling

Abstract

Human arm pose modeling attempts to recover the position of each human arm part from images. It is useful to automatically understand human behavior in many applications. The major challenges in performing this task are: the huge solution space of human arm pose for the various actions, and the diversity of the appearance of human arm parts.

Motivated by the challenges, three aspects of human arm pose modeling have been studied: the basic human arm pose modeling of a specific action based on the theory of graphical model, the relationship between human arm pose modeling and actions categorization, and the learned features of human arm parts and joints from unconstrained still images.

Firstly, a basic human arm pose modeling has been proposed. It focuses on the images of a specific action to reduce the uncertainty of arm movement. It employs a Bayesian network to describe the human arm articulated structure and the dependency between arm parts. The parameters of an arm part are simplified to an orientation and a scaling factor to its real length. Some hand-craft features are used for likelihood computation. The prior of walking people is estimated by the CMU motion dataset in a non-parametric manner. It performs effectively on the CAVIAR dataset and a HKU campus dataset with 82.1% and 93.8% of all arm parts modeling at 0.3 PCP threshold respectively. It also is tested on different human sizes and lighting conditions.

Secondly, the relationship between human arm pose modeling and action categorization is studied. It not only categorizes action but also provides the arm pose. Segmentation is used to improve the likelihood computation, and the semi-parametric distribution estimator for prior is used to overcome the limited training data. Dynamic programming is used to speed up pose candidates modeling. Temporal arm poses (TAP) word from the arm pose candidates for an image sequence within a window is constructed as feature for its action categorization. A discriminative classifier is trained with the TAP word to categorize the action type and the related pose candidate is the estimate result. It is evaluated on videos of MHAD with categorization success rate of 91.47% and 95.83% for single and multiple frames respectively, and images from the HumanEva-I dataset with categorization success rate of 96.67%. Its arm pose modeling performance also has improvement for the actions with high dynamics in upper extremities.

Finally, the method is proposed for the still images with unconstrained actions and backgrounds. An energy-based graphical model is employed to describe the relationship between the joints and arm parts. The functions of the potentials of human arm parts and joints are implemented by CNNs and trained jointly to learn their features from the raw images. The multi-scaled images are used to construct the positive and negative training instances. The local rigidity of arm part is incorporated in the training and testing stages to satisfy the arm structure. The result has better performance on FLIC dataset than the state-of-the-art methods using hand-crafted features.