Hand detection and hand shape and posture analysis in images

Abstract

Hand is an important part of human in communicating with other persons and interacting with objects in the world. In this thesis, we will explore hand analysis, which includes hand detection, hand posture and hand shape analysis.

First I will focus on hand detection. We start by exploring pixel-level hand detection. It is a challenging problem because the appearance of hands can vary greatly in images. We use a random forest framework in order to predict hand mask and hand part labels in an efficient and robust manner. Our approach exploits the inherent contextual information from structured hand labelling for pixel-level hand detection and hand part labelling. Having obtained a pixel-level hand probability map, we present an end-to-end system to detect hands in a bounding box fashion. Our model can not only detect and localize the hands but also provide detailed information within the bounding box by using a multi-task convolutional neural network. Experimental results show that hand detection can be improved by utilizing more information such as hand shape and hand landmark points than only using bounding boxes.

In the second part of my thesis, we will concentrate on two specific problems based on the good hand detection results. The first problem is to recognize the hand postures efficiently. We present a flexible method for hand posture recognition by fusing information from color and depth images using kernel descriptors. Existing methods usually focus on designing intuitive features for color and depth images. On the contrary, our method extracts common patch-level features, and combines them by means of kernel descriptors. Linear SVM is used to predict the class label efficiently. The second problem is related to hand shape. We present an interesting application of shape analysis to train a hand shape recognizer automatically from simple sketches, such as a “stick-figure” of a hand shape. We introduce the Hand Boltzmann Machine, a generative model built upon unsupervised learning, to represent the hand shape space of a binary image, and formulate the user provided sketches as an initial guidance for sampling to generate realistic hand shape samples. A hand shape recognizer is trained from these samples and used to classify the hand shapes intuitively.