Human knowledge for robotic dexterous grasping and motion planning with applications

Abstract

Grasping is an important and fundamental skills of robots. This thesis aims to enhance robotic grasping by integrating human knowledge through cutting- edge AI techniques and a pioneering dataset. We present a novel approach for hand-object interaction reconstruction and propose a framework to generate human-like robotic dexterous grasping. Furthermore, we extend robotic motion planning techniques to orthodontic, providing a motion planning method for creating clear-aligner design.

The task to reconstruct hand-object interactions from a single-view image is ill-posed, demanding massive prior knowledge of the 3D world. In contrast to most of prior works that require multi-view images, videos, or 3D templates as additional inputs, this thesis aims to reconstruct generic hand-held objects and hand poses from a single RGB image. We propose a novel method that unleashes the power of large-scale models for construction of hand-object inter- actions. Our method is capable of reconstructing hand-object interactions with high fidelity, even in challenging scenarios with occlusions. Extensive exper- iments demonstrate the effectiveness of our approach on both public datasets and self-collected data.

The development of five-fingered dexterous hands offering enhanced flex- ibility for robotic grasping. Existing datasets for dexterous grasping are syn- thesized and fail to model human-like behaviors, which limits their ability to generalize. To bridge this gap, we introduce RealDex, a pioneering dataset cap- turing authentic dexterous hand grasping motions infused with human behav- ioral patterns, enriched by multi-view and multimodal visual data. Utilizing a teleoperation system, we seamlessly synchronize human-robot hand poses in real time. This collection of human-like motions is crucial for training dexterous hands to mimic human movements more naturally and precisely. Moreover, we introduce a dexterous grasping motion generation framework, which aligns with human experience and enhances real-world applicability through effec- tively utilizing Multimodal Large Language Models. Extensive experiments have demonstrated the superior performance of our method on RealDex and other open datasets.

Clear alignment treatment offers a more comfortable alternative in orthodon- tic care, attracting considerable attention in the dental community recently. To produce clear aligners, the tooth motion planning of from pre-treatment ar- rangement to the post-treatment target setup is essential. This thesis proposes the first automatic method for generating collision-free teeth motion planning and develops an interactive system to assist orthodontists in designing treat- ment plans. Our experiments and user studies confirm the effectiveness of this method in planning teeth movement.

In conclusion, this thesis explores the effective integration of human knowl- edge to robotic hand grasping and the application of motion planning tech- niques in healthcare. By utilizing foundational models in hand-object recon- struction, we can provide richer geometric information about human grasping, enhancing the training of robotic grasping systems. The collection of a realistic dataset, coupled with a framework to synthesize grasping motions, enhances real-world applicability of robotic dexterous grasping. Finally, we adopt the techniques widely used in robotic motion planning for orthodontic treatment, enabling the automated production of clear aligner design.