Vision-tactile sensing for robust robotic grasping

Abstract

Robotic grasping is a field that has been approached for many decades. It aims at creating cognitive robots that can operate at the same level of dexterity as humans to explore and manipulate objects. When humans see novel objects, they can flexibly determine the way to grasp them. However, the robots’ capabilities fall behind. Despite the interest in research and industry, it remains a critical challenge in detecting and handling grasping failure in real time and grasping deformable objects. To circumvent this issue, in this thesis, we present two robotic grasping systems for different goals with the combination of tactile and visual information.

We first investigate an effective grasping system from the beginning to the endpoint, by considering the partial object occlusion as a normal condition. We especially focus on the failure detection and recovery framework in the grasping system by combining the specific proprioceptive capability of our soft gripper and the visual cues from the highly obstructed view when the failure occurs. The proprioceptive soft gripper used in this work was developed in our previous work. It was pneumatically driven by soft bellows actuator and the pressure of the actuator was leveraged for sensing the gripper movement and external contact. In the work, we explored more accurate pose estimation of a known object by considering the edge-based cost besides the image-based cost. We utilized robust object tracking techniques which work even when the object is partially occluded in the system and achieve mean overlap precision of up to 80\%. Moreover, we discussed the contact and contact-loss detection between the object and the gripper by analyzing the internal pressure signals of our gripper. We robustly handled the failure in robotic pick and place tasks with the combination of visual cues under partial occlusion and proprioceptive cues from our soft gripper to effectively detect and recover from different accidental grasping failures.

Second, we propose a deep visuo-tactile model for real-time estimation of the liquid inside a deformable container in a proprioceptive way. We fuse two sensory modalities, i.e., the raw visual inputs from the RGB camera and the tactile cues from our specific tactile sensor without any extra sensor calibrations. The robotic system is well controlled and adjusted based on the estimation model in real time. The main contributions and novelties of our work are listed as follows: 1) Explore a proprioceptive way for liquid volume estimation by developing an end-to-end predictive model with multi-modal convolutional networks, which achieve high precision with an error of $\sim2$ ml in the experimental validation. 2) Propose a multi-task learning architecture that combines consider the losses from both classification and regression tasks, and comparatively evaluate the performance of each variant on the collected data and actual robotic platform. 3) Utilize the proprioceptive robotic system to accurately serve and control the requested volume of liquid, which is continuously flowing into a deformable container in real time. 4) Adaptively adjust the grasping plan to achieve more stable grasping and manipulation according to the real-time liquid volume prediction.