A study of sensor fusion of a depth camera and ultrasonic sensor

Abstract

Depth camera has in the recent decade become a vital tool for three-dimensional reconstruction of object in various fields. While Lambertian surfaces are able to be easily resolved, transparent and reflective surfaces have proved to be difficult for depth camera due to its inherent limitation of using optical technique. In an attempt to remedy this, we proposed a solution of using an additional ultrasonic sensor in conjunction with the depth camera, effectively a sensor fusion method. Since an ultrasonic sensor is not constrained by the transparent surface, it is able to provide the correct depth value. However, one sensor is limited compared to a depth camera, which provides data for the whole scenery, when ultrasound is only able to provide the depth of one singular point. Our practical solution is to use a rail sliding mechanism, and mount the sensor, providing it with one dimensional movement. As such, we can obtain data from the horizontal plane, and use this information together with a selected plane from the depth camera. A simple calibration step can be done to align the two sensors accurately. We then model the fusion method with a modified Bayesian network, which is needed to take into account the possibility of occlusion due to an opaque object residing behind the transparent object. In practice, we are able to recover the location of a transparent object surface, and provide a new depth value for further processing. The width of the transparent object is also obtained, which was not possible before.