Development of generative adversarial tri-model (GAT) method for multi-modal sensing

Abstract

Integrating scientific knowledge with machine learning is becoming crucial for solving complex scientific and engineering problems. However, there still lacks a generic framework. This thesis proposes a generative adversarial tri-model (GAT) method that fuses a machine learning model, analytical model, and logic model for knowledge representation. The realization of the GAT method is illustrated using two types of neural networks. The application of the GAT method is demonstrated by solving ordinary differential equations (ODEs) with various definite conditions. The theoretical error bound is proven for initial-value problems. This thesis also presents the development of two sensors, the balance sensor and skin sensor, both based on the principle of frustrated total internal reflection (FTIR). The structure and operating method of both sensors are described in detail. The balance sensor can measure the high-resolution pressure distribution variation under human feet with high frame rate while they are standing still, from which, human balance ability can be evaluated. Preliminary experiment result shows that by applying deep learning method, the balance sensor is able to recognize different human motion patterns. Experiment is conducted to quantitatively calibrate the relationship between pressure and pixel intensity and a physical model is established for this relationship. The skin sensor can realize multi-modal sensing of optical and mechanical skin properties. Experiment is conducted to calibrate the relationship between contact force and FOV (field of view) size. A physical model is established for skin elasticity measurement purpose. Algorithms to extract optical and mechanical skin properties from skin sensor data are developed, which are simple, efficient, and moreover, self-decoupling for different sensing modes. Human experiments are conducted to verify the comprehensive skin evaluation ability of the skin sensor and show positive results. The GAT method also shows great superiority in the calibration work of the balance sensor and skin sensor. Through the GAT method, qualitative physical knowledge can be used to facilitate network training so that the training data demand can be significantly reduced. This advantage is demonstrated by the calibration process of the balance sensor. The calibration results using the GAT method are far more accurate and faster than other neural network initialization methods, especially for a small experiment dataset. Furthermore, training results can be transferred to the calibration of other types of sensors, which is demonstrated by the transfer attempt from the balance sensor calibration to that of the skin sensor. This transfer ability can further improve the performance of the already powerful GAT method. Specifically, the GAT method is able to solve nonlinear ODEs with an integral-value condition. This ability enables it to solve human body oscillation during standing using the balance sensor measurements for human balancing evaluation, which has been validated in experiments. Further experiments demonstrate that the combination of the balance sensor and GAT method can successfully detect the deterioration of human balance ability under the effect of alcohol. The GAT method provides a new framework for knowledge representation that combines machine learning and analytical models, which shows encouraging success and broad potential.