Human-machine-environment interaction investigations towards smart places : neural decoding and building energy conservation

Abstract

The latest “Smart Place” vision emerges with the advent of modern intelligent technologies represented by machine learning and artificial intelligence. The mission of a smart place is to enhance the quality of human life in terms of comfort, convenience, efficiency, and so on, without sacrificing the economic and ecological interests. A smart place can be defined as a machine-space complex, which comprises an intelligent machine/AI agent and a physical space (i.e. the “inside environment”) linked to a specific human experience. The intelligence of a smart place is twofold, namely the “internal intelligence” embodied through the interactions between humans and the machine-space complex, and the “external intelligence” embodied through the interaction between the machine-space complex and the “outside environment”. In terms of the interactions between humans and the machine-space complex, the brain-machine interface (BMI) provides a promising path to achieve direct communication between humans and their surroundings. However, deriving useful information from brain activities, especially from the ones evoked by natural stimuli, still remains technically challenging for BMI applications. To this end, two natural-stimuli-oriented neural decoding investigations were conducted: (1) A novel spatiotemporal model based on optical flow estimation and convolutional features was established for decoding dynamic natural vision from brain activity, and significantly better performance was demonstrated over all subjects than the spatial method in tests with 11.5 hours of visual cortex functional magnetic resonance imaging (fMRI) measurements while movie watching. (2) A chord-based neural decoding method was developed with dense networks to directly estimate the chords from whole-brain fMRI mappings while natural music listening, and decent accuracy (>85%) was achieved in the major/minor chord classification task for both musicians and non-musicians. For the interactions between the machine-space complex and the outside environment, energy-related interactions between buildings and the anthropogenetic meteorological impact have been the major concern. Firstly, how the altered meteorological factors such as climate change and urban heat island jointly impact the building energy consumption is not clear. Secondly, high-performance energy forecasting model is needed urgently for effective building energy management. Two investigations were conducted correspondingly: (1) The joint impact of meteorological factors was quantitatively examined and emphasised through multiple whole building energy simulations using 30 years of actual meteorological data from both an urban and a rural site. (2) SEF-GAN, a novel meteorology-based deep generative learning method that converts time series forecasting into an image inpainting problem, was presented for short-term building energy forecasting and achieved state-of-the-art accuracy (CV-RMSE < 13% for one-day-ahead hourly forecasting) with noise robustness and cross-case generalisability under multiple effectiveness and granularity settings. Lastly, the abovementioned studies were fused into an integrated framework for exploring the possible technical trajectories towards smart places through neural decoding and building energy conservation. The prospective direction of utilising neural decoding for advancing building energy conservation was then discussed. Multidisciplinary insights on intelligent technologies towards smart places combined with contributions to domain knowledge of computational neuroscience and building energy science via a human-machine-environment interaction perspective are provided in this doctoral work.