Synergy of dynamic perception and static vision in neuromorphic imaging

Abstract

The human retina, stimulated by light, transmits electrical impulses to the brain in which image formation and message acquisition occur. Neuromorphic imaging is a bio-inspired solution that revolutionizes the way visual scenes are captured — by asynchronous streaming events instead of synchronous full images. Compared with frame imaging, it proudly enjoys several remarkable features, such as microsecond-level temporal resolution that enables blur-free recordings of fleeting motion moments, a high dynamic range where there is a clearer observation under dazzling sunlight or in dark midnight, along with minimal latency and low power for seamless integration into portable devices of restrictive computing resources. Nevertheless, this modality has limitations, and the resulting event stream is also a challenge for humans and machines that are accustomed to comprehending static images. We thus have a strong motivation to prompt the synergistic interaction between the two information sources. In this thesis, we bridge dynamic perception with static vision via the proposed approaches to leverage the strength of each, such that elevating imaging quality and paving the way for intelligent neuromorphic systems.

The sensitivity of neuromorphic imaging comes at the cost of susceptibility to interference, leading to informative events triggered along with a storm of noise, which compromises the accuracy of subsequent evaluations. We suggest a blind denoising approach that couples event priors with density statistics for noise removal in a sub-quadratic fashion. The refined output greatly enhances downstream reasoning tasks.

In privacy-sensitive scenarios, the use of neuromorphic imaging may raise security concerns. We introduce an event encryption method based upon event denoising, where events are inversely filled with noise until being obfuscated. The encrypted ones can thwart attacks that harness intensity reconstruction and high-level vision analysis, endowing systems with more robust privacy-preserving capabilities.

Neuromorphic imaging fails to capture low-frequency signal, which is inferior in blurry images due to frame imaging with a low frame rate. A unifying framework, which reaches a maximal exploitation of the complementary nature of the two imaging results, is proposed to jointly reconstruct blur-free images and noise-free events in parallel. Evaluated on a rich range of real samples, this solution brings a convincing quality improvement.

Also, it has limited spatial resolution and fails to deliver richness of visual clarity. We present a self-supervised neuromorphic super-resolution prototype that is adaptive to per input without lengthy training on side knowledge, showing a higher level of practicality and flexibility in the present situation where high-resolution devices remain imperfect and expensive.

Lastly, we rethink event-based representations that are seamlessly compatible with learning platforms, and move forward to event graphs beyond images. With a compact memory footprint, such a synchronous graph expression enables fragmented inference on limited events for efficient recognition, which is more friendly to edge computing and mobile applications where computational resources are critical.

In closing, we provide a summary of our research endeavors and outline a prospect for the synergy of neuromorphic imaging and machine intelligence.