Next Generation Digital Camera Imaging: Human Centered Image Reconstruction, Enhancement and Evaluation

Professor Yu, Yizhou   (Co-Principal Investigator (Co-PI) (for projects led by other university))

54

2019-06-30

714285

Next Generation Digital Camera Imaging: Human Centered Image Reconstruction, Enhancement and Evaluation

Camera imaging pipeline, Image reconstruction, Image enhancement, Image quality evaluation, Deep learning

Others - Computing Science and Information TechnologySignal and Image Processing

R5001-18

Research Impact Fund (RIF) 2018/19

2018

On-going

(1) To develop a unified end-to-end pipeline for in-camera image reconstructionThe traditional camera imaging pipeline includes a set of cascaded in-camera modules to reconstructa displayable image from the sensor raw data. These modules, however, are usually designedseparately and in a hand-crafted manner, resulting in low quality outputs such as noisy images, limiteddynamic range and low contrast. By leveraging the recently rapidly developed machine learningtechniques, we propose to learn a unified compact model of camera imaging pipeline in a data-drivenand end-to-end manner. The new pipeline model is expected to faithfully reproduce the originalscene, even under bad lighting conditions, with high image quality concerting human visualperception.(2) To develop intelligent deep image enhancement techniquesWhile the camera imaging pipeline focuses on the faithful reconstruction of scene content, imageenhancement aims to further promote human perceptual attributes of an image such as resolution,sharpness, color, and aesthetics. Traditional image enhancement methods are mostly manuallydesigned operators, and they need many human interactions to output a perceptually acceptable result.We will investigate effective, lightweight, and human perception-friendly deep neural networkmodels as intelligent image enhancers. By generating a sufficient amount of human annotated data,semi-supervised and re-enforcement learning algorithms will be designed to exploit both humanhigh-level perception information and natural image data distribution information for model training.The learned deep models are expected to produce human favorable photographs with as less userinteraction as possible.(3) To develop human-centered indices for image quality/aesthetic evaluationHow to evaluate the perceptual quality and aesthetics of an image is of great concern to both theimaging system manufactures and end-users. Existing industrial measures of image quality do notdirectly consider common users’ subjective perception of images, which may lead to unsatisfactoryuser experience when taking photos. In this project, we will build new image quality/aestheticdatasets in a cost-effective way, and develop robust learning methods to train practical humancentered image quality/aesthetic models. The developed models will not only be used to predict thequality/aesthetic levels of captured or enhanced images, but also be used to guide the learning ofhuman-centered imaging pipeline and image enhancement.(4) To develop a prototype next generation imaging systemBy integrating the developed human-centered image reconstruction, enhancement and evaluationtechniques, we will deliver an embedded prototype imaging system. With the same sensor andimaging conditions, the developed prototype imaging system will deliver visually more favorablephotos than existing commercial systems.