Learning vision-language representation for multimodal understanding

Abstract

Humans comprehend and interact with their surroundings through the integration of multi-sensory information, including visual, linguistic, and auditory cues. The field of vision-language representation learning is dedicated to enabling machines to learn multimodal associations and interactions between visual and textual data. This thesis tackles three pivotal problems: scalability of the pretraining data, efficiency of the pretraining objectives and fine-grained vision-language alignments.

Regarding data scalability, we focus on scalable vision-language representation learning that leverages unpaired images and texts. To enhance the implicit alignments between modalities and augment data diversity, we introduce cross-modal cutmix, a technique for blending visual patches with sentences to create multimodal sentences, i.e., a multimodal view of a sentence. By incorporating diverse multimodal sentences into contrastive learning, instance-level alignments between textual and multimodal samples are effectively exploited. Our model circumvents the constraints of paired datasets, facilitating scalable multimodal representation learning with a broader and more varied collection of unpaired data.

In terms of learning efficiency, we investigate the acceleration method of vision-language pretraining. We empirically find that an essential obstacle to training efficiency lies in the entangled prediction rate (percentage of tokens for reconstruction) and corruption rate (percentage of corrupted tokens) in masked language modeling, that is, a proper corruption rate is achieved at the cost of a large portion of output tokens being excluded from prediction loss. To overcome the limitation, we propose free language modeling (FLM), a new pretraining objective that decouples the prediction rate from the corruption rate in masked language modeling. Our method achieves faster convergence by allowing customization of corruption spans for each token, while maintaining competitive performance on downstream vision-language tasks.

Concerning cross-modal alignment granularity, we delve into fine-grained alignments between untrimmed videos and natural language. We propose a grounded vision-language learning (GVL) framework for untrimmed videos, focusing on detecting informative events and aligning multi-sentence descriptions with corresponding event segments. We introduce the parallel decoding paradigm for dense video captioning (PDVC) to segment videos effectively, enhancing the coherence and readability of generated dense captions. Furthermore, two dual pretext tasks are proposed to encourage fine-grained segment-level alignments: text-to-event contrast and event-to-text generation. The framework is versatile and applicable to visually-grounded language understanding and generation tasks.

We conduct extensive experiments to validate our proposed methodologies. These efforts not only advance the frontiers of multimodal learning but also pave the way for more efficient and effective integration of vision and language in machine intelligence systems. (400 words)