Interpreting and analyzing neural networks for NLP : a knowledge management perspective

Abstract

Language technology has become pervasive in everyday life, with applications like Google Search, Amazon Alexa, Apple Siri, etc. The success of such Natural Language Processing (NLP) systems is built on the advance in deep neural networks, thus it also comes at the expense of model becoming less interpretable, i.e., such models are often perceived as ``black box''. The lack of model transparency not only hinders future research effort in advancing the field, but also limit these models' utility in applications that have certain requirement in terms of reliability, ethic, and legitimacy.

Opening up the black box of neural NLP models has attracted attention from different NLP sub-field. This thesis offers a novel perspective and investigates methods to analyze how NLP models cope with real-world knowledge, since the ability to connect textual symbols with human knowledge and exploit these knowledge in decision making is the key to NLP models' success.

Toward this goal, we first develop perturbed masking -- a parameter-free method for analyzing and interpreting a recent prevalent pre-trained language model -- BERT. The analysis quantifies the linguistic knowledge BERT captured and sheds light on its remarkable success in many NLP tasks. Second, we present a directly interpretable Multimodal Machine Translation (MMT) model, which consists of an interpretable component that quantifies its dependency on visual knowledge. Our findings stress the importance of interpretability in MMT and suggest potential directions for improvement. Third, we explore how interpretability can lead to a better model design and demonstrate with an application on knowledge-based question answering.