Keyword and fact identification and annotation : machine learning approaches

Abstract

In the big data era, automatic retrieving useful information from various documents is a hot topic in artificial intelligence study. The popularity of social media makes people overwhelmed by the massive amount of information, it is necessary to develop some techniques to identify useful information efficiently. To better analysis this problem, we divide the documents into two types, one is the traditional document and the other is social media document. The traditional documents include news and articles, etc., which using formal language structures and strictly obey the grammatical rules. On the other hand, social media documents are such as posts on Facebook or Twitter, which are often short and informal. In this dissertation, we investigate keyword and key sentence identification and annotation in various types of documents. Machine learning is used in the information retrieval process. For social media documents, we first propose two novel frameworks of new word detection in Chinese tweets. Then we design a word annotation mechanism which interprets the Tweet-born words by automatic tagging with text labels. In addition, a hierarchical clustering algorithm is introduced to realize relevant words clustering. For the traditional documents, we propose a relevant sentence selection method which can improve the performance of question-answering systems.