Exploratory Analysis of Highly Heterogeneous Document Collections

July, 2013
IDA document: D-4931
FFRDC: Systems and Analyses Center
Division: Information Technology and Systems Division
Arun S. Maiya, John P. Thompson, Francisco Loaiza-Lemos, Robert M. Rolfe See more authors
We present an effective multifaceted system for exploratory analysis of highly heterogeneous document collections. Our system is based on intelligently tagging individual documents in a purely automated fashion and exploiting these tags in a powerful faceted browsing framework. Tagging strategies employed include both unsupervised and supervised approaches based on machine learning and natural language processing. As one of our key tagging strategies, we introduce the KERA algorithm (Keyword Extraction for Reports and Articles). KERA extracts topic-representative terms from individual documents in a purely unsupervised fashion and is revealed to be significantly more effective than state-of-the-art methods. Finally, we evaluate our system in its ability to help users locate documents pertaining to military critical technologies buried deep in a large heterogeneous sea of information.