Pecos: A Scalable Solution for Analyzing and Managing Qualitative Data

Large, heterogeneous, and complex data collections can be difficult to analyze and manage manually. There is a need for scalable and user-friendly approaches that can automate the analysis and management of such collections in a timely and efficient manner. To meet the aforementioned need, we are developing a system named Pecos which combines (1) an android application, (2) cloud computing middleware and resources, and (3) High Performance Computing (HPC) platform and software, for performing large-scale data analysis through an easy-to-use interface. Currently, Pecos can be used to analyze and manage data collections residing on Google drive or on remote Linux systems that are accessible via SSH connection. Some of the steps in the data analysis and management process that are already enabled through Pecos are (1) content-based classification and clustering of documents, (2) filtering or searching the documents on the basis of a text pattern, (3) performing checksum analysis and metadata extraction, (4) supporting format conversion of documents, and, (5) developing visualizations for analyzing the collection. Work is under progress to (1) optimize the algorithms in Pecos for document analysis, (2) develop a recommendation system, and (3) integrate the functionality for analyzing data from social media.