Design Challenges/Solutions for Environments Supporting the Analysis of Social Media Data in Crisis Informatics Research

Crisis informatics investigates how society's pervasive access to technology is transforming how it responds to mass emergency events. To study this transformation, researchers require access to large sets of data that because of their volume and heterogeneous nature are difficult to collect and analyze. To address this concern, we have designed and implemented an environment - EPIC Analyze - that supports researchers with the collection and analysis of social media data. Our research has identified the types of components - such as NoSQL, MapReduce, caching, and search - needed to ensure that these services are reliable, scalable, extensible, and efficient. We describe the design challenges encountered - such as data modeling, time vs. Space tradeoffs, and the need for a useful and usable system - when building EPIC Analyze and discuss its scalability, performance, and functionality.