Software Architecture Challenges for Data Intensive Computing

Data intensive computing is concerned with creating scalable solutions for capturing, analyzing, managing and understanding multi-terabyte and petabyte data volumes. Such data volumes exist in a diverse range of application domains, including scientific research, bio-informatics, cyber security, social computing and commerce. Innovative hardware and software technologies to address these problems must scale to meet these ballooning data volumes and simultaneously reduce the time needed to provide effective data analysis. This paper describes some of the software architecture challenges that must be addressed when building data intensive applications and supporting infrastructures. These revolve around requirements for adaptive resource utilization and management, flexible integration, robustness and scalable data management.