Big Data Challenges: A Program Optimization Perspective

Big Data is characterized by the increasing volume (of the order of zeta bytes) and velocity of data generation. It is projected that the market size of Big Data shall climb up to $53.7 billion by 2017 from the current market size of $5.1 billion. Big Data in conjunction with emerging applications such as RMS applications and others has sown the seeds of exascale computing. In a similar vein, In [12], Sexton argued that applications from domains such as materials science, energy, environment and life sciences will require exascale computing. Recent studies directed towards challenges in building exascale systems and charting the roadmap of exascale computing conjecture that exascale systems would support 10-to 100-way concurrency per core and hundreds of cores per die. In [15], HPC Advisory Council predicts that the first exaflop system will be built between 2018 -- 2020. In this paper present a program optimization perspective to the challenges posed by Big Data.