[Keynote Speaker-2] Challenges in Handling and Processing Huge Data

Summary form only given, as follows. Data-intensive computing is considered as the fourth paradigm in science. The term “data-intensive computing” did not establish in other communities although they are also confronted with enormous amounts of data. Nowadays, Big Data refers to data sets that are too large, too complex, too distributed for analysing them by conventional methods. One strategy for handling Big Data is known as “software to the data” which is applicable when it is more efficient to bring the analysis tools to the data than, vice versa, to apply traditional methods where, for example, all data are collected at some place and analysed there. The data production rate is expected to increase exponentially for the time being. This is particularly true in science where the resolution power of experiments is steadily improving. Sooner or later it has to be taken into account that it is not feasible to store all data anymore. A new era is on the horizon: Huge Data. Huge Data have to be pre-analysed during the data-taking period in order to extract a sufficiently small subset of data that is worth to be analysed in more detail later on. An effective and efficient preselection in real-time or near-realtime is most critical for successfully handling Huge Data. This is made more challenging if during the pre-analysis that has to be done in parallel, intermediate results have to be exchanged. The talk considers selected challenges of Huge Data. Some examples from different scientific communities are presented. The complete presentation was not made available for publication as part of the conference proceedings.