Analysis of Large and Complex Engineering and Transport Data

This chapter discusses the development of data-intensive methods in research over the last 10 years, centered on the lessons learned through a number of large, national-scale UK projects. The work is focused on the analysis of large and complex data, mainly time-series data from engineering and science. The work has demonstrated two key requirements of systems that need to handle large volumes of data. The first one is that data are typically too large to move to allow them to be processed, requiring the need to move the computation needed to process the data and not the data. The second is that methods to process the data need to be optimized to handle large data. One reasonably good approach to this is to design the methods to recognize the bottlenecks in computer systems. The chapter gives a number of example problems that have demonstrated how this can be achieved.