The Importance of System Engineering Competences and Knowledge in Large Scale Digital Research Infrastructure Projects

Modern data driven science extensively uses digital technologies and requires large scale Research Infrastructures (RI) combining computational, storage and data management facilities to support data collection, processing and scientific workflow management. The success of building dedicated RIs and their efficiency will depend on adopting modern technologies and well-defined architectures that support the sustainability and long term evolution of technical solutions. Building and operating modern data driven RIs and scientific instruments require a wide spectrum of competences and knowledge related to modern technologies, system and software engineering, including also Big Data infrastructure engineering. The paper provides background information on large European RI and e-Infrastructure projects and analyses different cases/scenarios where the availability of necessary system and software engineering competences and knowledge are critical for the success of RI. The paper summarises the sustainable architecture design principles that can provide both guidance for the technical design of RI and a basis for targeted training for engineering and scientific personnel. The presented work is based on the author's experiences of developing and teaching courses on Big Data Infrastructure Technologies for Data Analytics (BDIT4DA) and DevOps for Software Engineering (DevOps4SE) that include necessary competences and learning outcomes required for building modern data-driven infrastructures and applications. The described courses are developed following the methodology proposed in the EDISON Data Science Framework (EDSF). The paper shares the experience of how both courses have evolved and been adapted to the continuous technology development and specific needs of the developer teams.