The Danish National Energy Data Lake: Requirements, Technical Architecture, and Tool Selection

Renewable Energy Sources such as wind and solar do not emit CO
₂
but their production vary considerably depending on time and weather. Thus, it is important to use the flexibility in device loads to shift energy consumption to follow the production. For example, an Electrical Vehicle (EV) can be charged very flexibly between arriving home at 5PM and leaving again at 7AM. Utilizing all available energy flexibility requires applying machine learning and AI on massive amounts of Big Data from many different actors and devices, ranging from private consumers, over companies, to energy network operators, and using this to create digital solutions to enable and exploit flexibility. The project Flexible Energy Denmark (FED) is building the foundation for this for the entire Danish society. Specifically, FED collects data from a number of Living Labs (LLs) in representative real-life physical environments. The data is stored in the Danish National Energy Data Lake, called FED Data Lake (FEDDL) to enable efficient and advanced analysis. FEDDL is built using only open source tools which can run both on-premise and in cloud settings. In this paper, we describe the requirements for FEDDL based on a representative LL case study, present its technical architecture, and provide a comparison of relevant tools along with the arguments for which ones we selected.