On the Design of a Data Engineering Learning Platform Using Web Technologies and LLMs

As data surge, the demand for skilled data engineers significantly increases, underscoring the importance of data engineering. However, learning data engineering skills can be daunting due to the complexity of setting up multiple platforms, often unnecessary as companies typically employ other professionals to handle infrastructure. Additionally, data engineering is rarely taught in traditional educational settings, leaving interested students at a disadvantage. To address this, this project aims to develop a web-based platform that simplifies data engineering learning, providing hands-on experience for free without complex setups for users from different backgrounds. The platform includes a Large Language Model (LLM)-powered chatbot for real-time guidance, creating an interactive learning environment. With access to our platform, users can instantly access the necessary tools and resources. Typically, a web page will have everything required for a course, streamlining the virtual learning process and reducing setup time.