A systematic data preprocessing approach based on Three-Tier architecture: ensuring reproducibility, version control, and use of cleaned data for digital twins in mineral processing

Clean data is crucial for creating dependable digital twins in mineral processing. Incorporating Machine Learning (ML) algorithms within digital twins further strengthens the significance of utilizing clean data. A systematic and organised data preprocessing approach based on Three-Tier architecture is utilized for methodical and transparent data transformations in this work. Such an approach ensures that the data used by ML algorithms is consistently clean and reliable, enhancing the overall effectiveness of digital twins. The approach has been conceptually well-grounded in a Three-Tier architecture (User tier, Computation tier, Data tier), where different tiers can independently deal with the distinct aspects of data preprocessing. The User tier primarily handles the interaction with the user interface. Data preprocessing fits into the Computation tier, where it handles core preprocessing tasks (different tasks are organised into different modules) related to data cleaning, transformation, feature engineering, and data integration. Lastly, the Data tier focuses on storing and retrieving data. Such separation ensures a modular, extensible, and maintainable approach to managing data processing tasks within the application, thereby significantly enhancing the reproducibility, version control, and reliability of cleaned data in the project.