Empowering Hybrid Models with Attention-Based Time-series Transformers: A Case Study in Batch Crystallization

Due to safety concerns, direct implementation of black-box tools for fully data-driven deep-neural-network (DNN) digital twins faces challenges. To overcome this, hybrid models that combine physics-based first principles (FP) with machine learning (ML) have gained popularity, seen as a ‘best of both worlds' solution. Recently, time-series transformers (TSTs) using multi-headed attention mechanisms have shown excellent predictive performance for a wide array of time-series prediction tasks. To this end, a novel TST-based hybrid modeling framework for batch crystallization was developed, offering enhanced accuracy and interpretability compared to traditional black-box models. Specifically, the developed TST-based hybrid model allows online estimation of instantaneous growth and nucleation rate (i.e., G&B) for any arbitrarily chosen operating condition in batch crystallization of dextrose. The estimated G&B values can then be directly fed to an FP crystallization model to generate the evolution of crystal size distribution (CSD) and other system states. Moreover, the proposed TST-based hybrid model utilizes powerful attention mechanisms to (a) under short and long-term contextual information, and (b) seamlessly learn the functional dependence of system states, thereby setting the foundation stones for next-generation attention-powered hybrid models.