Multiobjective Hierarchical Reinforcement Learning for Plant-Wide Optimization of Bayer Process

The plant-wide optimization of the Bayer process aimed at increasing yield and reducing steam consumption is challenging as it optimizes conflicting production indices under dynamic environments and involves multiple unit processes and time scales. This optimization problem is difficult to solve through single-layer approaches. To this end, this article proposes a novel multiobjective hierarchical reinforcement learning algorithm for coordination across multiple time scales to better adapt to complex dynamic environments and changing market demands in the Bayer process. First, we design a hierarchical plant-wide framework to learn the hierarchical policy: the upper-layer policy optimizes multiple monthly production indices to plan feasible target values of daily production indices; the lower-layer policy decides on hourly operational indices of each unit process to make actual daily production indices follow their prescribed target values. Next, the decomposition-based multiobjective hierarchical policy gradient combined with a population update and task-selection strategy is developed in the upper layer to obtain a Pareto policy set that corresponds to different market demands. Finally, in order to enable adaptive decision-making when raw ore properties change, environmental context is introduced as the policy input. The hierarchical policy is learned directly from operational data in a bottom-up training manner and realizes online sequential decision-making in the application phase. The effectiveness of the proposed algorithm has been demonstrated by the simulation experiments on a large-scale alumina refinery in Shanxi Province, China.