Process Control in Semiconductor Manufacturing Based on Deep Distributional Soft Actor-Critic Reinforcement Learning

The quality of semiconductor fabrication processes is typically degraded by variations in the manufacturing environment, which can be suppressed by run-to-run (R2R) control schemes. The performance of controlling systems to the produce process which is always highly complex and nonlinear physical model thus is strongly associated with the controlling strategy. However, previous works focusing on less complex semiconductor fabrication processes or linear controlling strategy are both hard to extend the application scenario. A novel structure for a R2R control system based on a distributed form of deep reinforcement learning (DRL), namely, distributional soft actor-critic (DSAC) DRL with twin-value distribution learning, is proposed for multizone pressure control in the chemical mechanical planarization (CMP) process, which is one of the most crucial manufacturing processes for the fabrication of ultra-large integrated circuits (ICs). In addition, several optimization algorithms for DRL, such as twin value distribution learning, are applied, further improving DSAC DRL to enhance the control performance. Compared with other reinforcement learning (RL)-based controllers, the proposed RL control policy achieves better control performance when tested using a multizone CMP virtual metrology (VM) model based on long short-term memory (LSTM) and one-dimensional convolutional neural network (1DCNN) architectures. This deep neural network (DNN) VM model, which is applied for the first time here to test the proposed DRL-based R2R controller for semiconductor manufacturing, is designed to preserve the complexity and nonlinearity of the CMP process by using data recorded from a practical manufacturing process at an IC fabrication facility in Tianjin, China. The novel model-free controlling schemes combined with the new VM model can be used in different R2R application scenarios. Meanwhile the results achieved using the proposed DRL control strategy strongly support its potential application in modern industrial semiconductor manufacturing and offer practical guidance for the further development of CMP procedures.