Sensorless Speed Control of Ultrasonic Motors Using Deep Reinforcement Learning

Compactness, lightweight, and a simple structure are typical features of ultrasonic motors (USMs), but these features are often compromised by the addition of external sensors. Toward reduced cost and system complexity, sensorless speed control of USMs has been proposed. Yet, decoupling the speed estimation and control problems has led to complex, suboptimal controllers that cannot handle the nonlinearity of USMs. This article presents an end-to-end deep reinforcement learning (DRL) speed controller that performs model-free implicit speed estimation and optimal control. First, the response of the USM speed and electric states was characterized. Then, an approximate simulation environment was developed to design and pretrain the controller before transfer to real-world settings. To optimize the controller, a modified soft actor–critic (SAC) algorithm was used, incorporating a multiheaded
${Q}$
network and online entropy/action regularization. The control problem was formulated as a Markov decision process (MDP) with a control Lyapunov reward function and a driving frequency control action. To determine the optimal sensorless input state, a neural network (NN) speed estimator was applied to various combinations of inputs. The optimal state, which resulted in the lowest speed estimation loss (around 5 r/min), consisted of only the feedback voltage and current signals, eliminating the need for large torque or temperature sensors. Upon successful training, the proposed controller robustly tracked varying target speeds and load torques while also providing an accurate estimate of the true speed.