Multihead Discrete Action Calibration Proximal Policy Optimization Method for Pixel Antennas With High Degrees of Freedom

Pixel antennas expressed in the form of discrete variable structure (DVS) demonstrate immense potential in electromagnetic (EM) performance, but the design process relies on a huge full-wave simulation dataset. To efficiently design pixel antennas with high degrees of freedom (DoFs), in this study, a multihead discrete action calibration proximal policy optimization (MDAC-PPO) reinforcement learning (RL) method is proposed. Specifically, a sophisticated multihead discrete mechanism is developed to generate the placement status of all metal patches in a probabilistic form. Furthermore, a coupling calibration mechanism that considers local metal density is proposed to break down the intricate optimization target, expediting the iterative process. In addition, to delve deeper into potential designs that might otherwise be neglected, an exclusive or (xor) operation state update manner is integrated into the PPO agent. Three design examples on a pixel antenna with a 2256-sized solutions space, including optimization targets on impedance matching, gain, efficiency, and related bandwidth, are presented to show the optimization efficiency and performance of the MDAC-PPO method. The measured $\vert \mathrm {S}11 \vert $ of the fabricated prototypes aligns well with simulations, further validating the reliability of the MDAC-PPO method in practical applications. Meanwhile, this method also provides a new perspective for designing other EM devices with DVS.