Neuron-Level Control Based on Multi-Frequency Policy Dissection

Neuron-level control is an effective approach to enhancing the intrinsic decision-making capabilities of reinforcement learning models, obviating additional model training, as well as the need to modify reward functions. However, existing neuron-level control algorithms are difficult to be widely applied due to poor control performance, which is potentially attributed to the use of a single DC signal when overwriting the hidden states of neurons. To achieve high-performance neuron-level control, this study proposes the Multi-frequency Policy Dissection (MPD). MPD firstly identifies the neuron most relevant to each kinematic attribute by considering all frequency components within the temporal signals of both the neuronal hidden states and kinematic attributes comprehensively. Then MPD uses mixed signals containing multiple main frequency components to overwrite the hidden states of the identified neurons to achieve the desired agent behaviors. Compared with the state-of-the-art neuron-level control algorithm, MPD exhibits significant advantages across various evaluation metrics and unlocks complex behaviors in the bipedal robot. Furthermore, we visualize the correlation between the kinematic attributes and the hidden states of the matched neurons, and explore the gradual establishment of this correlation during the model training.