Mode Collapse Alleviation of Reinforcement Learning-based GANs in Drug Design

De novo drug design is a challenging task that involves understanding the principles of chemistry, chemical properties, and the rules that govern molecular interactions. Deep learning-based generative models, such as MolGAN, offer a promising approach for generating new molecules with the desired chemical properties from molecular graphs. Such models often combine a discrete generative adversarial network (GAN) and reinforcement learning (RL) to produce highly valid and novel molecules. However, the severe mode collapse problem leads to low performance. This study aims to alleviate and investigate the effect of multiple factors on mode collapse. We conducted experiments on different sampling methods, training epochs, and datasets of various volumes and evaluated the experimental results using performance metrics such as validity, uniqueness, novelty, and diversity. The experimental results demonstrate that noise sampling distributions, training epochs, and training data volumes affect performance. The experimental results provide a direction for mitigating the mode collapse problem for RL-based discrete GANs.