Toward Enhancing Sequence-Optimized Malware Representation With Context-Separated Bi-Directional Long Short-Term Memory and Proximal Policy Optimization

Malware proliferation is a major threat to computer systems, and malware classification techniques are effective for analyzing and identifying malware. Recent intelligent malware classifiers intend to integrate natural language processing techniques for better identification performance, however, representation learning presents a fundamental challenge in this new paradigm. Currently, representation models either rely on predetermined structures or ignore structure. Thus, we propose a malware vector representation model utilizing an improved context-separated bi-directional long short-term memory (CS-Bi-LSTM) network with reinforcement learning (RL) to discover optimized structures for learning sentence representations. Our model filters out irrelevant representations and captures long-term dependencies. To generate word vectors, we use the CS-Bi-LSTM network, which employs two unique unidirectional long short-term memory (LSTM) cells for each context. Following that, we develop a proximal policy optimization (PPO)-based RL structure to capture relevant representations with feedback from a quality estimator. Through intrinsic and extrinsic evaluations on two malware datasets, our proposed model demonstrates better performance by filtering redundant information and achieving an acceptable vector representation. Then, our proposed method achieves the highest predictive performance with a classification accuracy of 98.54%.