Evaluating Large Language Models Versus Traditional Tools in OS Command Injection Exploit Generation

The advancement of AI and machine learning has led to significant innovation, with generative AI and Large Language Models (LLMs) transforming academia, corporations, and cybersecurity. Traditional tools like Commix are commonly used for vulnerability testing but often lack adaptability and efficiency in complex environments. This paper proposes the use of ChatGPT turbo 3.5 and Command R by COHERE to generate OS command injection payloads for web application vulnerability testing, comparing their effectiveness with traditional tools like Commix in exploiting vulnerabilities.The research involved generating payloads using traditional tools and fine-tuning ChatGPT and COHERE with a custom dataset to improve their performance in creating tailored malicious payloads. Conducted in a controlled Kali Linux environment with Mutillidae II as the vulnerable web application, the results showed the Fine-Tuned ChatGPT turbo 3.5 model's success rate is 63.9% higher than that of traditional tools. Additionally, ChatGPT turbo 3.5 outperforms the COHERE model by a substantial margin, with a success rate that is 475.6% higher than COHERE. Finally, the traditional tools show a 251.2% higher success rate than the COHERE model. This demonstrates ChatGPT's potential as an effective tool for vulnerability scanning and highlights the importance of integrating LLMs into cybersecurity strategies to address evolving threats.