AutoRed: Automated Attack Scenario Generation Framework for Red Teaming of LLMs

Even though Large Language Models (LLMs) are highly beneficial, they pose significant security concerns, particularly in the realm of privacy protection. Sensitive information is often provided to LLMs during conversations and may be retained as in-context memory. This raises the risk of unintended data exposure. In the existing paradigm, a red team comprising human testers is tasked with generating input prompts (i.e., test cases) to provoke undesirable responses from LLMs. Yet, relying solely on human testers is both costly and time-intensive. This paper presents AutoRed, an innovative learning framework developed to automatically generate malicious attack scenarios for extracting sensitive information from LLMs. Our framework places particular emphasis on prompt injection—the process of injecting malicious prompts to extract conversation histories from LLMs to uncover private data. AutoRed comprises three key components: malicious prompt generator, sensitive information extractor, and stop point identifier. These components work together to enable prompt injection, ensuring a seamless process. Our extensive experimental evaluation, spanning diverse defense strategies and various LLMs, demonstrates the efficacy of AutoRed. This evaluation not only rigorously assesses the resilience of defense mechanisms but also measures the safety alignment of LLMs, thereby highlighting the potential of our automated framework as an efficient red teaming tool for identifying vulnerabilities and enhancing security within LLMs.