Red Teaming LLMs: A Stackelberg Game Approach to AI Safety

Artificial Intelligence (AI) models such as large language models (LLM) have demonstrated substantial advancements in recent years. Along with capabilities, researchers have started to study the potential risks of AI generated content. These risks encompass reflective bias, toxic outputs, factually incorrect information, and adversarial outputs. However, in these existing works the focus has primarily been on the assessment of AI models not on the analysis of adversarial use-cases of these models. Red teaming research has focused in considerable depth on the adversarial use-case of conventional AI models, however very few works explore in detail the game-theoretic aspects of the red-teaming task for LLMs. LLMs can provide responses to questions posed in natural language, necessitating exploration of ways in which uncommon or non-obvious questions could be generated. Constructing an effective ensemble policy that ensures diversity and uncertainty in the query responses from different LLMs is non-trivial, as it requires a game-theoretic formulation to evaluate the selection of policies. In this work, a novel red teaming game framework (RTG) for LLMs and the Game-Theoretic Red Teaming System (GRTS) solver to balance the textual LLM policies is proposed. GRTS is guaranteed to converge to a Nash equilibrium (NE) when inner iterations converge and also incorporates lie detection for robustness against gradient leakage. The effectiveness of GRTS is demonstrated in extensive experiments in query generation and diverse query selection tasks, as well as a case study showing GRTS-LLM could generate effective adversarial prompts to find zero-shot prompt injection attacks on internal systems of LLMs.