CRALA: A Chinese-centric Risk Simulation and Assessment Framework for LLM Agents

In recent years, large language models (LLMs) become more and more powerful. They can do many interesting tasks, like connecting to different external tools—sometimes we call them “plugins”—to help handle real things in the world. But at the same time, giving LLM agents more real-world power also creates a lot of risks we must be careful about. Checking these risks is not easy at all. It can cost lots of time and energy because we have to make special plugins and do complicated tests just to see if the model might do something unsafe. Most of the “serious” security checks, like the ones done on GPT-4, focus mostly on English. In Chinese, we know very little about the dangers with existing research. Many research pays attention to English, but not so much to Chinese cases, even though it’s super important. To fix this problem, we made a new framework called CRALA (Chinese-centric Risk Assessment for LLM Agents). It can simulate many situations in Chinese and then carefully check if there are any safety issues hiding inside these LLM agent tasks. CRALA includes two main parts: a Chinese Interaction Simulator, which creates and records pretend conversations, and an Evaluator, which looks at all these interactions and decides if interaction might be unsafe. We tested CRALA with human reviewers on 55 different Chinese scenarios. The results show that about $\mathbf{8 1. 5 \%}$ collected test cases are realistic. When the CRALA flags a case as dangerous, about 63.5% really were problems. We even tested 100 risky cases in Chinese apps with popular LLMs like GPT-4. GPT-4 only handled around 35% of them safely. This shows that we still have a long way to go before we can trust these agents completely, at least in Chinese contexts. In conclusion, CRALA makes checking these risks much less painful and gives us a better understanding of how safe Chinese LLM agents really are.