User Privacy Risk Analysis within Website Privacy Policies

The websites’ privacy policies reflect how their users’ personal data is collected, processed, stored, and used. This paper proposes a risk analysis method to assist users in understanding and identifying potential privacy risks in these policies. The method consists of two phases. First, we propose a transfer learning process to refine an existing large language model (LLM) to enable it to understand and answer legal questions effectively. Second, we introduce a retrieval-augmented generation (RAG) technique applied to the refined LLM to analyze risks in privacy policies. Additionally, we define a checklist to assess the compliance of privacy policies with legal regulations. This checklist is mapped to 20 queries, which are used to build prompts for the refined LLM. This allows our refined LLM to provide explanatory answers to users regarding the overall compliance level of privacy policies with relevant regulations, stating the potential risks in these privacy policies. The transfer learning process has been conducted on two existing LLMs. Experiments were performed on 200 random legal questions to compare these fine-tuned LLMs with the originals, demonstrating the efficiency of our proposed method in terms of the cosine similarity index and the F1 score.