LLM-Based Topic Modeling for Dark Web Q&A Forums: A Comparative Analysis With Traditional Methods

Topic modeling is a critical tool for understanding the thematic structures of unstructured text data, particularly in specialized domains like the dark web. This study compares the effectiveness of large language models (LLMs) and traditional topic modeling techniques in analyzing dark web Q&A forums, characterized by short, informal, and context-specific posts. We evaluate two LLMs—GPT and Gemini—against traditional methods, TF-IDF (scikit-learn) and Latent Dirichlet Allocation (LDA, Gensim), in their ability to generate meaningful and coherent topics. Our findings suggest that LLMs consistently outperform traditional methods, capturing contextually relevant themes such as “scam,” “bitcoin,” and “hacking” with an average semantic similarity score of 0.724 between GPT and Gemini, compared to 0.477 with Gensim. Additionally, LLMs produced significantly lower Levenshtein distances, with an average of 13.49 between GPT and Gemini, compared to 18.22 with scikit-learn and 17.97 with Gensim, indicating greater alignment in topic extraction. In exploring the impact of topic granularity, we found that while single-word topics effectively capture the main themes of dark web discussions, two-word topics provide additional context and specificity, enhancing the interpretability of complex discussions. For instance, topics like “bitcoin wallet” and “hacking tools” illustrate how two-word labels can convey nuanced meanings that single-word topics may overlook. This qualitative analysis underscores the importance of selecting appropriate topic modeling techniques based on the characteristics of the text and the requirements of the analysis, highlighting their potential as valuable tools for analyzing complex and unstructured text data in practical applications for law enforcement and research.