From Hidden Wiki 2020 to Hidden Wiki 2021: What Dark Web Researchers Comprehend with Tor Directory Services?

The dark web searching mechanism is unlike surface web searching. On one popular dark web, Tor dark web, the search is often directed by directory like services such as Hidden Wiki. The numerous dark web data collection mechanisms are discussed and implemented via crawling. The dark web crawler assumes seed link, i.e. hidden service from where the crawling begins. One such popular Tor directory service is Hidden Wiki. Most of the hidden services listed on the Hidden Wiki 2020 page became unreachable with the recent upgrade in the Tor version. The Hidden Wiki 2021 page has a limited listing of services compared to the Hidden Wiki 2020 page. This motivated authors of the present work to establish the role of Hidden wiki service in dark web research and proposed the hypothesis that the dark web could be reached better through customized harvested links than Hidden Wiki-like service. The work collects unique hidden services/ onion links using the opensource crawler TorBot and runs similarity analysis on collected pages to map to corresponding categories.