Dmap: Automating Domain Name Ecosystem Measurements and Applications

Behind each Internet domain name, there is a set of entities/companies responsible for delivering the various services associated with it, such as Web hosting and e-mail. Together, they form what we refer to as DNS ecosystem. Currently, there is no single measurement tool designed to measure this ecosystem altogether. As a result, researchers that aim at analyzing (parts of) this ecosystem often have to spend significant amounts of time preparing and executing the multiple application measurements and post-processing their heterogeneous raw datasets. Given that time is a scare resource, this complexity diverts researcher's time from actual analysis, ultimately limiting how far many studies go. To help researchers facing this situation, we present Dmap, an active measurement application that reduces the complexity of executing both measurements and analysis. It does so by (i) automating the crawling of several application protocols (DNS, HTTP, TLS/SSL, SMTP, both over IPv4 and IPv6) and (ii) storing the results into a relational data base, enabling researchers to quickly perform hypothesis tests within interactive response times using SQL. Dmap current version has 40 classifiers that generate 166 derived features (e.g., CMS detection, page language), which can be used by researchers and operators to build applications and services. We present an evaluation of Dmap and show three applications that it can be used for, such as profiling the Alexa 1 million domains. We use Dmap at SIDN (.nl registry) for research on the. nl zone and make it open-source for researchers.