RPZ-Based Suspicious Direct Outbound DNS Traffic Detection Mechanism With Adaptive Policy Updates

Domain Name System (DNS) based C&C communication has gotten much attention over the past decade due to the increase of malware using DNS protocol. A legitimate usage of DNS protocol in domain name resolution requires Name Server (NS) records and the corresponding glue A records achievements before sending DNS queries to the authoritative DNS servers. The achievements generally are conducted on DNS full-service resolvers. However, malware-infected computers tend to communicate with the C&C servers directly using DNS protocol, which is called direct outbound DNS traffic in this research, without via any DNS full-service resolvers on purpose to conceal their malicious behavior. In this research, we present a suspicious direct outbound DNS traffic detection mechanism using DNS Response Policy Zone (RPZ) in order to detect and block the malicious behavior of malware-infected computers during the C&C communication stage. All NS records and the corresponding glue A records achieved in an organization network via DNS full-service resolvers will be registered in the RPZ and the DNS packets whose destination IP addresses have no match in the RPZ will be identified as suspicious direct outbound DNS traffic. We implemented a prototype system and evaluated the features of the proposed mechanism in a local Software Defined Network based (SDN-based) experimental environment. The evaluation results confirmed that the prototype system correctly detected and blocked the target DNS traffic based on the policies of RPZ. Moreover, we confirmed that adaptive policy update is applicable to the mechanism and the performance of the DNS traffic process is acceptable for deployment in the real network.