yoU aRe a Liar://A Unified Framework for Cross-Testing URL Parsers

A variety of attacks, including phishing, remote-code execution, server-side request forgery, and hostname redirection, are delivered to users over the web. The beginning of most of the web exploits is an innocent-looking URL. Malformed or misinterpreted URLs can lead to remote code execution attacks as well. The IETF and WHATWG standards organizations define the components of a URL and act as an implementation guide for URL parsers. They state which characters are allowed in each portion of the URL and loosely suggest what to do in case an undefined character is present in the URL. The existence of two standards is the first concern, and the addition of server-side request forgery in the latest version of OWASP Top 10, suggests that neither of these standards is being followed accurately and concisely. Moreover, neither of these specifications describe an exact implementation standard, causing inconsistencies in the way the various parsers interpret the same URL. For example, malicious users can find ways to craft URLs to look like they are pointing to one resource but actually direct the user to different one. This problem is worsened when one application uses two separate parsers for validation and resource fetching.In this paper, we design a framework that unifies the testing suites of 8 URL parsers from popular web-related projects and highlights the inconsistencies between them. We examine and dive deep into the URL parser implementation across the most popular libraries, browsers, and command-line tools, and discover many open areas for exploitation. Our findings include identifying categories of inconsistencies, developing proof-of-concept exploits, and highlighting the need for a comprehensive implementation standard to be developed and enforced at the earliest.