Security vulnerabilities assessment of the X.509 protocol by syntax-based testing

This paper describes a methodology for syntax-based vulnerability testing of computer-network protocol implementations, by mutating the protocol data units (PDUs) transmitted to the target implementation. The implementers of a protocol are under a number of different constraints: time, budget, throughput and memory footprint-size. Adequate attention to secure handling of data structures in a PDU can give way to other pressures. The implementation may be designed to meet conformance-testing cases but can have open vulnerabilities to more obscure cases that might not even be possible during normal operation of the protocol. The vulnerabilities can lead to a compromise of the target's security, e.g. buffer overflow. The vulnerability testing approach described in this paper manipulates the grammar of the targeted network protocol to generate a large number of mutated test-cases that can be used to identify security vulnerabilities. This work builds on that of Beizer and the PROWS research group who propose a functional method for assessing protocol implementation security. It adopts a more general approach in its modelling of protocols in order to take advantage of similarities between protocol data structures and to better utilise common abstract syntax constructs (in this case ASN.1), and common transfer syntaxes. It focuses on the mutation of a representation of PDU syntax that is derived from actual protocol PDUs "by example" rather than by specifying and mutating the grammar for the protocol itself. This results in the production of a more universal testing tool applicable to many ASN. 1-based protocols with little or no modification. The methodology and tools developed as part of this work were used with success to test a number of network protocols, including a commercial product using ASN.1-specified X.509 public key certificates.