vRM: Verifying Reference Monitors via Exhaustive Access Pattern Generation

A sandbox isolates program execution in a restricted environment to reduce the risk of compromise. In application sandboxes, reference monitors mediate system calls issued by application processes and enforce predefined security policies through access control. However, guaranteeing that reference monitors are robust against all attacks is challenging, as attackers who compromise sandboxed applications can issue arbitrarily complex system call sequences. Formal verification is a promising approach for ensuring software security, but modeling all access patterns a reference monitor must handle remains difficult. In this paper, we propose vRM, a verification scheme for reference monitors based on systematic access pattern generation. We leverage SMT solvers to systematically generate concurrent access patterns that could lead to unauthorized accesses, using OS resource models and predefined access control policies. We then use concurrent model checkers to verify whether reference monitors prevent all such unauthorized accesses. To demonstrate feasibility, we implement a proof-of-concept verification for Time-Of-Check to Time-Of-Use (TOCTTOU) vulnerabilities. Our implementation uses the Z3 SMT solver for exhaustive access pattern generation and applies CDSChecker to explore concurrent access patterns. Our evaluation confirms that vRM effectively identifies known TOCTTOU vulnerabilities in the reference monitor of Apache 2.4.58. Furthermore, we formally verify our implementation of a known TOCTTOU-resistant algorithm.