An approach to testing COTS software for robustness to operating system exceptions and errors

One of the least tested but most critical portions of software systems is error and exception handling. Error/exception handling routines are the safety net for any system to handle unexpected circumstances such as when operating system (OS) or hardware failures occur. As more critical systems are developed from commercial off the shelf (COTS) software, the robustness of these applications to operating system failures, and in general, to failures from third party software, becomes increasingly critical. We present an approach and tool for assessing the robustness of COTS applications to failures from OS functions or other third-party COTS software. The approach consists of wrapping executable application software with an instrumentation layer that can capture, record, perturb, and question all interactions with the operating system. The wrapper is used to return error codes and exceptions from calls to operating system functions. The effect of the failure from the OS call is then assessed. If the application crashes under these anomalous conditions, the application is determined to be non-robust to a particular failing OS call. A failure simulation tool has been developed for testing the robustness of Win32 applications to these types of anomalous OS conditions.