Strategies for structured and fault-tolerant design of recovery programs

A recovery program is a software component of a fault-tolerant computer system which is responsible for establishing an operational hardware configuration and resurrecting the computation interrupted due to component failures. Development of an effective recovery program is made difficult by the unusual input to the program, i.e., a faulty machine whose behavior is undefined. In this paper we present several program design techniques and system recovery strategies that have been found useful in obtaining an easily understandable recovery program. The strategies are first presented in abstract forms and then illustrated with a model and a recovery program for the Fault-Tolerant Spaceborne Computer (FTSC). Because of the difficulty of obtaining a single perfect recovery procedure, the FTSC recovery program is equipped with both an alternate recovery procedure that is used when the primary procedure is not effective, and an acceptance test that determines the need for invoking the alternate.