Real time recovery of fault tolerant processing elements

A critical problem in the design of ultra-reliable fault tolerant systems is that of how to bring a redundant member back on-line, after a transient fault, without degrading critical real-time functions. Recovery from transients is imperative to maintain necessary system reliability in the face of transient errors which have been estimated to occur at a rate of 5 to 100 times that of permanent failures. Excessive delays associated with recovery become a problem when as much as 1 Mbytes of RAM in the faulty processor must be made congruent with the processing majority while maintaining full functionality of critical, real-time control algorithms. This paper describes a hardware assisted recovery technique which uses memory "tags" to determine which memory segments need to be restored such that recovery can be performed incrementally without affecting real-time operational tasks. Also presented is performance data associated with this technique's application to a Draper Laboratory quad-redundant processor responsible for vehicle control of a manned undersea vehicle.