Reliability demonstration testing: Can we afford 80% confidence?

The current Department of Defense (DoD) emphasis on improving the reliability of fielded systems by enhancing reliability-related activities during the acquisition process is clearly documented in Interim DoD Instruction (DoDI) 5000.02 [1], DoD Directive-Type Memorandum (DTM) 11-00 3 [2], and the Reliability, Availability, Maintainability, and Cost (RAM-C) Rationale Report Manual [3]. Program Managers are required to formulate a comprehensive and integrated Reliability and Maintainability (R&M) program using an appropriate strategy to ensure R&M requirements are achieved. Required activities include reliability testing at the system and subsystem level and inclusion of reliability growth curves in the program Systems Engineering Plan (SEP) beginning at Milestone A, and in the Test and Evaluation Master Plan (TEMP) beginning at Milestone B. This strong emphasis on RAM-C during acquisition is necessary and welcomed to ensure the warfighter's requirements are met. There are, however, challenges to be faced by program offices as they attempt to comply with these requirements, while cost effectively meeting the operational need. Emphasis on accomplishment of reliability demonstration testing and reliability growth planning during new system acquisition has increased in recent years. While these activities provide an excellent opportunity to improve the fielded reliability of systems, challenges exist in balancing the risk of not achieving the required reliability with the cost of design, production, and reliability demonstration testing of systems. In order to obtain the desired 80% confidence that a system meets the stated reliability requirement, a “Goal” value of Mean Time Between Failure (MTBF) (or whatever reliability metric defines the system requirement) must be established that is greater than the requirement. The exact value of the goal is dependent on the warfighter's requirement, planned demonstration test duration, and desired Producer and Consumer confidence levels. When these factors are all considered, the established Goal may become significantly higher than the warfighter requirement. This inflation of the requirement can lead to over-design of the system and a significant investment in reliability growth testing. This additional acquisition phase investment may not be cost-effective or justified unless sustainment phase cost can be significantly reduced over the service life of the system as a result of the improved reliability. However, a reduction in sustainment phase cost with a more “reliable” system is not always possible. For example, in the case where redundancy is added to a system to meet availability or mission reliability requirements, there is typically an increase in sustainment cost as a result of the additional complexity from the added equipment. Extensive guidance and proven methodologies are currently available on reliability growth planning, but none adequately address the cost-benefit of achieving the inflated Goal. The return on investment associated with identifying and implementing design improvements and the increased test time associated with achieving 80% confidence in the system reliability should be an integral part of the decision making process throughout system acquisition and life cycle support activities.