NHPP Testability Growth Model Considering Testability Growth Effort, Rectifying Delay, and Imperfect Correction

Over the last several years, many testability growth models (TGMs) have been developed to greatly facilitate engineers and managers in tracking and measuring the growth of testability as system is being improved. Most TGMs consider only one or two variation patterns of the aspects, such as the testability growth effort (TGE) in testability design limitation (TDL) identification, the rectifying delay and the new TDL introduction in TDL correction. However, the ignorance of such joint consideration may lead to a lower fitting ability to the fault detection/isolation data. Inspired by the counting idea of non-homogeneous Poisson process (NHPP), a NHPP based testability growth model (TGM) considering the recurrence rate function (RRF) of TDL identification, TDL correction and new TDL introduction is proposed for the foundation of TGM. A real data set of a missile control system is used to validate the above TGMs in fitting ability, estimation accuracy and prediction capability. Results show that the bell-shaped curve can fit the identification process and rectifying delay process of TDL well, and the imperfect correction of TDL really exists in the testability growth test (TGT), and the inflected s-shaped and Gamma function based TGM gives good capacity to the real data set.


I. INTRODUCTION
Testability is defined as the probability of fault detection and isolation, which is quantified by various testability indexes, such as fault detection rate (FDR), fault isolation rate, fault alarm rate, and so on [1]- [3]. Good testability is a very important data source for condition monitoring, fault diagnosis, health prognostics and residual life prediction of equipment [4]- [12]. In fact, testability design limitations (TDLs) like unanticipated failure, test vacancy, ambiguity group, fuzzy point, improper threshold, and faulty diagnosis method are unavoidable in testability engineering. Testability growth is an efficient way to enhance the system testability through design modification and/or other corrections performed throughout a system's life cycle. Therefore, TGT is the most effective approach to identify and correct the TDLs, and increase the value of testability [1], [2].
The associate editor coordinating the review of this manuscript and approving it for publication was Yu Liu .
TGM is an analytic, parametric, and time variable function. A good TGM can track and predict the testability level well. Based on the TGM, the testability designer can plan TGT and make some decisions like the time to stop testing or continue to test.
Over the last several years, many TGMs have been proposed and these models greatly facilitate engineers and managers in tracking and measuring the growth of testability as the system is improved [13]- [20]. Li et al. in [13]- [15] found a TGM based on bell-shaped testability growth effort function, in which the main attention is paid on TDL identification. In these works, it is assumed that the delay between TDL detection and correction is zero, the TDL detection rate and correction rate are constants, and there is no new TDL and new failures introduced into the system during the whole TGT cycle. Zhao et al. in [16]- [18] proposed a Markov chain based TGM for the in-time fix program. In this work, the imperfection of TDL correction is considered, but the undetectable rate and uncorrectable rate of any failure cannot be gained easily, especially for large-scale and complicated equipment. As a result, the accuracy of the TGM cannot be ensured. Yang et al. in [19], [20] proposed a Bayesian testability growth model based on the test data. However, the TGM can only estimate the value of testability,but the prediction of the variation tendency of testability cannot be achieved.
All the above mentioned TGMs have important theoretical guiding significance and practical value to carry out TGT. However, the existing studies ignore some key aspects, like the learning process of testability designer, the rectifying delay between the TDL detection and TDL correction, and the imperfect correction of TDL. Designer's learning means that, with the process of TGT, the designer becomes more and more familiar with the system, and the TDL correction rate becomes higher. Nevertheless, it is noted that there is time lag between TDL identification and TDL correction. In this paper, we call this time lag as the rectifying delay. In addition, for large-scale and complicated system, new TDL can be introduced into the system with the process of the TDL correction, this new TDL introduction is termed as imperfect correction in this paper.
The target of TGT is to identify the TDLs and correct them one by one. In fact, not all the hidden TDLs can be identified, the identification rate of TDLs depends on the testability effort consumed on the TGT. In addition, not all the identified TDLs can be corrected and the correction rate of TDLs depends on the level of designers. At the same time, new TDLs will be introduced into the system, especially for large-scale and complicated system. In this case, the introduction rate of new TDLs depends on the level of designers as well. Therefore, the identification rate of TDLs depends on the testability growth effort consumed on TGT, and the correction rate of TDLs and the introduction rate of new TDLs depends on the level of designers which is time-varying with the learning process of testability designer. Together with these discussions, it is concluded that how to jointly consider TDL identification, TDL correction and new TDL introduction to propose TGM is a very important problem. To the best of our knowledge, there is no reported result on joint consideration of TDL identification, TDL correction and new TDL introduction in constructing TGM.
In general, TDL identification, TDL correction, and new TDL introduction will not occur at the same time intervals. Therefore, it is relatively easy to confirm the number of identified TDL and the number of corrected TDL with the progress of TGT. Based on this, this paper proposes a new NHPP TGM to jointly consider the testability growth effort, rectifying delay, and imperfect correction.
There is an extensive body of literature on software reliability growth model based on NHPP [21]- [25]. These methods consider the debugging process as a counting process characterized by its mean value function (MVF). The relationship between the test time and the amount of test-effort expended during that time was considered, in which the test effort was often described by the traditional bell-shaped curves [25]- [32]. Schneidewind et al. in [33] pointed out that the fault correction process was the most key factor which influences the reliability of software. He founded a software reliability growth model considering fault rectifying delay, and analyzed the fault rectifying delay from the time angle. However, in this work, the time lag is assumed to be a constant. inspired by [33], other scholars founded software reliability growth model considering different time delay functions and imperfect corrections. Li et al. in [25] proposed NHPP software reliability models considering fault removal efficiency and error generation, and the uncertainty of operating environments with imperfect debugging and testing coverage. Song et al. in [34] studied NHPP software reliability models with various fault detection rates considering the uncertainty of operating environments. Zhu et al. in [35] proposed a NHPP software reliability model with a pioneering idea by considering software fault dependency and imperfect fault removal.
Referring the ideas of software growth models, we propose a TGM considering the testability growth effort, rectifying delay and imperfect correction simultaneously based on NHPP. Firstly, this paper uses the above some bell-shaped curves to describe the relationship between the test time and the amount of TGE expended during that time by analyzing the consumption rule of TGE so as to obtain the identification rate of TDLs. Secondly, the delay mechanism between TDL identification and TDL correction is analyzed, and some time-dependent bell-shaped functions are introduced to describe the rectifying delay. Thirdly, new introduction TDLs which are called imperfect correction are considered in this paper. Finally, we validate the proposed model by a real data set of a missile control system.
The remainder of the paper is organized as follows. In section 2, we propose some TGMs considering the testability growth effort, rectifying delay and imperfect correction simultaneously. In section 3, we give some criterions for TGM assessment. Section 4 discusses the goodness of the above TGMs on the fitting ability, estimation ability and prediction ability with the application to a real data set. Section 5 concludes the paper.

II. TGM DEVELOPMENT A. FRAMEWORK OF TGM
The assumptions used for description of TGT are as follows.
(1) During the whole course of TGT, n stages will be carried out successively.
(2) Testability growth is the result of an iterative ''test, identify TDL one by one, correct TDL together, and test'' process. Here, we call the process as the delay correction mode.
(3) With the progress of TGT, q i , (i ∈ {1, 2, . . . , n}) is the true value of FDR after the ith TGT stage, where q 1 < · · · < q i < · · · < q n . At the same time, q i = q i+1 , where q i+1 is the true value of FDR before the (i + 1) th TGT stage.
(4) In general, the TGT time can be represented by the serial number of testability growth test stage instead of the  Based on the above assumptions, the growth of FDR with stepped distribution tendency is shown in Fig. 1. The target of TGT is to identify the TDLs and correct them one by one. Therefore, there are two main steps in the progress of TDL. One is TDL identification, and the other is the identified TDL's correction. In fact, not all the hidden TDLs can be identified, and not all the identified TDLs can be corrected. At the same time, new TDLs will be introduced into the system, especially for large-scale and complicated system. The three critical actions at any stage are shown in Fig. 2.
In general, as for the ith stage, we introduce the following notations: a 0 (i) is the number of TDLs hidden in the system before the ith stage, m d (i) is the cumulative number of TDLs identified up to the end of ith stage, m c (i) is the cumulative number of TDLs corrected up to the end of ith stage, and m r (i) is the cumulative number of new TDLs introduced into the system caused by TDL correction process up to the end of ith stage.
A good TGT means that high and efficient TDL identification and correction can be achieved, and at the same time, no new TDLs will be introduced into the system. Hence, the number of TDLs can describe the level of system testability. After the ith TGT stage, the number of TGLs hidden in the system is expressed by a 0 (i + 1). At the same time, a 0 (i + 1) is the initial number of TDLs hidden in the system at the (i + 1) th stage. That is to say, a 0 (i) represents the value of testability after the ith stage, a 0 (i − 1) represents the value of testability before the ith stage. The number of TDLs which still hide in the system can represent the testability quality.
As a result, a 0 (i) can be expressed as where a 0 is the number of TDLs hidden in the system at the beginning of TGT. Testability growth models are mathematical functions describing the TDL detection process, TDL correction process, and new TDL introduction process. The general model is proposed based on the following assumptions: (1) TDL identification rate function is highly related to the amount of testability growth effort expenditures spent on identifying the TDLs.
(2) TDL correction rate function is proportional to the designer's level. Considering that not all identified TDLs can be corrected timely, there is time delay between TDL identification and TDL correction considering TGT designer's learning effect.
(3) New TDLs will be introduced into the system. The cumulative value of m r (i) is proportional to the level of testability engineers.
Based on the above assumptions, we define the TDLs can be identified but not be corrected as the remaining testability design limitations (RTDLs), and denote it as y(i). Then, y(i) can be written as As such, Eq(1) can be rewritten as FDR is defined as the capability to detect fault occurred in the unit under test. The mathematical model of FDR can be formulated as [2] where M is the whole number of failure mode set which can be gained by failure mode effects and criticality analysis (FMECA), that is, M is a constant when the unit under test is fixed, and that, M is a prior information of TGT, N D is the number of failure mode which can be detected by testability design successfully. At the ith stage, N D (i) can be expressed as: Substituting N D (i) in Eq(5) into Eq(4) yields where q(i) is the expression in general for FDR. According to Eq(6), if we know the mathematical expressions of m d (i), y(i),and m r (i), we can found the TGM of FDR accurately. Therefore, how to describe the variation trend of m d (i), m c (i), m r (i) is the main aspect in TGT. In the following, we apply NHPP to describe the time-dependent behavior of the cumulative number of m d (i), m c (i), and m r (i) up to a certain testing time.

B. NHPP
The NHPP is one of the simplest and most important types of counting processes that are commonly used for the modeling of certain kinds of recurring events, including the cumulative TDL identification, the cumulative TDL correction, the cumulative TDL introduction in TGT, and so on. In this 9074 VOLUME 8, 2020 paper, considering the time-dependent behavior of the cumulative number of m d (i), m c (i), and m r (i) up to a certain testing time, we propose to apply the NHPP to model these quantities. A counting process {N (t), t ≥ 0} is said to be a NHPP if it satisfies the following axioms: The increments of the process in disjoint time intervals are mutually independent.
where m(t) is called the mean value function (MVF), k is the count up to time t. Making , m(t) can be expressed as where λ(t) is the intensity of m(t).

C. NHPP IN TDL IDENTIFICATION
The variation partner of m d (t) can be expressed by NHPP with special λ d (t). Thus, the main attention in this section is paid to analyzing the variation mechanism of m d (t) and select a proper form of λ d (t).
In general, the number of m d (t) is highly related to the amount of testability growth effort (TGE) expenditures spent on identifying the TDLs. In TGT, the TGE can be represented as man-hour, TGT cost, the times of fault injection, and so forth. The functions that describe how a TGE is distributed over the TGT phase are referred to as testability growth effort function (TGEF). LI et al. in [14] analyzed that, at the whole TGT phase, λ d (t) increases firstly and then decreases at different rate. Based on the increase-decrease characteristic, an inflected s-shaped curve is proposed to fit the variation tendency of TGEF in testability growth model, and the application results show that the inflected s-shaped curve has the smallest value of bias, variation, and prediction error, compared with the other four TGEFs. Here, we will use the inflected s-shaped function and a delayed s-shaped function which increase firstly and then decrease to fit the practical growth rate of λ d (t), respectively.
Specifically, the λ d (t) can be formulated as an inflected s-shaped function as follows Based on Eq (9), the instantaneous TGE with inflected sshaped is shown in Fig. 3.
Based on Eq(8) and calculating the integral, we can further have With the certain fixed parameters, we can draw the variation curve of m d (t) with time t as shown in Fig. 4. Fig. 4 shows that: at the beginning of the TGT, FDR level is relatively lower, TDLs can be identified easily, and the variation curve of m d (t) increase. With the progress of VOLUME 8, 2020  TGT, the FDR increases to a certain level. That is to say, the number of TDLs hidden in the system is less and the TDL identification is very difficult. Therefore, the number of m d (t) consumed in identifying the TDLs cannot increase considering the restrain of excessive cost and too much time consumption. This is reasonable because no system testability design company will spend infinite resources on TGT considering cost & development cycle.
Similarly, λ d (t) can be written by a delayed s-shaped function as follows With reference to Eq(11), the instantaneous TGE based on delayed s-shaped is shown in Fig. 5.
Based on Eq(8) and calculating the integral, we can further have The curve of m d (t) based on Eq(12) is drawn as follow.

D. NHPP IN TDL CORRECTION
TDL identification cannot increase the value of FDR, but TDL correction can increase the value of FDR. The TDL correction process can be viewed as a learning process because the designers become more and more familiar with the testing environments and tools with time progresses. It is assumed that the designers' skills are gradually improved over the test time, and then level off as the  residual identified TDLs become more difficult to correct. Therefore in the process of TDL correction, the learning factor of testability designers should be considered. In this case, the intensity of λ c (t) should be a time-dependent increasing function, which can be expressed as an exponential function.
With reference to Eq(13), the variation curve of λ c (t) is shown in Fig. 7.
Based on Eq(8) and calculating the integral, there is With the fixed parameters, we can draw the variation curve of m c (t) with time t, as shown in Fig. 8.
We define the TDLs can be identified but cannot be corrected as the remaining testability design limitations (RTDLs), and denotes it as y(i), (i ∈ {1, 2, . . . , n}). Then, considering the TGEF and learning factor, y(i), (i ∈ {1, 2, . . . , n}) can be calculated based on Eq(14)  With reference to Eq(15), the variation curve of y(t) is shown in Fig. 9. With the effect of learning factor, the variation curve of y(t) is bell-shaped which is shown in Fig. 9. In [10], [11], three bell-shaped curves are used to describe the variation tendency of RTDLs. Here, we will use two functions to describe the variation tendency of y(t).
One is the Gamma function which can be written as The other is the Rayleigh function which can be written as The curves of Gamma function and Rayleigh function have similar shapes to Fig. 9.

E. NHPP IN NEW TDLs INTRODUCTION
In fact, the correction process is usually far from being perfect. Some TDLs encountered by users are those introduced during the correction process. Therefore, it is essential to incorporate imperfect correction into the foundation of TGM. Considering the learning factor of testers, the RF of m r (t) should be a time-decreasing function such as With reference to Eq(18), the variation curve of λ r (t) is shown in Fig. 10.
Based on Eq(8) and calculating the integral, we can further have The TDL identification ability is proportional to the numbers of TDLs which still hide in the system, i.e.
where, b 2 is a constant. By substituting the y 1 (t) in Eq (16) and Eq (17), m r (t) in Eq (22) into Eq(20) respectively, we can get With the help of software Mathematica, the Eq(21) and Eq (22) can be solved under the boundary condition m d (0) = 0, we can further have x 0 e −t 2 dt.
(1) A TGM with inflected s-shaped TGEF, Gamma RTDL function, and m r (t) in Eq (19) is called IG-TGM which is expressed in (25), as shown at the bottom of this page.
(2) A TGM with delayed s-shaped TGEF, Gamma RTDL function, and m r (t) in Eq (19) is called DG-TGM which is expressed in (26), as shown at the bottom of this page.
(3) A TGM with inflected s-shaped TGEF, Rayleigh RTDL function, and m r (t) in Eq (19) is called IR-TGM which is expressed in (27), as shown at the bottom of the next page.
(4) A TGM with delayed s-shaped TGEF, Rayleigh RTDL function, and m r (t) in Eq (19) is called DR-TGM which is expressed in (28), as shown at the bottom of the next page.

III. ESTIMATION OF TGM PARAMETERS AND COMPARISON CRITERIONS A. ESTIMATION OF TGM PARAMETERS
Fitting a proposed model to actual TGT data involves estimating the model parameters from the real TGT data set. Two popular estimation techniques are the maximum likelihood estimation (MLE) and the least square estimation (LSE) [36]. The MLE method estimates parameters by solving a set of simultaneous equations. However, the equation set may be very complex and usually must be solved numerically. The LSE minimizes the sum of squares of the deviations between what we observe and what we expect. Here, we employ LSE to estimate the parameters of the above four TGMs.
For illustrative purposes, only the parameters N 1 , β 1 , N 2 , β 2 , θ, A, b, a 0 in (26) are estimated by LSE. S m d (N 1 , β 1 ), S m c (N 2 , β 2 , θ) and S m dr (A, b, a 0 ) can be written as follows. Firstly, where m dk is the cumulative number of identified TDLs obtained from actual TGT data set, and m d (k) is the cumulative number of identified TDLs given by Eq (13).
Taking the partial derivatives of S m d with respect to N 1 , β 1 , ϕ to zeros, we can obtain the least squares estimators of N 1 , β 1 . Similarly, where y k is the cumulative number of RTDLs generated from actual TGT data set, and y(k) is the cumulative number of RTDLs which can be got by Eq (16).
Taking the partial derivatives of S y with respect to N 2 , β 2 , θ to zeros, we can obtain the least squares estimators of N 2 , β 2 , θ.
Then, substituting the least squares estimators of N 1 , β 1 , N 2 , β 2 , θ into Eq(24), we have (31) where m dr (k) is the cumulative number of identified TDLs which can be got by Eq (24). Taking the partial derivatives of S yr with respect to A, b, a 0 as zeros, we can obtain the least squares estimators of A, b, a 0 .
Similarly, the parameters of other TGMs like Eq(25), Eq (27), and Eq(28) can all be estimated by LSE.

B. COMPARISON CRITERIONS
A TGM can generally be analyzed according to its estimation ability, fitting ability and predictive ability. In this paper, the TGMs are compared with each other based on the following three criterions.

1) THE ACCURACY OF THE ESTIMATION CRITERION [36]
For practical purposes, we will use the accuracy of estimation (AE) to calculate the accuracy of estimation. The AE is defined as where m d is the actual cumulative number of identified TDLs after the TGT, and a 0 is the estimated number of initial TDLs.
Here, m d is obtained from system testability TDL tracking after TGT. A smaller AE indicates a smaller estimation error and better performance.

2) THE GOODNESS-OF-FIT CRITERIA
To quantitatively compare long-term predictions, we use mean square of fitting error (MSE) because it provides a well-understood measure of the difference between actual and predicted values. The MSE is defined as [31], [32] where y(t i ) is the expected number of TDLs by time t i estimated by a model and y i is the actual number of TDLs by time t i . A smaller MSE indicates a smaller fitting error and better performance.

3) THE PREDICTIVE VALIDITY CRITERION
The capability of the model to predict TDL identification and removal behavior from present & past TDL behavior is called the predictive validity. This approach proposed by Musa et al. in [36] can be represented by computing relative error (RE) for a data set Assuming we have identified and removed D TDLs by the end of TGT time t e , we employ the TDL data up to time t a (t a ≤ t e ) to estimate the parameters of m r (t). Substituting the estimates of these parameters in the MVF yields the estimate of the number of TDL m r (t e ) by time t e .The estimate is compared with the actual number D.The procedure is repeated for various values of t a .We can check the predictive validity by plotting the relative error for different values of t a . RE closer to zero imply more accurate prediction. Positive values of error indicate overestimation and negative ones indicate underestimation.

IV. CASE STUDY
To validate the proposed TGMs, TGT on a missile control system has been performed. The TGT data set employed in this paper (listed in Table 1) was from the testability laboratory of National University of Defense Technology for a missile control system. FMECA of the missile control system had been done and had gained that the missile control system consisted of approximately 400 functional circuit level failures. Over the course of 12 weeks at the design & development stage, 167 TDLs were identified by injecting 243 functional circuits level failures. Failures were injected     Table 2 lists the estimated values of parameters of different TGMs based on LSE. Fig. 11 describes the fitting effect of inflected s-shaped function and delayed s-shaped function to the actual number of cumulative identified TDLs. We can see that the inflected s-shaped function has better fitting ability than the delayed s-shaped function for the actual data set which listed in Table 1. Fig. 12 describes the fitting effect of Gamma function and Rayleigh function to the actual number of cumulative RTDLs.  It is observed that the Gamma function has better fitting ability than the Rayleigh function for the actual data set which listed in Table 1.
Based on the estimated parameters listed in Table 2, we can calculate the values of AE and MSE of the above four TGMs, the values of AE and MSE are listed in Table 3.
From Table 3 we see that the IG-TGM and IR-TGM have lower values of AE than the DG-TGM and DR-TGM TGMs. In addition, the IG-TGM and DG-TGM have lower values of MSE than the IR-TGM and DR-TGM. A smaller AE indicates a smaller estimation error and better performance. At the same time, a smaller MSE indicates a smaller fitting error and better performance. Therefore, the IG-TGM yields a better fit for this data set.
Then, Figs 13-16 depict the RE curves for different selected TGMs.
RE closer to zero imply more accurate prediction. Therefore,  show that the IG-TGM has lower value of RE than the other three TGMs.
Substituting the estimated value listed in Table 2 into Eq (25), Eq (26), Eq (27),and Eq(28), we can get four TGMs of FDR. At the same time, we collect the fault detection data of the missile control system through the whole life cycle, and draw the variation curve FDR based on the 9080 VOLUME 8, 2020   actual data. The actual value of FDR is calculated by Eq(4). Fig. 17 depicts the growth curve of FDR and the actual variation curve at the whole TGT stage. Fig. 17 shows that the IG-TGM has better ability, fitting ability and predictive ability than the other three TGMs.
Further, if the imperfect correction have not been considered like paper [13]- [15], based on the data set listed in Table 1, Fig. 18 gives the comparative result of our method with the methods in [13]- [15] in the predictive and fitting ability. Fig. 18 shows that the IG-TGM gives better tracing ability than the methods of Li et al. in [13]- [15] which do not consider the rectifying delay and imperfect correction.
Although the results obtained by the proposed method show some various advantages over existing methods, it should be noted that the performance of TGM strongly depends on the kind of data set. If a system testability designer  plans to employ TGM for estimation of testability growth of unit under test during system development processes, the testability designers need to select several representative models and apply them at the same time. As indicated by the above application results to real data set, the proposed method can be considered as an underlying alternative to plan test growth test.

V. CONCLUSION
In this paper, we propose a TGM considering the testability growth effort, rectifying delay and imperfect correction simultaneously based on NHPP. Unlike existing studies, this paper first uses some bell-shaped curves to describe the relationship between the test time and the amount of TGE expended during that time by analyzing the consumption rule of TGE so as to obtain the identification rate of TDLs. Second, the delay mechanism between TDL identification and TDL correction is analyzed, and some time-dependent bell-shaped functions are introduced to describe the rectifying delay. Third, new introduction TDLs which are called imperfect correction are considered in this paper. From application results of the proposed method to real data, we can draw the following conclusions.
(1) The bell-shaped TGEF is a good approach providing a more accurate description of resource consumption during TGT phase. Particularly, the inflected s-shaped TGEF has the best fitting ability compared with delayed s-shaped TGEF when applied to data set for a missile control system.
(2) The rectifying delay and imperfect correction really exit in the process of TGT. Therefore, the TGMs considering testability growth effort, rectifying delay and imperfect correction are reasonable. Comparatively, the Gamma curve has better fitting ability than the Rayleigh curve when applied to data set for a missile control system.
(3) The IG-TGM gives better estimation ability, fitting ability and predictive ability than the other three TGMs when applied to data set for a missile control system. At the same time, the IG-TGM gives better tracing ability than the results which do not consider the rectifying delay and imperfect correction. ZONGHAO YANG received the B.S. and M.S. degrees in control science and engineering from the High-Tech Institute of Xi'an, Xi'an, China, in 2015 and 2017, respectively.

ACRONYMS
He is currently a Lecture with the PLA Rocket Force Academy, Qingzhou, China. His research interests focus on e testability design, testability demonstration, testability growth, testability evaluation, prognostics and health management, reliability estimation, predictive maintenance, and lifetime estimation. His research interests focus on testability design, testability demonstration, testability growth, testability evaluation, prognostics and health management, reliability estimation, predictive maintenance, lifetime estimation, and sneak circuit analysis. VOLUME 8, 2020