Exploiting the Largest Available Zone: A Proactive Approach to Adaptive Random Testing by Exclusion

Adaptive random testing (ART) has been proposed to enhance the effectiveness of random testing (RT) through more even spreading of the test cases. In particular, restricted random testing (RRT) is an ART algorithm based on the intuition of skipping all the candidate test cases that are within the neighborhoods (or zones) of previously executed test cases. RRT has higher effectiveness than RT in terms of failure detection but incurs a higher time cost. In this paper, we aim to further reduce the time costs for RRT and improve the effectiveness for RT and ART methods. We propose a proactive technique known as “RRT by largest available zone” (RRT-LAZ). Like RRT, RRT-LAZ first defines an exclusion zone around every executed test case in order to determine the available zones. Unlike the original RRT, RRT-LAZ then compares all the available zones to proactively pick the largest one, from which the next test case is randomly generated. Both simulation analyses and empirical studies have been employed to investigate the efficiency and effectiveness of RRT-LAZ in relation to RT and related ART algorithms. The results show that RRT-LAZ has significantly lower time costs than RRT. Furthermore, RRT-LAZ is more effective than RT and related ART methods for block failure patterns in low-dimensional input spaces. In general, since RRT-LAZ employs a proactive technique instead of a passive one in generating next cases, it is much more cost-effective than RRT. RRT-LAZ is also more cost-effective than RT and other ART methods that we have studied.


I. BACKGROUND
The advancement of information technology has triggered an increasing interest in the use of software applications for various activities such as online shopping, mobile apps, and train and taxi booking. Software has become an integral part of people's daily life in all aspects. By 2020, the market for software and software-based services is estimated to be nearly e290 billion for Europe alone [1]. While software brings convenience to us, it also causes many problems owing to the numerous faults that remain undetected. For instance, The associate editor coordinating the review of this manuscript and approving it for publication was Xiao Liu . the inadequacy in software testing is estimated to cost the US 0.6 percent of its GDP [2], or $112 trillion based on the latest World Bank statistics [3]. The quality of software testing is, therefore, of vital importance.
Random testing (RT) is a fundamental software testing technique [4]. It randomly selects test cases from the entire input space and then executes them. Hence, the probability of selecting any location in the input space as the next test case is the same. It does not require any information other than the data types and ranges of the inputs. It is especially good at executing the software in unforeseen circumstances not obvious to human testers [5]. Furthermore, previous experimental results show that RT is efficient in generating test cases [6]. VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Making use of the general property that failure-causing inputs tend to be clustered in contiguous regions, Chen et al. have proposed an improved method known as adaptive random testing (ART) [7]. It has subsequently been applied to different programming languages [8]- [10]. Compared with RT, ART requires fewer test cases to reveal program failures, and the test cases selected by ART are more evenly distributed throughout the input space [11]. The effectiveness of failure detection by ART has been validated to be better than that of RT [7]. Various ART algorithms have been developed to further improve the diversity of test cases based on different concepts.
ART by bisection (ART-B) [12], ART by random partitioning (ART-RP) [12], [13], and ART by two-point partitioning (ART-TPP) [14] are adaptive random testing algorithms based on the idea of partitions. ART-B divides the input space into equal-sized partitions. Test cases can only be selected from those partitions that do not contain previously executed test cases. When all the existing partitions contain previously executed test cases, further partitioning takes place. ART-RP partitions the input space at the location of every previously executed test case and selects the next test case from the largest region. ART-TPP also generates the new test case from the biggest region, but the partitioning does not occur at the locations of executed test cases. Instead, each partitioning location is determined from two points in the input space. The first is the location of a previously executed test case (or a random point if no test case has yet been executed). The second is selected according to farthest distance criterion. Then, the current region is partitioned at the midpoint of these two locations.
Restricted random testing (RRT) [15], [16] is an ART algorithm based on the idea of exclusion. It generates an exclusion zone in the input space around every previously executed test case. Then, it repeatedly generates random test cases until one of them is outside all the exclusion zones. This test case will be selected as the next one to be executed. Experimental results show that RRT performs better than RT and previous ART algorithms. On the other hand, it has to spend more time in test case selection by passively judging whether each candidate case is selectable. In addition, RRT is inappropriate in some situations, such as when the failure rate of the program is too small.
In view of the shortcomings of RRT, we propose a proactive algorithm known as RRT by largest available zone (RRT-LAZ), which reduces the effort in judging whether a candidate case is selectable. Experimental results show that RRT-LAZ uses significantly less time to select test cases when compared with RRT, but the effectiveness of failure detection is preserved.
The remaining parts of the paper are organized as follows: Section II lays the theoretical groundwork for the rest of the paper. Section III describes the original RRT algorithm and the RRT-LAZ algorithm. Section IV presents a set of experiments to compare RRT-LAZ with RT and other ART algorithms, followed by analyses of the results.  Section V investigates the threats to validity of our study. Section VI presents selected related work. Finally, Section VII concludes the paper.

II. FAILURE PATTERNS
A program is said to fail if it produces an incorrect result. An input is said to be failure-causing if it leads to a failure of the program. Bishop [17] has observed that failurecausing inputs tend to be clustered in contiguous failure regions. Chan et al. [18] have further classified the patterns of failure-causing inputs into three categories, namely, point, strip, and block patterns. The main characteristic of the point pattern is that the failure-causing inputs are either stand-alone points or form many regions of very small sizes scattered over the input space. For the strip pattern, the failure-causing inputs form the shape of a narrow strip. A typical example is the domain error highlighted by White and Cohen [19], where ''a specific input follows the wrong path.'' For the block pattern, the main characteristic is that the failure-causing inputs are concentrated in a more compact form than a strip pattern and in a small number of regions. Examples of the three patterns are shown in Figure 1 and the relevant code is shown in Table 1. In the figure, the outer boundaries represent the borders of the input space and the filled regions denote failure-causing inputs.
A straightforward improvement of RT would be to make use of the generic information on typical failure patterns. Intuitively, it would be more effective to select test cases far from non-failure-causing inputs because the failure patterns tend to be clustered. Adaptive random testing makes use of this additional information to enhance its failure-detection capability over random testing.

III. IMPROVING THE PROACTIVENESS OF THE RRT ALGORITHM A. THE ORIGINAL RRT ALGORITHM
The original RRT is an ART algorithm based on exclusion. The main idea of the RRT algorithm is as follows: First, when there are n (≥ 1) previously executed test cases, RRT defines an exclusion zone around each executed test case. The size of an exclusion zone is determined by the size of the input space s, the exclusion ratio R (which is set by human testers), and the number of previously executed test cases n. In a one-dimensional input space, the size of each exclusion zone is set to R * s/n and the exclusion radius is R * s/(2 * n). RRT repeatedly generates candidate test cases randomly from the input space until a candidate test case falls outside of all the exclusion zones. Figure 2 shows a onedimensional input space with values from 0 to 1. We set R as 0.75 and generate the first test case randomly. The second test case can only be selected from (0, t 1 − 0.375 * s/1) and (t 1 + 0.375 * s/1, 1). That is, if RRT generates a point not in (0, t 1 -0.375 * s/1) or (t 1 + 0.375 * s/1, 1), it will repeatedly generate another point from the input space until it falls within (0, t 1 − 0.375 * s/1) or (t 1 + 0.375 * s/1, 1). Similarly, as shown in Figure 2 (b), the third test case can only be selected from (0, t 2 -0.375 * s/2), (t 2 + 0.375 * s/2, t 1 -0.375 * s/2), or (t 1 + 0.375 * s/2, 1). Hence, RRT selects candidate test cases from the whole input domain and then discards the candidates that are inside the exclusion regions. This is a passive approach.
Simulation and empirical studies have shown that RRT is more effective than RT in terms of failure detection capability [20]. However, in order to select the nth test case, RRT needs to generate, on average, log n candidates for each previously executed test case [21], and in order to determine whether the candidate test case is in the exclusion zone, RRT needs to measure the distance between the candidate test case and every executed test case. RRT requires O(n log n) time to generate the nth test case, and hence the time complexity of RRT is O(n 2 log n). As it can be seen, a disadvantage of RRT is that its time overhead is too large. An improved approach, known as RRT-LAZ, is proposed in the next section.

B. RRT-LAZ ALGORITHM FOR ONE-DIMENSIONAL INPUT SPACES
RRT-LAZ is an enhanced version of the RRT algorithm. The first test case t 1 is randomly selected from the input space in the same way as RRT. When there are n (≥ 1) previously executed test cases, RRT-LAZ similarly generates exclusion zones around each executed case. The size definition of exclusion zones is the same as that in RRT. Unlike RRT, however, RRT-LAZ maintains the exclusion zones in sequence of the Hilbert values of the test cases (which will be explained in Section III-C). After determining the exclusion zones and hence the available zones (that is, the regions outside the exclusion zones), RRT-LAZ picks the largest available zone, from which the next test case will be randomly generated. This method selects test case from the available regions and therefore no generated test cases will be discarded, greatly saving the time it takes to select the next test case. It represents a proactive approach that is different from the original passive approach.
Let us determine the time complexity of the RRT-LAZ algorithm. In each iteration, RRT-LAZ generates the nth test case from the largest available zone. To do so, it first computes all the available zones. It needs to traverse n-1 previously executed test cases to determine these zones, which takes O(n-1) time. In other words, it needs O(n-1) time to generate the nth test case. For the RRT-LAZ algorithm for one-dimensional input spaces, the time complexity is therefore O(n 2 ). For multidimensional input spaces, we should also consider the time complexity of the Hilbert space-filling function (to be introduced in Section III-C). The latter time cost is only related to the dimension d and the order of the Hilbert curve m, both of which are constants. Hence, the time complexity of RRT-LAZ is also O(n 2 ) for multidimensional input spaces, which is more favorable than RRT.
We will give a simple example to illustrate the RRT-LAZ process. Consider a one-dimensional input space with values from 0 to 1. We select the largest available zone as follows: Set the size of the largest available zone maxlength to an initial value of 0. The lengthstart and lengthend positions represent, respectively, the start and end points of the largest available zone. The size of an exclusion zone is equal to R * s/n, where R is the exclusion ratio, s is the size of the input space, and n is the number of previously executed test cases. We set the initial size of the exclusion zone to 1. As in RRT, the exclusion radius for one-dimensional input spaces is equal to half the size of the exclusion zone. In RRT-LAZ, the first and the last executed test cases are processed separately. If the value of the first test case t 1 minus the exclusion radius is greater than zero, it means that there is an available zone between 0 and t 1 -radius. That is, lengthstart = 0, lengthend = t 1 -radius, and maxlength = lengthendlengthstart. If the value of the last test case t n plus the radius is less than 1, it indicates that t n + radius is still in the input space. That is, lengthstart = t n + radius, lengthend = 1, and maxlength = lengthendlengthstart. When there is more than one test case, a vital condition in the algorithm must be satisfied. VOLUME 8, 2020 Algorithm 1. RRT-LAZ for One-Dimensional Input Space Inputs.
(1) Target exclusion ratio R (2) Boundaries min and max of input space (3) Predefined ceiling for number of test cases to be executed 1. read inputs R, min, and max; 2. set tcList = ; / * tcList is the list of executed test cases * / 3. set azList = ; / * azList is the list of available zones * / 4. set n = 0; / * n is the number of executed test cases * / 5. set tc = GenRandomTC([min, max]); / * Randomly select a test case from the set [min, max] * / 6. while not (Result(Execute(tc)) = fail or n = ceiling) do / * While the execution result of tc is not a failure and the number of executed test cases has not reached the predefined ceiling * / 7. set n = n + 1; 8. insert tc into tcList, maintaining the list in ascending order through binary search; 9. define exclusion zone for each test case in tcList; / * Length of each exclusion zone = R * (max−min)/n * / 10. availableZone = FindAZ([min, tcList [1]]); / * Find an available zone between min and the first test case. Return null if not found. * / 11. if (availableZone != null) / * If found * / 12.
then then AddtoAZlist(availableZone); 28. end if; 29. set LAZ = Largest(azList); / * Find the largest available zone in azList * / 30. set tc = GenRandomTC(LAZ); / * Randomly select a test case from LAZ * / 31. end while; 32. return; That is, the value of the current test case plus the radius cannot be greater than the next test case minus the radius; otherwise, there will be no available zone between the two test cases. The maxlength can be updated only if the above condition is met.
The implementation of the RRT-LAZ algorithm for onedimensional input spaces is further illustrated in Figure 3. The actual RRT-LAZ process is shown in Algorithm 1. Consider again the input space [0, 1]. Following Line 5 of the algorithm, the first test case t 1 is randomly generated with a value of 0.6, as shown in Figure 3(a). Following Line 9 of the algorithm, taking R = 0.75, and based on the formula R * s/n, the exclusion zone is computed to be [0.225, 0.975]. Following Lines 10 to 28 of the algorithm, the two available zones are [0, 0.225] and [0.975, 1]. The size of the available zone #1 is obviously larger than that of #2, and hence, following Line 29 of the algorithm, the next test case should be selected from the first available zone. This test case t 2 is randomly generated to be 0.2, as shown in Figure 3(b). It can be seen from the updated exclusion zones and available zones that the size of available zone #3 is the largest. As a result, the next test case is randomly generated in this zone.

C. RRT-LAZ ALGORITHM FOR MULTIDIMENSIONAL INPUT SPACES
In Sections III-A and III-B, we have introduced the RRT and RRT-LAZ algorithms for one-dimensional input spaces. This section presents the algorithm for multidimensional input spaces. We will use the Hilbert space-filling function to convert one-dimensional inputs into multidimensional test cases.
(1) Target exclusion ratio R (2) Dimension d (3) Boundaries of multidimensional input space (4) Predefined ceiling for number of test cases to be executed 1. read inputs R, d, and boundaries of input space; 2. set hvList = ; / * hvList is the list of Hilbert values in one-to-one correspondence with the executed test cases * / 3. set azList = ; / * azList is the list of available zones * / 4. set n = 0; / * n is the number of executed test cases * / 5. Map the multidimensional input space to [0, 1] based on Hilbert space-filling curve; 6. set hv = GenRandomHV([0, 1]); / * Randomly select a Hilbert value hv from the set [0, 1] * / 7. set tc = HVtoTC(hv); / * Map hv to multidimensional test case tc based on Hilbert space-filling curve * / 8. while not (Result(Execute(tc)) = fail or n = ceiling) do / * While the execution result of tc is not a failure and the number of executed test cases has not reached the predefined ceiling * / 9. set n = n + 1; 10.

1) HILBERT SPACE-FILLING FUNCTION
The Hilbert space-filling function [22] can map bijectively the points on a one-dimensional line (known as the Hilbert space-filling curve) to the points in a multidimensional space. Figure 4 shows how the points in a one-dimensional space are mapped to a two-dimensional space. The square is first divided into four sub-squares. A first-order curve is constructed by connecting the centroids of the sub-squares by straight edges. The sequencing of the sub-squares is so arranged that a common edge is always shared by a pair of sub-squares. The number at each centroid is known as the Hilbert value. Figure 4(b) shows the second-order curve. In this step, each of the four sub-squares is further divided into four new sub-squares. The first and last sub-squares need to be rotated to ensure that a common edge is always shared by a pair of sub-squares. There are 2 1 * 2 = 4 Hilbert values in the first-order curve for two-dimensional spaces, and 2 2 * 2 = 16 Hilbert values in the second-order curve for two-dimensional spaces.
Let r be the Hilbert value in an m-order space-filling curve to be converted to a d-dimensional test case. The value is represented by a binary number consisting of m groups with d bits in each group. The m groups are indexed by i = 1, 2, . . . , m.
To initiate the dimensional conversion process, we obtain the binary representation of the value r that needs to be transformed. We fill in the most significant part of the integer portion of the binary number with zeros and the least sig-  Then, the conversion process undergoes a sequence of transformations as follows: We use operator ρ to split the binary representation of r into m groups ρ 1 , ρ 2 , . . . , ρ m . 2. We perform the following processes for i = 1, 2, . . . , m: a) Use operator J to record the principal position of ρ i . b) Use operator σ to transform p i into σ i . c) Use operator τ to transform σ i into τ i . d) Copy σ 1 to q 1 . Use operator q to transform σ i into q i . e) Copy τ 1 to t 1 . Use operator t to transform τ i into t i . f) Use operator ω to transform t i into ω i . g) Use operator α to transform q i into a i . 3. The result is a sequence a 1 , a 2 , ..., a m .
The detailed actions of the transformation operators are shown in Table 2.

2) MULTIDIMENSIONAL RRT-LAZ ALGORITHM
We will apply the Hilbert space-filling function to support test case generation for programs with multidimensional input spaces. First, we generate one-dimensional Hilbert values similar to the RRT-LAZ algorithm for onedimensional input spaces in Section III-B. Then, we convert these values into multidimensional test cases using the Hilbert spacing-filling function. The specific RRT-LAZ algorithm for multidimensional input spaces is shown in Algorithm 2.

IV. EXPERIMENTAL EVALUATIONS
In this section, we report the comparisons of RRT-LAZ with related algorithms by applying them to simulated and real-life programs. Since RRT-LAZ involves two concepts, namely, exclusion and partitioning, we only compare it with RT, RRT (which is based on exclusion), as well as ART-B, ART-RP, and ART-TPP (which are based on partitioning). In our experiments, the exclusion ratio R was set according to the original RRT paper [15]. All the experiments were conducted on a PC with Intel Core i3-380M 2.53GHz dual core processor and 6GB RAM running the Windows 7 operating system. We made sure that the performance of the PC was steady in all the conducted experiments.

A. SIMULATION STUDY
We apply two metrics to compare the proposed RRT-LAZ algorithm with RT and popular ART techniques. First, we use the test case generation time, which does not include the time for test case execution as this cannot be simulated without bias. Second, we use the F-measure [7], which refers to the expected number of executed test cases to reveal the first failure.

1) TEST CASE GENERATION TIME
The main purpose of this part of the study is to observe and analyze the time costs of the algorithms. We do not require the ART algorithms to test programs, but simply use them to produce test cases. The test case generation time is the average of the sums of times obtained from 2000 independent tests for input spaces of all dimensions.
According to Figures 5 and 6, which illustrate the time costs for input spaces of one and two dimensions, RRT-LAZ spends less time to generate test cases than RRT and ART-RP. From Figures 7 and 8, which show the time costs for input spaces of three and four dimensions, it can fur-   ther be observed that RRT-LAZ spends less time to generate test cases than RRT, ART-RP, and ART-TPP.
As explained before, a significant disadvantage of the RRT algorithm is that it spends excessive time in test case generation. RRT-LAZ clearly improves on the time cost of its predecessor. Furthermore, RRT-LAZ uses less time than ART-TPP    as the dimension increases, because our method does not need to handle d-dimensional points or d-dimensional zones. On the other hand, RRT-LAZ uses more time than RT and ART-B.

2) EFFECTIVENESS OF TEST CASE GENERATION
We recall that the F-measure evaluates the expected number of test cases to expose the first failure. Thus, a lower F-measure indicates that an algorithm is more effective. We conducted a simulation study of the F-measure with respect to varying failure rates θ. A failure rate denotes the fraction of failure-causing inputs over the entire input space. A lower failure rate means that it will be more difficult to reveal failures.
We used failure rates of 0.0050, 0.0020, 0.0010, and 0.0015. For each failure rate, the seed of random numbers was set between 0 and 10000, and the interval between two seeds was 5. Each experiment was repeated 2000 times for each failure rate, and the average was taken as the overall result. We conducted the experiments in 1-, 2-, 3-, 4-, and 10-dimensional input spaces. As said before, there are three main patterns of failure-causing inputs. All the three patterns were tested in the experiments. The results of the F-measures for testing the simulated programs with the RRT-LAZ and other ART algorithms are shown in Tables 3 to 7.
It can be observed from the first four of these tables that RRT-LAZ is better than other methods for block failure patterns in low-dimensional input spaces. As shown in Tables 3 to 7, when the dimension increases, the effectiveness of ART algorithms decreases. RRT-LAZ is no longer the best algorithm for block failure patterns in 10-dimensional input spaces. Nevertheless, the F-measure of RRT-LAZ is   comparable to the best one. For strip patterns, Tables 3 to 7 show that RRT-LAZ is better than other methods in some cases but worse in other cases. Considering both the failuredetection effectiveness and overhead, we can conclude that RRT-LAZ is more cost-effective than other approaches. Since ART was not intended for point failure patterns, it is not expected to show much improvement over RT. Figures 9 to 12 show the experimental results of various algorithms in box plots, categorized by different failure rates and dimensions. We have run the experiments 100 times to obtain 100 F-measures for each algorithm before drawing the box plots. The dispersions of the F-measures can be observed from these figures.
It can be seen from Figures 9 to 12 that the ranges of the boxes for RRT-LAZ are smaller than those for other ART algorithms. This means that the dispersions of RRT-LAZ are smaller than those of other algorithms.

B. EMPIRICAL STUDY
Previous ART studies used 12 real-life programs to evaluate the effectiveness of ART algorithms in practical situations. These programs were extracted from Numerical Recipes [24] and ACM's Collected Algorithms [25]. Some faults were seeded into the subject programs using the following 4 mutation operators [26].
The input spaces and fault information of these programs are shown in Table 8. Note that a hyperrectangular input space was defined for each subject program.
We used related ART algorithms to test the subject programs in the empirical study. Table 9 shows the F-measures of RT and related ART algorithms. All the results in this   table are averaged over 2000 test runs for each subject program using different seeds. For 11 of the 12 real-life subject programs (except for Program bessj), we find that RRT-LAZ has a better F-measure than RT and related ART techniques. As for Program bessj, since there are two seeded faults via the AOR operator, and since its input space is two-dimensional, RRT-LAZ does not exhibit the best F-measure.
To show the distribution of F-measures for each algorithm in these programs, we also draw box plots of the respective F-measures as shown in Figure 13. For each F-measure, we ran the experiment 100 times and obtained the average value. The dispersions of the F-measures can also be observed in the figure. From these statistical data, we have the following observations. 1) For the majority of the subject programs (7 out of 12, except for Programs airy, el2, gammq, plgndr, and probks), the median of RRT-LAZ is smaller than those of related approaches. 2) For the majority of the subject programs (except for airy, bessj, el2, golden, and sncndn), the interquartile range (the box length) and the maximum (the top end VOLUME 8, 2020 of the whiskers) of RRT-LAZ are smaller than those of related approaches, which shows that RRT-LAZ has a more stable data distribution than the others.

V. THREATS TO VALIDITY
Construct validity refers to the extent that our experimental studies evaluate the actual comparisons of RT and ART techniques. We have adopted the F-measure and the test case generation time as the effectiveness and efficiency metrics. The use of other metrics may give different results.
Internal validity refers to how well our experimental studies are conducted, especially with respect to the dependency of the variables. We have used both simulated and real-life programs in our studies. We have carefully implemented and tested our evaluation tools to assure correctness. Also, the effectiveness of the RRT and RRT-LAZ algorithms depend on the exclusion rate R. In our studies, we have set the value of R to be the same for both algorithms. Setting distinct values of R may produce different results.
External validity refers to the degree that our experimental studies can be generalized. We have used 12 subject programs from Numerical Recipes and ACM's Collected Algorithms. While they are commonly adopted in other ART studies, the use of other programs may produce different results. Having more subject programs with larger sizes can strengthen the generalization.

VI. FURTHER DISCUSSIONS ON RELATED WORK
Numerous adaptive random testing algorithms have been developed by various authors. It is practically impossible to compare a new ART approach with all existing algorithms experimentally.
We have compared RRT-LAZ with RT because the latter is the original technique that adaptive random testing targeted to improve.
RRT-LAZ involves two concepts, namely, exclusion and partitioning. It is, therefore, appropriate to compare it with RRT (which is based on exclusion) as well as ART-B, ART-RP, and ART-TPP (which are based on partitioning).
The fixed-size-candidate-set (FSCS) algorithm using the max-min candidate selection criterion [7] is based on the idea of distances. It first generates a fixed number of candidate test cases and calculates the minimum distance between each candidate and all the previously executed test cases. Then, the candidate with the maximum distance among all the computed minimum distances is selected as the next test case. As we have explained, it is unrealistic to compare a new ART approach with all existing algorithms. Since FSCS is not related to the three basic concepts in RRT-LAZ, we have not included it in our experimental studies.
Chen et al. [27] proposed a mirror adaptive random testing (MART) approach. First, it partitions the given input space into m equivalent disjoint subspaces. It picks one of the subspaces and applies an ART algorithm (say, RRT) to select a test case. It then uses the mirroring function to generate a test case for each of the remaining m-1 subspaces without the need to apply RRT again. MART exhibits the same F-measure as the RRT algorithm but needs only 1/m 2 of the time for distance computation using RRT.
Chan et al. [28] recommended the concept of forgetting for enhancing adaptive random testing. It restricts the execution of an ART algorithm (say, RRT) to a limited number of previously executed test cases. In this way, the time complexity can be improved.
ART with dynamic non-uniform candidate distribution (ART-DNC) [29] is another attempt to improve adaptive random testing by the use of failure-driven test profiles. It reduces the time cost by tuning the process according to the test case distribution in each dimension.
The enhancement of RRT by mirroring, forgetting, and dynamic non-uniform candidate distribution are orthogonal to the enhancement of RRT through the largest available zone. In other words, RRT-LAZ can also be further improved by mirroring, forgetting, or DNC.
Shahbazi and others [30], [31] presented another alternative approach to random testing. They use centroidal Voronoi tessellations to maximize the test case coverage of the input space and presented an RBCVT-Fast algorithm with a time complexity of O(n). However, testers need to specify the target number of test cases (n) at the beginning. If testers overestimate n, they will waste a lot of time in test case generation. On the other hand, if testers underestimate n, after executing the first batch of targets, another set of n evenly spread test cases will need to be generated without taking the first set into consideration. Our algorithm does not need to assume the target number of test cases prior to testing. There is no problem related to overestimation or underestimation.
Recently, ART algorithms have also been proposed for non-numerical inputs. For instance, a linear-time ARTsum algorithm was recommended by Barus et al. [32] using the concepts of categories and choices in category-partition testing. Ciupa et al. [8] and Chen et al. [9] introduced the ARTOO and the OMISS metrics for modeling the distances between objects and between method innovation sequences. We have not compared RRT-LAZ with these algorithms experimentally because they cater for non-numerical inputs.

VII. CONCLUSION AND FUTURE WORK
Adaptive random testing enhances the effectiveness of random testing by making use of the failure distribution properties of most programs under test. Restricted random testing makes use of exclusion zones around previously executed test cases and selects the next test case outside these zones. A major shortcoming of RRT lies in the time overhead due to the generation of candidate cases, which may or may not be outside the exclusion zones. To alleviate the problem, we propose in this paper an RRT-LAZ approach that proactively determines the largest available zone from which the next test case is randomly selected. Compared with RRT, the RRT-LAZ algorithm significantly reduces the time cost while preserving the effectiveness of failure detection. Further experimental results show that, for block failure patterns in low-dimensional input spaces, RRT-LAZ is more effective than RT and the related ART techniques that we have studied. By considering the time cost of test cases, RRT-LAZ is more cost-effective than the other approaches for both block and stripe failure patterns in the input spaces for all dimensions As future work, we will extend our empirical study to higher-dimensional input spaces. We will test larger-scale programs with RRT-LAZ. We will also customize our RRT-LAZ tool to support the testing of OO software.