A Simulation Budget Allocation Procedure for Finding Both Extreme Designs Simultaneously in Discrete-Event Simulation

Although discrete-event simulation has been widely used in various engineering fields, its efficiency remains an issue. Ranking and selection (R&S) procedures can solve this efficiency problem by allocating a limited simulation budget intelligently. While the existing R&S procedures mostly aim to find only the best simulation input design, practitioners sometimes require the worst design as well to analyze systems requiring high reliability, such as military systems, municipal waste management, etc. Motivated by these practical needs, we propose a simulation budget allocation procedure for selecting both extreme designs simultaneously in the presence of large stochastic noise. To maximize the accuracy of the selections under a limited budget, the proposed procedure sequentially allocates a small budget and updates the simulation results such that they can be used as significant evidence for the correct selections. Our experimental results on benchmark and practical problems demonstrate improved efficiency compared to previous works. It is expected that the proposed procedure will be effectively utilized in the fields of the fourth industrial revolution, such as digital twins that demand quickly finding both extreme designs to maintain synchronization with the corresponding real systems.


I. INTRODUCTION
Discrete-event simulation is widely used to analyze modern industrial complex systems, such as manufacturing [1], military [2], smart grid [3], telecommunications [4], and transportation [5]. The most significant advantage of the simulation is that, with just a few assumptions, it can accurately analyze complex systems that cannot be described using closed-form analytic models [6]. However, the efficiency of discrete-event simulation is still a concern. Typical simulation models have random variables or processes to represent the uncertainty of the real world, resulting in stochastic noise in the simulation output. Thus, the expected value of the output for a simulation input design (e.g., a particular configuration of decision variables) can only be estimated as the average of output samples obtained from independently The associate editor coordinating the review of this manuscript and approving it for publication was Donatella Darsena .
repeated simulations (i.e., simulation replications). As the noise increases, a greater number of replications are required per design to obtain an accurate estimate, and the total computational costs may be prohibitively expensive as the number of inputs increases. Moreover, with the higher complexity of recent industrial systems, the increasing cost per simulation run intensifies the efficiency problem. Consequently, how to allocate a simulation budget (i.e., a limited number of simulation replications) efficiently to find the desired input design has drawn considerable attention in the field of simulation [7].
If the objective is to select the best simulation input designs from a finite set of alternatives, ranking and selection (R&S), a well-established branch of statistics, can be an efficient solution [8]- [10]. The best design here is to minimize (or maximize in some cases) the expected value of the simulation output. Based on the ordinal optimization, R&S intelligently allocates a limited number of simulation replications to correctly find the best design depending on the statistical comparisons and inferences for the designs' simulation results. Thus far, many efficient R&S procedures have been developed. Based on the sequential allocation procedure, these procedures iterate the following two steps to find the best design correctly: 1) allocate a few replications depending on the statistical inference for the designs' simulation results, then 2) update the results via additional simulations. R&S procedures' distinctive feature lies in the allocation step; that is, how to allocate those few further replications to maximize the statistical evidence for the correct selection of the best design [11].
The two-stage indifference-zone (IZ) procedure [12] focuses on frequentist evidence to achieve a predefined lower bound guarantee of the probability of correct selection P{CS}. It allocates further replications to the IZ designs that have simulation results similar to the current best design. Unlike the IZ procedure, the optimal simulation budget allocation (OCBA) procedure [13] applies Bayesian posterior distribution to describe the evidence of the correct selection. The OCBA procedure allocates further replications using the derived optimal allocation rules to asymptotically maximize an approximated lower bound of P{CS}. Although it is typically more efficient than the IZ procedure [11], due to the nature of the allocation rules that use the sample mean and variance values, the OCBA procedure may be inefficient in the presence of large stochastic noise, where the sample mean has an inaccurate value [14]. Recently, the uncertainty evaluation (UE) procedure has been proposed to efficiently solve practical problems that have large noise [15]. To maximize P{CS}, the UE procedure allocates further replications heuristically depending on the uncertainty defined with p-values in the statistical hypothesis test. Compared to the OCBA procedure, it has high robustness against noise by considering the precision of the sample mean additionally.
Most previous R&S procedures aim to find only the best input design or subset. However, practitioners in engineering sometimes require the worst design as well as the best design. In other words, both extreme designs are desired. For example, when analyzing a battleship's decoy system for torpedo avoidance, various scenarios that the battleship may encounter should be considered. The survival rate of the battleship for each scenario design can be estimated over many simulation replications of the anti-torpedo combat simulator [16]. Given k scenario designs, practitioners are interested in not only the best design, but also the worst design that minimizes the survival rate: even if the survival rate for the best scenario is very high, the decoy system should be reconsidered if its worst-case scenario occurs frequently in practice. In another example, when investigating the layout of a municipal sewage treatment plant in a city [17], practitioners demanded to find the best and worst layout designs simultaneously to make reliable and suitable layout plans.
In addition to these examples, selecting both extreme designs simultaneously is useful for multiple-criteria decision-making (MCDM) problems, which are commonly encountered in practice. This is because the selected extreme designs for each criterion can be used to construct the ideal and nadir points, providing insight to decision-makers. These designs are also of worth in best-worst scaling, a famous discrete-choice model that requires picking the highest-and lowest-utility items [18]. Several population-based optimization algorithms, such as the ant colony algorithm, genetic algorithm, and particle swarm optimization, require finding the best and worst designs among the population to increase its quality [19], [20]. For example, one of the selection schemes in a genetic algorithm is to push the elite design directly into the next population and eliminate the worst design. In summary, the philosophy of finding both extreme designs has tremendous potential.
However, little research exists on selecting the best and worst designs. Zhang et al. [21] proposed the OCBAbw procedure based on the OCBA procedure, and Xiao et al. [22] developed the OCBAmn procedure for selecting the best and worst subsets by extending the OCBAbw procedure. In addition, Zhang et al. [23] generalized the problem of OCBAbw and suggested the OCBAsr procedure to select n ranked subsets. As mentioned previously, these OCBA-based procedures may be inefficient when large stochastic noise exists, such as simulation models for modern industrial systems. Meanwhile, without loss of generality, finding the worst design is mathematically the same as finding the best one; thus, we can select both extreme designs at the same time by using the existing R&S procedures (such as the UE procedure) twice. For example, dividing a given simulation budget in half, half can be used to find the best design and the other half to find the worst design. This method can be effective in specific cases where the budgetary need to find both designs is similar; otherwise, it is inefficient (see the experimental results of the hOCBA and hUE procedures in Table 2). In addition, because prior information for the simulation output of given designs is typically unknown before conducting the simulation, it can be difficult to decide how best to divide a budget.
In this paper, we propose an R&S procedure for selecting both extreme designs simultaneously. The objective of the proposed procedure is to maximize P{CS} for both extremes under a constraint of simulation budget. To efficiently resolve practical problems that have typically large stochastic noise, we developed a procedure utilizing the UE approach with high robustness against noise. We characterize the uncertainty of each design based on statistical hypothesis test and p-value, and present a sequential procedure that gradually allocates a few further replications according to the evaluated uncertainty. Various experimental results on both numerical benchmark and practical problems demonstrate the necessity of the proposed procedure compared to previous methods, such as the OCBAbw procedure.
The remainder of this paper is organized as follows: Section II defines the problem, and Section III proposes the simulation budget allocation procedure. Section IV exhibits the experimental results. Finally, the conclusion is given in Section V. VOLUME 8, 2020

II. PROBLEM DEFINITION
This paper uses the following basic notations: T : total number of simulation replications (i.e., the given simulation budget); k : number of simulation input designs; x i : simulation input design, where subscript i represents the design index (i.e., i ∈ {1, . . . , k}); : set of input designs, = {x 1 , . . . , x k }; Y ij : simulation output sample of x i in the jth replication, number of actually allocated replications at x i ); Based on common assumptions used in the R&S literature, a simulation output sample is supposed to follow independent and identically distributed normal distribution with unknown µ i and σ 2 i for every replication of the same x i . That is, the simulation model has a Gaussian noise with zero mean in its output. This normality assumption can be reasonable in simulation fields as Y ij is typically obtained as an average value or batch means, so the central limit theorem holds [8].
In addition, we assume that no prior knowledge of µ i and σ 2 i for each x i is given before conducting simulations.
Suppose there is only one extreme design on each side, and denote that the design with the smallest µ i is the best and the inverse (i.e., greatest µ i ) is the worst. Then, based on the obtained simulation results of the designs, the selected best design x b and worst design x w are defined as follows: For the selections of x b and x w to be correct, the unknown µ b should be smaller than the other designs' µ i , and the unknown µ w should be larger than the other designs' µ i . Thus, similar to [13], P{CS} for both selected extreme designs can be defined as follows: Simply, P{CS} converges to 1 as T goes to infinity; however, a simulation budget is typically a constraint in practical situations. To improve the efficiency of simulation, our objective is to maximize P{CS} under a limited number of simulation replications T by allocating T intelligently to each design, which is defined as follows: Here, the constraint k i=1 N i = T implicitly supposes that the simulation cost of each replication across designs is similar. Otherwise, this can be roughly considered in (3) by reflecting the proportion of the increasing simulation cost for a replication of design in its variance [8].

III. PROPOSED PROCEDURE A. UNCERTAINTY EVALUATION
For the selected best design x b , the following k − 1 relationships between µ b and µ i depending on (2) should be verified for the correct selection of x b : On the other hand, for the selected worst design x w , the following k − 1 relationships between µ w and µ i depending on (2) should be verified for the correct selection of x w : For the remaining k − 2 designs, which are not selected as x b and x w , the following two relationships depending on (2) should be verified for the correct selection of each x i : As in the other R&S procedures, our proposed procedure also allocates a few further replications based on the current simulation results, such asμ i and s 2 i , to maximize P{CS}. If the relationships for the correct selection of design can be verified with the current simulation results, it is no longer necessary to collect simulation output samples for that design. Otherwise, if we cannot determine whether the selection for design is correct based on the current results, it is necessary to allocate further replications and collect more samples. The concept of uncertainty in the UE approach suggests an allocation criterion based on a statistical hypothesis test and p-value to accomplish this allocation strategy [15]. Utilizing the UE approach, our proposed procedure defines the uncertainty of each design and allocates further replications sequentially depending on the uncertainty to maximize P{CS}.
The statistical hypothesis test is a frequentist method to verify the single relationship between µ i and µ j based on simulation results. For example, when verifying µ i < µ j , this relationship is set to the alternative hypothesis H A , and the opposite relationship is set to the null hypothesis H 0 , as follows: Then, the p-value of the test p i,j can be calculated as follows: The function F v is the cumulative distribution function of the t-distribution with v degrees of freedom. (If σ 2 i and σ 2 j are known, then the normal distribution can be used instead of the t-distribution.) Mathematically, p i,j is the probability of obtaining the current simulation results or more extreme results (which are less likely to be obtained) when assuming that H 0 is true. However, in the framework of the hypothesis test, it can be interpreted as the degree to which the current results can be considered significant evidence to verify µ i < µ j . The lower the value of p i,j , the less likely it is that the current results will be obtained under the assumption that H 0 is true; however, the current results have been actually observed, so they can be significant evidence to verify On the other hand, as the value of p i,j increases, obtaining the current results becomes more probable under this assumption, so the obtained results cannot be significant evidence. That is, it is uncertain that H A will be verified based on the current results (but, this does not mean that H A is false). Although the possible range of p i,j is between 0 and 1, the meaningful range in the hypothesis test framework is between 0 and 0.5. This is because 1 − p i,j is the same as p j,i , which is the p-value for verifying the opposite of (7). Thus, p i,j = p j,i = 0.5 indicates the most uncertain case in which it is impossible to determine whether µ i < µ j or µ i > µ j based on the current simulation results [24].
If there is only one relationship (e.g., µ i < µ j ) to be verified for the correct selection of design x i , then p i,j can be the allocation criterion for the mentioned allocation strategy. That is, the higher value of p i,j , the more insufficient the simulation results of x i , so more output samples of x i should be collected by allocating further replications. However, as shown in (4)- (6), several relationships should be verified simultaneously for the correct selection of each design in this problem. In this case, the uncertainty of each design defined as the combination of p-values for verifying each relationship can be the allocation criterion. For the selected best design x b , each of the k −1 relationships in (4) can be verified by the hypothesis tests, and the p-value from each test is denoted by p b,i , where i = b. Each value of p b,i can be used to assess the degree to which the current simulation results of x b can be significant evidence for each relationship in (4). If the results of x b are not evidence for verifying even one relationship in (4), the results cannot be evidence for the correct selection of x b , and the selection becomes uncertain; thus, the uncertainty of x b , δ b can be defined as the maximum value among p b,i s, as follows: As the value of δ b decreases to zero, the current simulation results of x b can be significant evidence for the selection of x b ; thus, it is no longer necessary to allocate further replications to x b . However, as the value of δ b increases to 0.5, at least one relationship in (4) is uncertain. Accordingly, we cannot determine whether the selection of x b is correct based on the current results of x b . Therefore, it is necessary to allocate further replications to x b and update the insufficient results.
Similarly, for the selected worst design x w , each of the k − 1 relationships in (5) can be verified by the hypothesis tests, and the p-value from each test is denoted by p i,w , where i = w. In common with x b , the current simulation results of x w should be significant evidence for verifying every relationship in (5) for the correct selection of x w , so the uncertainty of x w , δ w can be defined as the maximum value among p i,w s, as follows: max p 1,w , . . . , p i,w , . . . , p b,w where i = w. (10) For each x i (i = b and i = w) of the remaining k − 2 designs that are not selected as x b or x w , the two relationships in (6) can be verified by hypothesis tests. p-values from both tests are denoted as p b,i and p i,w . The current simulation results of x i should be significant evidence verifying the two relationships in (6) for the correct selection of x i . Therefore, the uncertainty of x i , δ i can be defined as the maximum value between p b,i and p i,w , as follows: Strictly speaking, the uncertainty of each design in (9)-(11) includes the significance degree of not only this design's simulation results, but also another design's results. This is because the p-value in (8) is calculated using the two designs' results. However, if the estimated uncertainty is not very large or has specific extreme cases, it does not have a significant impact on allocation [25]. In addition, our objective is to develop a simple and practical means for allocating simulation budget to significantly improve the efficiency of simulation, as demonstrated in the experimental results later; thus, we neglect this problem in the proposed procedure.

B. ALLOCATION POLICY
To maximize P{CS}, the simulation results of all designs should be significant evidence verifying the relationships for the correct selection of each design shown in (4)- (6). To this end, more replications should be allocated to the designs of which the uncertainty has a relatively high value depending on the features of uncertainty. However, due to the type I error included in the uncertainty, a low uncertainty value also needs to be considered in the allocation. For example, suppose x 1 is the actual best design we must find, but unfortunately, after collecting some samples, the current simulation results of x 1 are very poor due to large stochastic noise. That is, x 1 is not selected as x b , andμ 1 is far from bothμ b andμ w (i.e., µ b μ 1 μ w ), so the uncertainty of x 1 , δ 1 has low value according to (11). Although x 1 requires updating of its poor simulation results more than any other design, it cannot be allocated further replications due to its lower uncertainty. This type I error results in a reduction in P{CS}. Since the simulation results in type I error are typically very biased, just one or two additional samples are sufficient to check and eliminate the error. In other words, it is necessary to allocate a few additional replications to the designs of which the uncertainty has a relatively low value to increase P{CS}. Accordingly, a heuristic allocation policy to accomplish the mentioned allocation strategy based on uncertainty can be defined as follows [15]: Given the number of further assignable replications , a i is the number of further allocated replications at x i depending on its uncertainty δ i . In the policy of (12), the coefficient C is a parameter that can adjust allocation depending on uncertainty, where the range of C is between zero and infinity. As the value of C increases, the difference of uncertainty is emphasized, so is concentrated on a few designs that have relatively higher uncertainty. On the other hand, as the value of C decreases to zero, the difference of uncertainty is thinned, so the designs with relatively low uncertainty are more likely to be allocated a few further replications. The optimal value of C that maximizes P{CS} is problemspecific. A high value can be effective in small-noise environments where a type I error is less likely to occur due to the relatively precise simulation results. However, in large noise environments where a type I error is more likely to occur due to inaccurate simulation results, a low value can be better.
In this paper, the default value of C is recommended as one. Although this value is not always optimal, it exhibits sufficiently good efficiency compared to the other R&S procedures, as shown in Table 2 and Fig. 1. In addition, if we consider selecting just a single extreme design such as the best one, the defined uncertainty in (9) and (11) becomes identical to that of the UE procedure finding the single best design. The UE procedure utilizes the allocation policy of (12) with C = 1 derived from the simulation-based optimizations; thus, this consistency gives a basis for this recommendation. In addition, the consistency can be a potential advantage of the proposed procedure over the OCBAbw procedure, which becomes identical to the OCBA procedure when considering a single extreme design. This is because the UE procedure has been proven empirically superior to the OCBA procedure in the presence of large noise [15].

C. SEQUENTIAL PROCEDURE
Algorithm 1 represents the proposed procedure for selecting both extreme designs simultaneously using the defined uncertainty of (9)-(11) and the allocation policy of (12). Prior to the iterative allocation of , the proposed procedure collects n 0 output samples for each design to obtain minimal simulation results to evaluate the uncertainty. Then, until the given simulation budget T is exhausted, the procedure iterates 1) evaluating the uncertainty of each design, 2) allocating a few further replications of depending on the policy, and 3) updating the simulation results. As the allocation repeats, the simulation results for each design are selectively and gradually updated to become significant evidence for their correct selections. Meanwhile, the first evaluations of δ b and δ w in the seventh line of Algorithm 1 decrease the computational cost

Algorithm 1 Select Both Extreme Designs Simultaneously
Control parameters: T (≥ kn 0 ) , n 0 , , and C(default value is 1) Procedure: 1: simulate n 0 times for each x i , i ∈ {1, . . . ,k} 2: updateμ i , s i , and N i for ∀i ∈ {1, . . . ,k} 3: select x b and x w with (2) 4: : evaluate δ b and δ w for x b and x w with (9) and (10) 7: evaluate δ i for ∀i ∈ {1, . . . ,k}, i = b, and i = w with (11) 8: calculate a i for ∀i ∈ {1, . . . ,k} with (12) 9 a : simulate round(a i ) times for each x i , i ∈ {1, . . . ,k} 10: updateμ i , s i , and N i for ∀i ∈ {1, . . . ,k} 11: select x b and x w with (2) 12: end while 13: return x b and x w a Due to the robustness of the sequential procedure, any reasonable method for converting a real number a i to an integer has not a significant impact on P{CS}. In Algorithm 1, we used the round function and scattered the rounding errors to several designs that have relatively larger a i . 93982 VOLUME 8, 2020 of the proposed procedure. This is because the uncertainty of the remaining k −2 designs in the eighth line can be evaluated at no extra cost using p b,i and p i,w calculated in the previous line.
The sequential allocation procedure is controlled by two parameters: n 0 (the initial number of replications) and (the number of further replications to be allocated per iteration). An optimal setting of n 0 and is problem-specific. However, a recommended choice of n 0 is between 5 and 20 [26]. The recommended value is between 0.5k and 0.05T [15] (see [15] for a detailed analysis of the UE approach according to n 0 and ). Meanwhile, a larger setting of n 0 and than recommended might be effective in the presence of large stochastic noise. Increasing n 0 can decrease the possibility of type I errors by improving the initial simulation results. Increasing raises the likelihood that one or two further replications will be allocated to designs with relatively low uncertainty, thereby checking and reducing the errors.

IV. EXPERIMENTS
In this section, we describe the experimental results on both numerical benchmark and practical problems that demonstrate the necessity of the proposed procedure. Table 1 presents the benchmark problems, which have been used to evaluate various R&S procedures, including the OCBAbw procedure [8], [21]. All five problems have 10 designs with increasing µ i according to the design index i, of which x 1 and x 10 are both extreme designs that we have to select. The designs in the equal variance (EV) problem have the same variance, whereas in the increasing variance (IV) and decreasing variance (DV) they have different variances that increase or decrease linearly according to i. While µ i in these three problems increases linearly, µ i in the flatand steep-case problems increases concavely and convexly, respectively. Each problem has a large noise version denoted by L, which has relatively larger variance than its original version. Thus, the experimental results for the L version may well indicate the effectiveness of the proposed procedure. For these five benchmarks, we compared the efficiency of the proposed procedure with other procedures: OCBAbw, hOCBA, hUE, PTV, and the equal allocation procedures. The hOCBA and hUE procedures are variants of the OCBA and UE procedures that find only the best design. They use the OCBA or UE procedures twice to find both extreme designs.

A. BENCHMARK PROBLEMS
That is, in each iteration of the sequential procedure, half of the given is used to find the best design, and the other half is used to find the worst one. Meanwhile, since the OCBAmn procedure is reduced to the OCBAbw procedure when m = n = 1, it was not considered in this experiment. For a fair comparison, n 0 and in every procedure were set to 10 and 20 according to the typical settings of the five problems [8]. We estimated P{CS} for each procedure with 10,000 independent repeated experiments while varying T . The several results for the L version of the problems are shown in Fig. 1. In addition, for numerical comparisons, the simulation budget required to find both extreme designs correctly (i.e., to achieve P {CS} of 0.99) is measured in Table 2. Our experimental results indicate the superior efficiency of the proposed procedure over other procedures. The results of the hUE and hOCBA procedures represent why a procedure for selecting both extreme designs simultaneously is necessary. Except for the EV problem, the simulation budget for selecting the best and worst designs is not the same due to the inconsistent µ i spacing (the flat-and steep-case problems) or the different variance (the IV and DV problems), which is a common case in practice. As shown in Table 2, compared to the results of the EV problem, the proposed procedure is more efficient than both procedures in these practical cases. For example, in the EV problem, the hUE and hOCBA procedures consumed 1.23 times and 1.30 times more replications than the proposed procedure, respectively; however, in the IV problem, these numbers increased significantly to 1.86 and 1.87, respectively.
In addition, the results of the OCBAbw procedure, especially in the L version of the problems, emphasize the high robustness against noise of the proposed procedure. VOLUME 8, 2020 For example, in the flat case problem, while the OCBAbw procedure consumed 1.09 times more replications than the proposed procedure, the number increased to 1.17 in the '(L)' version, as shown in Table 2. Similar trends were observed in the other problems. Compared to the OCBAbw procedure, the higher robustness of the proposed procedure is attributed to considering additional information when allocating further replications. Although both procedures take into account the sample meanμ i and the sample variance s 2 i , the proposed procedure considers the number of collected samples N i as well, as shown in (8). However, the OCBAbw procedure uses onlyμ i and s 2 i as specified in its main allocation rule below [21]: Here, N * i is the optimal number of replications that should be assigned to x i to maximize a lower bound of P{CS} based on the current simulation results. That is, N * i is different from N i , the number of actually allocated replications so far, and the further replications actually allocated to x i is defined as max 0, N * i − N i in the OCBAbw procedure. Simply put, while the proposed procedure takes into account the precision of the sample mean s i √ N i when allocating , the OCBAbw procedure cannot consider it (s 2 i itself is not enough to indicate the precision ofμ i ). Accordingly, the OCBAbw procedure is likely to make an inefficient allocation due to an inaccurateμ i value, which often occurs in the presence of large stochastic noise.
For example, in the EV problem, suppose the observedμ 1 of the true best design x 1 is unfortunately high (i.e., close to 5). Then, the OCBAbw procedure cannot allocate further replications to x 1 due to the inaccurate value ofμ 1 . Sinceμ 1 cannot be updated, the situation in which no further replications are allocated to x 1 continues. If T is infinite, x 1 will be allocated further replications depending on changes in the simulation results of the other designs and will be selected as x b eventually. However, numerous replications are wasted on other marginal designs in the meantime. On the other hand, our proposed procedure, which considers the precision ofμ 1 , can allocate further replications quickly to x 1 because the precision ofμ 1 that is not updated becomes relatively lower. Thereby, it can select x 1 as x b efficiently. Meanwhile, if the noise level is small, considering N i may reduce efficiency. However, this reduction is insignificant since the required T for the correct selection is small in this case. In addition, most complex practical problems have large stochastic noise. Thus, the proposed procedure can be effective, as shown in the next subsection.

B. PRACTICAL PROBLEM: MUNICIPAL WASTE MANAGEMENT IN A CITY
As urbanization intensifies in modern society, municipal waste management in dense residential environments has become a social problem. As it is directly related to the satisfaction and hygiene of residents, many complaints are generated in this regard, which wastes administrative power. The government allocates garbage trucks depending on the number of residents and periodically collects municipal waste; however, various variables, such as regional characteristics, residential patterns, local developments, etc., make it difficult to manage effectively. A studio apartment complex in South Korea is a representative place where many complaints against municipal waste management arise because inexpensive rent often changes the residential patterns. To reduce complaints, discrete-event simulations can be effectively used for preliminary analysis of a given management policy [27]. Given 30 scenario designs that reflect the possible residential patterns, practitioners are interested in not only the best design, but also the worst design under which the number of complaints is maximized. That is because even if the number of complaints in the best-case scenario is very small, the evaluation of this policy should be reconsidered if its worst-case scenario frequently occurs. Our proposed procedure can be effectively applied to such an analysis of municipal waste management policy based on the simulation.
As shown in Fig. 2, the relatively large σ i compared to the small difference in µ i between both extreme designs and their neighborhood designs acts as a large stochastic noise making it difficult to find the extreme designs. To demonstrate the necessity of the proposed procedure for such a practical problem, other procedures used in the benchmark experiments were applied together, and Fig. 3 represents the results. While the proposed procedure required only 5,880 replications to correctly find the best and worst designs (i.e., to achieve P{CS} of 0.99), the OCBAbw procedure required 8,760, and the hUE and hOCBA procedures required 7,890 and 12,450, respectively. With no R&S procedures applied, 50,700 replications were required. This simulation model is expensive per simulation run (i.e., about 1 min.) due to the interactions between many agents; thus, applying the proposed procedure can greatly reduce the time needed to analyze the municipal waste management policy by enhancing the simulation's efficiency.

FIGURE 2.
The precisely estimated value of µ i (i.e., the number of complaints) and σ i for 30 given designs using many repeated simulations [27]. The two dark gray bars represent the results of both extreme designs (i.e., x 16 and x 30 ) that we have to select. Like this practical problem, the proposed procedure is effective in problems where the simulation model has large stochastic noise and high evaluation costs. A digital twin, drawing attention in the era of the fourth industrial revolution, is an example that meets these conditions. The digital twin is a simulation model in cyberspace corresponding to a system in physical space. It is continuously updated to maintain synchronization with the real system, and the best and worst control settings should be found quickly based on the updated model to optimize the system. In this case, the superior efficiency of the proposed procedure can help the use of digital twins, especially for systems that demand high reliability.

V. CONCLUSION
Finding both extreme input designs has tremendous potential in discrete-event simulation as well as in other areas. In this paper, we proposed an efficient R&S procedure for selecting the best and worst designs simultaneously from a finite set of design alternatives under large stochastic noise. To maximize P{CS} in the constraint of simulation budget, we described evidence of correct selection using uncertainty based on the statistical hypothesis test and p-values. Then, we proposed a procedure that sequentially allocates a few further simulation replications depending on the evaluated uncertainty, such that the simulation results can provide significant evidence for the correct selection. The proposed procedure is superior to using basic R&S procedures (e.g., OCBA or UE procedures) twice because it efficiently distributes the budget to both extreme designs. Furthermore, since it additionally considers the number of allocated replications so far in allocation, it is more efficient than the OCBAbw procedure in the presence of large stochastic noise. The experimental results on benchmark problems demonstrate this, and the municipal waste management problem emphasizes the necessity of the proposed procedure for practical simulation problems. As mentioned earlier, discrete-event simulation is used in a variety of engineering fields, and the philosophy of selecting both extreme designs simultaneously also has tremendous potential for application. As a future study, we will apply and customize the proposed procedure for improving the efficiency of learning automata [28], network resource allocation strategies [29], and digital twins.