By Topic

IEEE Quick Preview
  • Abstract
Development of heatmap of feasible region with cp lines for a representative problem.



TRUE multiobjective optimization, where multiple objectives are optimized simultaneously, seeks a set of solutions that minimize (or maximize) multiple competing objectives, in the sense that each solution in the set outperforms all other solutions in at least one objective (i.e., are non-dominated). The resulting solution set forms what is called the non-dominated front, sometimes referred to as the Pareto front if it is truly an optimal set [1].

Deb and Srinivasan [2] introduced the term innovization to refer to the process of examining patterns in decision variables along the non-dominated front to identify fundamental design principles, thus deepening the understanding of a class of similar problems. “Such… principles… should provide a reliable procedure of arriving at a ‘blue-print’ or a ‘recipe’ for solving the problem in an optimal manner” [2]. This recipe can also be used to better inform the optimization process for related problems. Whereas prior work focused on finding principles from the non-dominated front exists (e.g., [3], [4]), this was the first time a general set of steps was proposed. The innovization process proceeds by finding an evenly-spaced set of points along the non-dominated front, using a combination of single-objective optimization, multiobjective evolutionary optimization, and local search. The normal constraint method [5], a numerical optimization method, is then used to verify that the obtained front reasonably reflects the true Pareto front. The resultant non-dominated front is analyzed to extract design principles. Deb and Srinivasan [2] applied their method to find design principles in several problems. For example, they found that: (i) in a two-member truss design problem, all Pareto-optimal solutions have equal stress on both truss members and also have a constant product of the maximum stress on the truss and the volume of the truss members; (ii) in a multiple-disk clutch brake design problem, increasing the number of disks monotonically improves stopping action while increasing mass, and all Pareto-optimal solutions have the same disk thickness and the same actuating force applied; (iii) in a spring design problem, all Pareto-optimal solutions have the same spring stiffness; and (iv) in a welded beam design problem, the thickness of the beam remains constant over most of the non-dominated front while the shear strength of the material is the limiting factor in improving a solution (all solutions on the non-dominated front have the maximum shear stress allowed).

Other methods for extracting design principles from non-dominated fronts have also been proposed. Obayashi and Sasaki [4], Chiba, et al. [6], and Doncieux and Hamdaoui [7] all used Kohonen self-organizing maps (SOMs) to extract design solutions from the non-dominated front. SOMs are unsupervised neural networks that spatially separate multidimensional data into groups that share similar characteristics, with closely related groups arranged next to each other [8]. SOMs automatically cluster data with similar characteristics into groups (i.e., clustering), producing taxonomies of solutions, thus finding semantic patterns in unordered data. Obayashi and Sasaki [4] successfully used SOMs to find patterns in the multiobjective solutions to two supersonic transport aerodynamic problems, classifying both wing and fuselage designs. The SOM was also used to classify the impact of the 131 different design variables on the objectives, to find which ones had the greatest impact on overall performance. Similarly, Chiba, et al. [6] used a SOM to find which design variables had the greatest impact on wing design. Doncieux and Hamdaoui [7] employed a SOM in the design of a flapping wing aircraft to identify design variables that significantly affected the velocity of the aircraft.

Ulrich, et al. [9] used dendrograms to hierarchically cluster non-dominated solutions and then discover design principles. They developed a method to build the dendrograms from a dataset and then applied their method to the design of a network processor. Dendrograms are binary trees that represent organizational structures. The left branch of each node has solutions that include the parent node's characteristics while the right branch has solutions that do not include the parent's characteristics. The leaves of the tree are the solutions themselves and nodes physically near each other generally share common traits. The authors also vertically arranged the dendrogram nodes based on the order in which the characteristics were selected, so that no two nodes had the same distance from the root.

Askar and Tiwari [10] presented an analytic approach for discovering the Pareto front that leads directly to optimal design principles. They first establish the boundaries of the decision space, defined by all constraints in the system. The boundary points are connected with straight lines to define a region of feasible decision variables. The (linear) equations for these lines are found in terms of the decision variables. Several additional lines through the middle of the decision space are added to divide this region into smaller pieces and their (linear) equations are found. Values along all of these lines are substituted into the objective functions and dominated solutions are thrown out. Additional lines through the region are added until the non-dominated front is well-represented. The lines that lead to solutions on the non-dominated front are then used to simplify the objective functions and the constraint functions, and the resulting functions are examined to find underlying design principles. The method is demonstrated on both a welded beam design problem and a subwatershed stormwater drainage problem.

Brownlee and Wright [11] highlighted solutions on the non-dominated front in an Excel spreadsheet to more easily discern patterns. They used a translucent bar graph in each cell to show the relative scale of each design variable, but reported that they found it difficult to find patterns among the decision variables.

Kudo and Yoshikawa [12] used isometric feature mapping (isomap) to extract design principles in a hybrid rocket design problem. Like SOMs, isomap reduces high-dimensional data into low-dimensional semantically-organized spatial patterns. For non-linear data sets, isomap works better than either principal component analysis or multidimensional scaling [13]. Kudo and Yoshikawa [12] created an isomap that related the design parameters along the front to the resultant objective values to help map design space to objective space and find patterns in the design variables that led to different objective outcomes.

Ulrich [14] used a biobjective evolutionary algorithm to partition both the decision space and the non-dominated front into related clusters. The algorithm, Pareto-Front Analyzer (PAN), locally optimizes every evolved solution before selection and has specialized recombination and selection operators for populations of partitions. The method successfully clustered similar designs in a truss bridge design problem.

Bandaru and Deb [15], [16], [17] developed an automated method to extract design principles from non-dominated fronts. This method creates mathematical rules that relate the decision variables, objective function values, and problem constraints. For example, in the two-member truss design problem they found that Formula$S\!\cdot\!V$ is constant along the non-dominated front, where Formula$S$ is the maximum stress on the truss and Formula$V$ is the volume of the truss. Similarly, in the welded beam design problem they found that Formula$d\cdot L^{0.333}$ is constant along the non-dominated front, where Formula$d$ is the deflection at the end of the beam and Formula$L$ is the buckling load of the beam. This method depends on all solutions being very close to the true Pareto front. Bandaru and Deb [18] later extended their automated methods to extract principles over pre-specified regions of the non-dominated front (called low-level innovization) and also across several non-dominated fronts discovered using different problem parameters (called high-level innovization).

While the methods using clustering techniques to extract design principles from dominated solutions can be applied to problems with more objectives [4], [6], [7], those that rely on visualization techniques [2], [9], [10], [11], [12], [14] are generally only applied to biobjective problems, due to difficulties in visualizing higher dimensional spaces.

All of these prior innovization approaches have focused on extracting information from solutions along the non-dominated front. However, to our knowledge, no attention has been given to discovering design principles from the wealth of additional information available from patterns in dominated solutions. In this paper, we introduce visualization approaches to help identify patterns in dominated solutions across the fitness landscape of biobjective design problems. (One could conceivably use the method described here to visualize selected two dimensional (2D) slices of higher dimensional solution spaces in problems with more than two objectives, but here we limit our demonstration to biojective problems.) Dominated solutions that cover the entire feasible region give a much broader view of the impact of decision variables on the objectives than do solutions on the non-dominated front alone, leading to greater confidence that the derived design principles are truly fundamental and not just artifacts of the front. Our results illustrate how patterns along the non-dominated front can sometimes be misleading, how information present in dominated solutions can lead to additional insights and design principles that cannot be determined from the non-dominated front alone, and how visualizing sensitivity of variables across the feasible region can help one to identify solutions that are more robust to uncertain external forcing conditions.



In this section we describe our approaches to visualizing patterns in dominated solutions of biobjective problems, followed by a description of the three design problems we will use to illustrate these approaches.

A. Visualization Approaches

To visualize the space of dominated solutions, we follow these general steps:

  1. Generate a variety of solutions throughout the feasible region as described in Section II-A-I.
  2. For particular variables of interest (e.g., design variables, constraints, aggregate metrics, and sensitivities of variables to changes in external forcing conditions):
    1. Generate a heatmap of the variable over the set of obtained solutions as described in Section II-A-II.
    2. For design variables, overlay ceteris paribus (cp) lines on the heatmap as described in Section II-A-III.
    3. Examine the heatmaps and cp lines to discover meaningful patterns within the fitness landscape.

1) Populating the Feasible Region

To plot a heatmap of the feasible region, a sufficient number of potential solutions (both non-dominated and dominated) must be found. In some cases, simply saving all intermediate solutions from a biobjective evolutionary run may provide a sufficient density of solutions to answer the questions at hand, especially if one only needs to examine patterns in dominated solutions that are relatively close to the non-dominated front. However, to obtain a more fully populated feasible region, we do the following:

  1. Starting from random sets of solutions, evolve towards the non-dominated front several times, saving all solutions from intermediate generations (Fig. 1(a)).
  2. Starting from random sets of solutions, reverse the objectives (i.e., if it's a minimization problem, turn it into a maximization problem and vice-versa) and evolve towards increasingly dominated solutions several times, saving all solutions from intermediate generations (Fig. 1(b)). Note this finds the most dominated front, i.e., pessimal solutions, and thus defines the bounds of the feasible region.
  3. Generate additional random solutions as many times as needed to fill out the region between the non-dominated and fully-dominated fronts. If necessary, evolve these solutions a few generations in either direction until the entire region is adequately sampled, saving all intermediate solutions. We define a “cell” as a square unit of area in the 2D solution space over which we average a variable of interest and display the averaged value as one pixel in a heatmap; the user chooses the cell size to obtain the desired level of resolution of the heatmap display. In this work, we generated new solutions until we had at least 10 solutions for each cell whose center lay within the region bounded by the non-dominated front and the fully-dominated front.
  4. Remove infeasible solutions (i.e., those that do not meet the problem constraints) and duplicate solutions from the collected set of all solutions.
  5. Check the density of cells and reapply steps 3 and 4 until the entire region is adequately sampled (Fig. 1(c)).
Figure 1
Fig. 1. Development of heatmap of feasible region with cp lines for a representative problem. (a) Evolve toward the non-dominated front (bottom and left sides) multiple times, saving all intermediate solutions, (b) Reverse objectives and evolve toward fully dominated front (top and right sides) multiple times, saving all intermediate solutions, (c) Add random solutions until a desired density of solutions is achieved across the feasible region, (d) Evaluate a variable of interest (here, a design variable) in solutions across the feasible region, (e) Create a 2D moving average across the feasible region and display as a heatmap, and (f) For heatmaps of design variables, identify points (marked with black asterisks) to calculate cp lines from, and compute and overlay the cp lines (shown in white) on the heatmap.

For the evolutionary steps above, any biobjective evolutionary algorithm can be applied. For this work we used Uniform Spacing Multiobjective Differential Evolution (USMDE) [19], with scaling parameter Formula$F=0.6$ and crossover rate Formula$Cr=0.2$. Because USMDE is designed to explicitly encourage uniformity of spacing in evolving populations of solutions, it helps one to obtain more spatially uniform coverage of solutions in the feasible region of the solution space in the steps above. Constraints were handled using Constraint Adaptation with Differential Evolution (CADE) [20], [21].

2) Creating Heatmaps

It is difficult to directly identify patterns in these solutions due to heterogeneities in nearby solutions (Fig. 1(d)). Thus, we create 2D heatmaps of locally averaged solutions to permit visualization of patterns in the self-organized design parameters (or other derived or aggregated variables) across the feasible solution set. We first divide the fitness landscape into a 2D grid with fixed-sized cells at the pre-specified level of spatial resolution. We then calculate a matrix Formula$(M)$ whose elements contain moving averages of associated overlapping cells of the variable in question across the fitness landscape, at the pre-specified level of spatial resolution. Thus, each element Formula$M_{i,j}$ corresponds to a cell in the grid with center Formula$(x,y)$. Given the radius Formula$r$ of the moving average window, each Formula$M_{i,j}$ is set to the average of the variable of interest values determined for every solution in the fitness landscape that is bounded by the rectangle described by the corners Formula$(x-r,y-r)$ and Formula$(x+r,y+r)$. Elements in Formula$M$ that correspond to cells whose centers lie outside the feasible region (i.e., beyond either the non-dominated front or the fully-dominated front) are marked as infeasible. We then display the feasible region of Formula$M$ using a pseudo-color plot (although any 3D visualization technique could be used) as shown in Fig. 1(e). One can also use this same approach to generate and display a matrix of moving standard deviations to assess the variability of values in the cells. Such variability maps can be helpful in determining the appropriate cell size as well as to assess how smoothly the values of specific variables change between nearby solutions in various regions of the solution space.

3) Creating the CP Lines

Ceteris paribus (cp) lines show where solutions will move in the feasible region when exactly one of the design variables is changed. These are used to ascertain whether the heatmap contours of each decision variable are sufficient for inferring how changing that single decision variable will affect the quality of the solution in various regions of the solution space, or whether the additional information provided by the cp lines is necessary for inferring design principles. The cp lines are generated as follows:

  1. From the moving-average matrix Formula$M$ of the variable in question, identify several places of interest to use as starting points. For example, one might chose starting points that are evenly spaced along or near the non-dominated front (see asterisks on Fig. 1(f)).
  2. For each of these points:
    1. Extract the set of design variables from one solution inside the cell associated with that point.
    2. Holding the other variables constant, re-evaluate both objectives at evenly-spaced intervals across the allowable range of the variable of interest.
    3. Plot the resulting curve on top of the heatmap, showing how the objectives change with the variable of interest (see lines on Fig. 1(f)).

B. Visualizing Robustness to Uncertain Forcing Conditions

When external forcing conditions are uncertain, it is desirable to identify solutions that not only balance the two primary objectives, but also maximize robustness to these uncertain forcing conditions. Whereas one could conceivably add a third robustness objective to the original optimization problem, adding this third objective would not only significantly increase the computational burden of the evolutionary optimization but, more importantly, would reduce the selection pressure among competing solutions (and thus make it more difficult to find the Pareto optimal solutions) and would require many more solutions to obtain adequate density and spacing along a 3D non-dominated surface. Instead, we propose to create a set of 2D solutions as in Section II-A-I with respect to the two primary objectives under one estimate of the forcing conditions, and then do the following:

  1. Re-evaluate all identified feasible solutions under one or more alternative estimate(s) of forcing conditions.
  2. For each solution and for each performance objective of interest, find the maximum difference for that objective between these sets of forcing conditions.
  3. For each objective of interest, generate a heatmap of the differences (i.e., a sensitivity map).

C. Test Problems Used

Three biobjective optimization problems illustrate the benefits of the proposed visualization approach: (i) the design of a simple two-member truss, (ii) the design of a simple welded support beam, and (iii) a watershed management design application. The first two problems were selected because they have been used previously to demonstrate innovization from the non-dominated front [2] and are easily understood, with only four design parameters each. The third problem is a complex design problem that we have previously formulated as a biobjective optimization problem [22], [23], and is used here to illustrate how our visualization approaches can provide valuable design insights for a real-world problem.

1) Two-Member Truss Design

This biobjective problem was originally studied in Chankong and Haimes [24] and is one of the problems analyzed by Deb and Srinivasan [2]. The problem is to design a truss with two beams at minimal cost that can carry the specified load of 100 kN without elastic failure. The volume of the truss members, which is linearly related to the cost of the truss, is used as a proxy for cost. The maximum stress developed on either member is minimized to avoid elastic failure. The original problem has three design variables (Fig. 2): Formula$A_{AC}$, the cross-sectional area of truss member AC Formula$({\rm m}^{2})$; Formula$A_{BC}$, the cross-sectional area of truss member BC Formula$({\rm m}^{2})$; and Formula$y$, the vertical distance from the support structure (at A or B) to where the members join at C (m). To better demonstrate our approach, we added a fourth design variable, Formula$x_{BC}$, the horizontal distance from B to C (originally fixed at 1 m in Chankong and Haimes [24]).

Figure 2
Fig. 2. Two-member truss problem [after 24].

The stress on each truss member AC and BC, named Formula$\sigma_{AC}$ and Formula$\sigma_{BC}$, respectively, is: Formula TeX Source $$\eqalignno{\sigma_{AC}=&\,{{20x_{BC}\sqrt{x_{AC}^{2}+y^{2}}}\over{yA_{AC}}}&{\hbox{(1)}}\cr\sigma_{BC}=&\,{{20x_{AC}\sqrt{x_{BC}^{2}+y^{2}}}\over{yA_{BC}}}&{\hbox{(2)}}}$$ where Formula$x_{AC}$ is the horizontal distance from A to C: Formula TeX Source $$x_{AC}=5-x_{BC}\eqno{\hbox{(3)}}$$

The optimization problem is to minimize both cost (volume) of the truss and the maximum stress on the truss: Formula TeX Source $$\eqalignno{cost(A_{AC}, A_{BC}, y, x_{BC})=&\, A_{AC}\sqrt{x_{AC}^{2}+y^{2}}\cr&+A_{BC}\sqrt{x_{BC}^{2}+y^{2}}&{\hbox{(4)}}\cr{max{\_}stress}(A_{AC}, A_{BC},y, x_{BC})=&\,\max (\sigma_{AC},\sigma_{BC})&{\hbox{(5)}}}$$ subject to the following constraints: Formula TeX Source $$\eqalignno{& max{\_}stress\leq 10 {\rm MPa}\cr& 0\leq A_{AC},A_{BC}\leq 0.01 {\rm m}^{2}\cr& 0\leq y\leq 3 {\rm m}\cr& 0\leq x_{BC}\leq 2.5 {\rm m}.&{\hbox{(6)}}}$$

2) Welded Beam Design

Deb and Srinivasan [2] present a biobjective problem of one beam being welded to another to carry a load Formula$F$ (6000 lb) applied at the end of the beam. Both the cost of the beam and the vertical deflection at the end of the beam must be minimized. There are four design variables (see Fig. 3): Formula$t$, the width (or height) of the beam (in); Formula$b$, the thickness of the beam (in); Formula$h$, the height (or thickness) of the weld (in); and len, the length of the weld (in). The length of the beam is Formula$14+len$ inches.

Figure 3
Fig. 3. Welded beam problem [after 2].

The shear stress on the beam, Formula$\tau$, in psi, is defined by the following set of equations: Formula TeX Source $$\eqalignno{\tau=&\,\sqrt{\tau_{1}^{2}+\tau_{2}^{2}+{{\tau_{1}\tau_{2}len}\over{\sqrt{0.25(len^{2}+(h+t)^{2})}}}}\cr\tau_{1}=&\,{{6000}\over{\sqrt{2}h\cdot len}}\cr\tau_{2}=&\,{{6000(14+0.5len)\sqrt{0.25(len^{2}+(h+t)^{2})}}\over{\sqrt{2}h\cdot len\left({{len^{2}}\over{12}}+0.25(h+t)^{2}\right)}}&{\hbox{(7)}}}$$

The bending stress on the beam, Formula$\sigma$, in psi, and the buckling load on the beam, Formula$L$, in lb, are: Formula TeX Source $$\eqalignno{\sigma=&\,{{504000}\over{t^{2}b}}&{\hbox{(8)}}\cr L=&\,64746.022(1-0.0282346t)tb^{3}&{\hbox{(9)}}}$$

The optimization problem is to minimize both cost for the welded beam and the vertical deflection at the end of the beam: Formula TeX Source $$\eqalignno{cost(b, t, h, len)=&\, 1.10471h^{2}len\cr&+0.04811tb(14+len)&{\hbox{(10)}}\cr deflection(b, t, h, len)=&\,{{2.1952}\over{t^{3}b}}&{\hbox{(11)}}}$$ subject to the following constraints: Formula TeX Source $$\eqalignno{\tau\leq 13600 {\rm psi}&\cr\leq30000 {\rm psi}\cr b-h\geq 0&\cr L\geq 6000 {\rm lb}&\cr 0.1\leq t, len\leq 10 {\rm in}&\cr 0.125\leq b, h\leq 5 {\rm in}.&&{\hbox{(12)}}}$$

3) Watershed Management Plan Design

Increases in impervious area and decreases in vegetation due to land use development for residential, commercial, industrial, and agricultural purposes can cause large increases in stormwater runoff, resulting in increased erosion and transport of sediment and associated soil contaminants into surface water bodies. To mitigate problems caused by non-point source impacts from developed lands, Best Management Practices (BMPs) such as detention ponds and rain gardens can be installed to reduce peak storm flows and remove pollutants from stormwater runoff. Some important differences between rain gardens and detention ponds are that (a) rain gardens infiltrate, whereas detention ponds do not, (b) rain gardens are more expensive per unit area than detention ponds, (c) detention ponds have much larger minimum size requirements, (d) rain gardens have stricter maximum slope restrictions, and (e) detention ponds cannot be used in high density residential areas. Finding the optimal set of BMPs for any given watershed is a complex problem [23]. Chichakly, et al. [22], [23] formulated this as a biobjective design problem with real-valued design parameters, to minimize both cost and sediment load through the optimal placement and sizing of BMPs within a mixed-use watershed. Specifically, they used a multi-scale decomposition of the problem, where subwatershed-level optimizations were pre-computed over the entire range of treatment options. The evolutionary optimization problem was thus formulated such that the design variables were simply the fraction of area that is treated in each subwatershed (referred to hereafter as the treatment fraction). Evolved treatment fractions were then mapped back to pre-computed optimal BMP configurations for each subwatershed.

The Bartlett Brook watershed in South Burlington, VT was divided into 14 subwatersheds provided by the Vermont Agency of Natural Resources and modeled using Hydrological Simulation Program Fortran (HSPF) [25]. Land-use patterns were used to determine the maximum land area that could be used by each BMP type within each subwatershed. The different costs and restrictions pertaining to rain gardens and detention ponds cause nonlinearities and discontinuities in cost as a function of area treated within the 14 subwatersheds. For example, if the area to be treated is less than the minimum required for a detention pond, one is forced to use the more expensive rain gardens to treat that area, whereas it is actually cheaper to install a detention pond to treat a larger area. At the other end of the spectrum, if the area to be treated is larger than the land available for detention ponds (e.g., due to residential development), then one may need to supplement detention ponds with additional rain gardens. Chichakly [23] established a strong relationship between the logarithm of the standard deviation of flow at the outfall and the logarithm of sediment load at the outfall (Formula$R^{2}$ values were above 0.87 for nine watersheds with varying characteristics). Thus, the biobjective optimization problem was reframed to minimize cost and the standard deviation of flow at the outfall, where the latter is an effective and computationally efficient proxy for sediment load at the outfall, and that is the optimization problem we examine here.

Ideally, watershed management plans, which are expensive to implement and difficult to change, will remain effective even as precipitation patterns change due to global climate change. Already, rainfall patterns in the Northeastern U.S. are becoming increasingly variable and uncertain, and climate change predictions are that the intensity of individual rainfall events will continue to increase [26]. To partially account for this, Chichakly, et al. [22], [23], after biobjective minimization of cost and standard deviation of flow, discarded solutions from the evolved front that were dominated with respect to robustness to estimates of potential increased rainfall intensity. This method helped identify which solutions on the non-dominated front were more robust to potential changes in precipitation. Here, we apply the methods described in Sections II-A and II-B to obtain greater insights into what factors make watershed management plans more effective in removing sediment in a cost-efficient way and more robust to increases in intensity of precipitation.



A. Truss and Beam Design

Selected two-member truss visualizations are shown in Figs. 46 and selected welded beam visualizations are shown in Figs. 79. In each of these figures, the left-hand panel shows how the selected variable changes along the non-dominated front, while the right-hand panel illustrates how the same variable changes in solutions across the feasible region. Although the feasible region for the welded beam problem extends up to a maximum cost of$334 and a maximum deflection of 0.07 in, in all figures for the welded beam problem (Figs. 710) we have limited the displays to a smaller region of interest near the non-dominated front. From Figs. 49, it is evident that contours in the heatmaps (i.e., bands of the same color) of the dominated solutions in the feasible region can follow a variety of patterns. Specifically, for these two design problems, we observed contours in these heatmaps that were roughly:

  • Parallel to the Formula$y$-axis: For example, in Fig. 4(b), it can be seen that over much of the feasible region, decreases in the value of the plotted variable Formula$(A_{AC})$ are associated with a large reduction in volume (Formula$x$-axis) but with relatively little effect on maximum stress (Formula$y$-axis).
  • Parallel to the Formula$x$-axis: For example, in Fig. 6(b), over much of the feasible region, increases in the value of the plotted variable Formula$(A_{BC})$ are associated with a large reduction in maximum stress (Formula$y$-axis) with relatively little effect on volume (Formula$x$-axis).
  • Parallel to the non-dominated front: For example, in Fig. 9(b), the plotted variable Formula$(t)$ increases in value as solutions near the non-dominated front.
  • Non-monotonic: For example, in Fig. 5(b), over much of the feasible region, the contours of the plotted variable Formula$(x_{BC})$ are roughly parallel to the Formula$y$-axis (stress), but the maximum values of the variable occur near the middle of the Formula$x$-axis range (volume), with values decreasing as one approaches either front (non-dominated or fully-dominated).
Figure 4
Fig. 4. Two-member truss design variable Formula$A_{AC}$: (a) Effect on each objective along the non-dominated front (‘⋅’ vs. stress and ‘×’ vs. volume) and (b) heatmap of Formula$A_{AC}$ across objective space with white cp lines.
Figure 5
Fig. 5. Two-member truss design variable Formula$x_{BC}$: (a) Effect on each objective along the non-dominated front (‘⋅’ vs. stress and ‘×’ vs. volume) and (b) heatmap of Formula$x_{BC}$ across objective space with white cp lines.
Figure 6
Fig. 6. Two-member truss design variable Formula$A_{BC}$: (a) Effect on each objective along the non-dominated front (‘⋅’ vs. stress and ‘×’ vs. volume) and (b) heatmap of Formula$A_{BC}$ across objective space with white cp lines. The asterisks show the effect of decreasing the maximum stress of an existing solution (top asterisk).
Figure 7
Fig. 7. Welded beam design variable Formula$h$: (a) Effect on each objective along the non-dominated front (‘⋅’ vs. deflection and ‘×’ vs. cost) and (b) heatmap of Formula$h$ across objective space with white cp lines.
Figure 8
Fig. 8. Welded beam design variable Formula$b$: (a) Effect on each objective along the non-dominated front (‘⋅’ vs. deflection and ‘×’ vs. cost) and (b) heatmap of Formula$b$ across objective space with white cp lines.
Figure 9
Fig. 9. Welded beam design variable Formula$t$: (a) Effect on each objective along the non-dominated front (‘⋅’ vs. deflection and ‘×’ vs. cost) and (b) heatmap of Formula$t$ across objective space with white cp lines.

In the following, we show how these visualized patterns, alone or in combination with cp lines, can provide insights into various design problems.

Comparing Fig. 4(a)9(a) with the corresponding Fig. 4(b)9(b), one can see that inferences from solutions along the non-dominated front alone can be misleading. For example, Figs. 4(a) and 5(a) exhibit very similar relationships between the maximum stress and cost objectives and the two-member truss variables Formula$A_{AC}$ and Formula$x_{BC}$, respectively, suggesting that volume can be reduced with minor impacts to maximum stress by decreasing either of these variables. However, whereas the heatmap and cp lines for Formula$A_{AC}$ (Fig. 4(b)) show that this design principle holds across the feasible region, the heatmap and cp lines for Formula$x_{BC}$ (Fig. 5(b)) show that changes in Formula$x_{BC}$ within the dominated region produce very erratic results that behave very differently in different parts of the dominated region.

The cp lines often provide additional information as to how to interpret the heatmap. As in Figs. 4(a) and 5(a), Figs. 7(a) and 8(a) exhibit very similar patterns along the non-dominated front for the welded beam variables Formula$b$ and Formula$h$, suggesting that a decrease in either of these variables causes a decrease in cost and an increase in deflection. In this case, however, the heatmaps for these two variables (Figs. 7(b) and 8(b)) have similar contour patterns (almost parallel to the Formula$y$-axis), indicating a decrease in either variable results in a decrease in cost with little to no impact on deflection. However, whereas the cp lines for Formula$h$ (Fig. 7(b)) show that this relationship holds throughout the feasible region, the cp lines for Formula$b$ (Fig. 8(b)) show that decreasing Formula$b$ in a dominated solution actually reduces cost and increases deflection, as was seen on the front.

Likewise, in Fig. 9(a) cost appears to be nearly independent of the welded beam variable Formula$t$, because Formula$t$ is close to its maximum value along the entire non-dominated front. The heatmap shows contours for Formula$t$ that are parallel to the front (Fig. 9(b)), suggesting that increasing Formula$t$ would move one closer to the front by following this gradient. However, the cp lines are actually nearly parallel to the front above the knee, showing that increasing Formula$t$ yields major decreases in deflection for only minor increases in cost over much of the feasible region; as deflection nears its minimum, one observes diminishing returns for continuing to increase Formula$t$, such that major increases in cost are required for only minor decreases in deflection.

The non-dominated front shows that increasing the two-member truss variable Formula$A_{BC}$ will decrease maximum stress in a non-dominated solution up to a volume of 0.01 Formula${\rm m}^{3}$ (where Formula$A_{BC}$ reaches its maximum) with a minor increase in volume (Fig. 6(a)). There is no useful information on the non-dominated front beyond a volume of 0.01 Formula${\rm m}^{3}$. Both the heatmap and the cp lines in Fig. 6(b) confirm that this pattern persists for larger volume solutions as well. i.e., given any two-member truss design, it is possible to reduce maximum stress in exchange for a relatively smaller increase in volume by increasing Formula$A_{BC}$. As an illustration of this principle, suppose a manufacturer of two-member trusses has a large inventory of trusses built for a specific application (as detailed in Section II-C-I and with a maximum stress of 40 MPa), but now wishes to modify these trusses for use in a more demanding application (maximum stress of 20 MPa). The existing trusses were built with design parameters Formula$A_{AC}=0.00747 {\rm m}^{2}$, Formula$A_{BC}=0.00319 {\rm m}^{2}$, Formula$y=1.00 {\rm m}$, and Formula$x_{BC}=1.56 {\rm m}$, supporting the 100 kN load at the already specified maximum stress of 40.0 MPa with a volume of 0.0327 Formula${\rm m}^{3}$ (top asterisk in Fig. 6(b)). By increasing Formula$A_{BC}$ to 0.00637 Formula${\rm m}^{2}$, e.g., by welding a steel plate onto each existing member, the 50% reduction in maximum stress to 20.0 MPa can be attained with an attendant increase in volume of only 18% to 0.0386 Formula${\rm m}^{3}$ (bottom asterisk in Fig. 6(b)).

For the welded beam variable Formula$h$, the non-dominated front suggests that decreasing Formula$h$ not only reduces cost, but also produces an attendant increase in deflection (Fig. 7(a)). In contrast, the heatmap and cp lines show that reducing Formula$h$ actually reduces cost with little to no change in deflection (Fig. 7(b)). To illustrate this principle, suppose a manufacturer of welded beams has an order for a large number of welded beams that meet the specifications detailed in Section II-C-II. From the evolved non-dominated front, the manufacturer has picked a 6.349 and Formula${\rm deflection}={\rm 0.002896}$ in. The design parameters of the selected solution are Formula$b=0.7581$ in, Formula$t=10.00$ in, Formula$h=0.6754$ in, and Formula$len=1.431$ in. The variable Formula$h$ can be further reduced until the maximum shear stress is reached (that is, when Formula$h=0.6682$ in). With this modification, deflection remains unchanged (0.002896 in) while cost is reduced to$6.334, a 0.2363% savings, relative to the originally selected solution on the evolved non-dominated front (which could result in significant savings on large orders). This simplistic example serves to prove the point that design principles learned from dominated solutions can actually be used to push currently non-dominated solutions further towards Pareto-optimality.

In many engineering design problems, there is uncertainty in what the magnitude of the external forcing conditions will be over the lifetime of the design. Although properly designed beams are generally designed to handle maximum anticipated external forcing conditions, the following simple example demonstrates how one can find solutions that are more robust to uncertain forcing conditions. Suppose a manufacturer of welded beams needs to design a beam that meets the conditions given in Section II-C-II, but, in a specific application, the beam may be subject to varying loads and there is some uncertainty as to the true maximum load. Clearly, when the load exceeds the rated 6000 lb constraint, the stresses on the beam in solutions along the non-dominated front will also increase beyond the allowable maximum. To find a design solution that is more resilient to unexpectedly high loads, all solutions were re-evaluated with a 6,600 lb load and a heatmap was created of the differences in both shear stress and bending stress (Fig. 10). Across the entire non-dominated front for shear stress, one observes that moving a small distance inside the non-dominated front dramatically increases the robustness of the sheer stress of the solution to changes in load (Fig. 10(a)), i.e., the differences in sheer stress decrease markedly. The inset of Fig. 10(a) shows how rapidly shear stress drops when moving orthogonally away from the non-dominated front, starting from a representative point at the knee (7.43, Formula${\rm deflection}={\rm 0.00275}$ in) and following the path of the white line. For example, if one instead implemented the solution indicated by the circled point in the Fig. 10(a) inset, for only a 9% increase in cost (to$8.12) and an 11% increase in deflection (to 0.00306 in), the difference in shear stress drops disproportionately by 23% (from 1330 psi to 1030 psi), making the beam much more robust to variations in loading. On the other hand, bending stress in this same area (along the white line in Fig. 10(b)) is already at its maximum robustness.

Figure 10
Fig. 10. Difference in shear stress (a) and bending stress (b) between applying a 6600 lb load at the end of the welded beam and the base case of applying a 6000 lb load. Inset graph (a) shows stress decreasing as move orthogonally away from the front at a representative part of the knee (Formula$x$-axis of inset corresponds to points along the white line).

B. Watershed Management Plan Design

In Chichakly, et al. [22], [23], non-dominated solutions to a complex watershed problem were evolved to minimize cost of the BMP implementation plan and standard deviation of flow at the outfall of the watershed. In the current work, we show how valuable watershed management design principles can be discerned from patterns in dominated solutions.

Three characteristic patterns appeared in the heatmaps and cp lines for the design variables (i.e., subwatershed treatment fractions) of the Bartlett Brook watershed: (a) Contours in the heatmaps for subwatersheds 1, 2, 3, and 5 were roughly parallel to the non-dominated front above the knee, with treatment fraction increasing from right to left exerting corresponding increases in sediment control (i.e., decreases in standard deviation of flow) but with little impact on cost (e.g., Fig. 11(a)); (b) Contours in the heatmaps for subwatersheds 9, 11, 12, and 14 were roughly parallel to the non-dominated front below the knee, with treatment fraction increasing from bottom to top incurring increases in cost but relatively little effect on sediment control (e.g., Fig. 11(b)); (c) Contours in subwatersheds 4, 6, 7, 8, 10, and 13 exhibited a non-monotonic pattern where the treatment fractions on the fully-dominated front are roughly a mirror-image of those along the non-dominated front (e.g., Fig. 11(c)). The cp lines close to the non-dominated front were approximately orthogonal to the contours of the heatmaps in the first two cases (e.g., Fig. 11(a) and (b)), but are roughly parallel to the entire non-dominated front in the third case (e.g., Fig. 11(c)). Fig. 12 shows heatmaps of all 14 design variables for the Bartlett Brook watershed. In this visualization, we have sized each heatmap proportional to the area of its subwatershed, centered each heatmap vertically at the mean elevation of its subwatershed, and showed how the subwatersheds are connected based on drainage patterns. Those subwatersheds whose feasible solutions contained only rain gardens are indicated with an asterisk.

Figure 11
Fig. 11. Heatmaps with cp lines for three of the Bartlett Brook watershed design variables, illustrating the three types of observed patterns in dominated solutions. Specifically, the heatmaps represent the treatment fractions for subwatersheds (a) 1, (b) 9, and (c) 4.
Figure 12
Fig. 12. Heatmaps of treatment fractions for all subwatersheds in the Bartlett Brook watershed solutions, with associated subwatershed identification numbers and interconnections based on drainage topology (subwatershed 9 is at the outfall). Axes and color scale for each heatmap are identical to those shown in Fig. 11. In this combined plot, the Formula$y$-axis indicates the average elevation of each subwatershed (m), and each heatmap is sized proportional to the area of the subwatershed. Subwatershed numbers in bold with an asterisk next to them contain only rain gardens, whereas the other subwatersheds contain solutions using both detention ponds and (to a lesser degree) rain gardens.

Although the novel visualization in Fig. 12 conveys a wealth of information about each subwatershed, patterns yielding useful design principles governing treatment fractions are still not immediately apparent. For example, there is no consistent relationship between size of the subwatershed and its treatment fraction in feasible solutions. One might initially conclude that it is more cost-efficient to treat larger subwatersheds before smaller subwatersheds, since the smaller subwatersheds have little to no treatment in solutions that are on or near the non-dominated front below the knee. However, we observe very similar patterns in treatment fractions in subwatersheds 1 vs. 3, 2 vs. 5, and 4 vs. 7, even though the subwatersheds in each of these pairs are very differently sized (Fig. 12). Nor is mean subwatershed elevation a consistent indication of the pattern of treatment fraction in feasible solutions. For example, note that only four of the seven highest elevation subwatersheds (1, 2, 3, and 5) receive much treatment in solutions that are on or near the non-dominated front below the knee. The relative position between subwatersheds in the drainage topology is also not consistently associated with patterns in treatment fractions. For example, subwatersheds 1, 2, 4, 5, and 12 are all at the top of the watershed, but exhibit different patterns in treatment fractions across the feasible region, whereas subwatersheds 12 and 9 share very similar patterns even though 12 is at the top of the watershed and 9 is at the bottom.

Plots of aggregate measures across the watershed proved more informative, enabling us to see patterns resulting from the interactions of individual decision variables. For example, we computed the total fraction of the watershed treated by detention ponds and rain gardens in all feasible solutions. Not surprisingly, along the non-dominated front the treatment fractions for both detention ponds (Fig. 13(a)) and rain gardens (Fig. 13(b)) increase from right to left, corresponding to more effective but more costly management plans. In all solutions, the treatment fractions for detention ponds were an order of magnitude higher than those for rain gardens. This occurs because, for a given overall treatment fraction, our pre-computed subwatershed-level optimizations favored the cheaper detention ponds over rain gardens, except where not possible due to residential development. Thus, the heatmap for overall treatment fraction is visually nearly identical to Fig. 13(a) and is not shown.

Figure 13
Fig. 13. (a) Fraction of watershed treated by detention ponds, (b) fraction of watershed treated by rain gardens, (c) average slope of detention pond treatment area, and (d) average slope of rain garden treatment area.

However, specifying an overall treatment fraction is not sufficient for specifying an adequate and cost-efficient solution. Note that the contours in Fig. 13(a) are generally diagonal, such that solutions with the same detention pond treatment fraction that are farther from the non-dominated front are both more expensive and less effective in controlling sediment load. The increased cost is partially explained by an increase in more expensive rain gardens (Fig. 13(b)). However, as one moves farther from the non-dominated front, the rain garden treatment fraction begins to fluctuate around a relatively constant value, and as one nears the lower portion of the fully-dominated front, both detention pond treatment fraction and rain garden treatment fraction actually decrease, yet the cost continues to increase (Fig. 13(a) and (b)). This non-linearity occurs due poor watershed management plans near the fully-dominated front that land at high points in the non-linear subwatershed cost-per-area-treated functions.

Why are the most costly solutions with the same detention pond treatment fractions less effective in controlling sediment load? Heatmaps of area-weighted elevation and impervious area treated (not shown) exhibit the same pattern as detention pond treatment fraction, so do not shed any additional light on this matter. However, a heatmap of the weighted average of the slopes of the subwatersheds, weighted by the area treated by detention ponds (Fig. 13(c)) and rain gardens (Fig. 13(d)), helps to explain this. Comparing Fig. 13(a) and (c), it is immediately apparent that, for the same detention pond treatment fraction, detention ponds on steeper slopes are less effective. Thus, an inferred design principle is: Detention ponds on shallower slopes are more effective in reducing sediment load. Although this principle was derived by comparing treatment fractions between subwatersheds, it could also provide useful guidance for specific placement of the actual detention ponds within each subwatershed.

The relationship between rain garden treatment fraction (Fig. 13(b)) and slope (Fig. 13(d)) is less clear. Further investigation revealed that the subwatersheds selected for treatment with rain gardens on or near the non-dominated front were subwatersheds 6, 8, and 10, where detention ponds were not feasible due to existing development.

The watershed problem provides a more realistic application of design in the face of uncertain forcing conditions. By law, states are only required to treat watersheds sufficiently to meet contaminant targets under current precipitation patterns. However, precipitation patterns in the Northeastern U.S. are expected to become increasingly intense due to climate change [26]. It would therefore be wise to implement plans today that are better able to manage the expected (but unknown) increased pollutant loads in the future. Thus, to assess the robustness of solutions to anticipated increases in intensity of precipitation, inFig. 14 we display the difference in standard deviation of flow with respect to two different rainfall patterns, the actual rainfall pattern from 2008 and a more intense synthetic precipitation pattern with the same total precipitation concentrated into more intense storms (specifically, with a standard deviation of rainfall 28% higher than the 2008 pattern; see Chichakly [23] for more details). In several places (see white lines on Fig. 14), solutions a short distance away from the non-dominated front were found to be more robust (i.e., show smaller increases in standard deviation of flow in response to increases in the intensity of precipitation) than the solutions on the front. For example, consider the solution close to the knee of the front, at a cost of$327,000 and a standard deviation of flow of 0.0727 cms, marked with an asterisk on Fig. 14. The sensitivities to increased intensity of precipitation along the horizontal and vertical white lines shown emanating from this point in Fig. 14are plotted in Fig. 15(a) and (b), respectively. Moving horizontally there is a rapid nonlinear drop in sensitivity by moving only a short distance; for example, if one is willing to accept a small (2%) increase in the current standard deviation of flow, but one is not willing to increase implementation costs, one can reduce the sensitivity by 4% (Fig. 15(a), circled point). On the other hand, moving vertically, if one is willing to increase the cost but is not willing to allow an increase in the current standard deviation of flow, then one achieves only a linear decrease in sensitivity (Fig. 15(b)); specifically, each 4% increase in cost reduces the increase in future standard deviation of flow by only 1%, so moving in this direction is less cost-effective. The corresponding detention pond and rain garden treatment fractions along these two lines help explain the reasons behind these decreases in sensitivity (increases in robustness) just inside the front. As shown in Fig. 15(c) and (d), rain garden treatment fraction increases in the more robust solutions; when cost is held constant, this is accompanied by a decrease in detention pond treatment fraction (Fig. 15), whereas when cost is allowed to increase the detention pond treatment fraction also increases. These relationships also hold along the other white lines in Fig. 14. Since rain gardens infiltrate whereas detention ponds do not, the increase in robustness from increased treatment with rain gardens may be tied to increased infiltration. Thus, a design principle is: Whereas rain gardens are less cost-effective for a given level of control, modest increases in the proportion of rain garden treatment fraction, relative to detention pond treatment fraction, may increase the robustness of the watershed to increases in the intensity of precipitation events.

Figure 14
Fig. 14.Difference in pollutant load for different BMP treatments between a more intense precipitation pattern predicted by [26] and the existing precipitation in Bartlett Brook. White lines indicate places where moving away from the front may produce a solution that is more resilient to changes in precipitation.
Figure 15
Fig. 15. Breakout of leftmost set of white lines from Fig. 14, both with increasing standard deviation of flow (left) and increasing cost (right). The relative differences in standard deviation of flow between the two precipitation patterns are shown in the top row and the corresponding change in treatment fraction of each BMP type is shown in the bottom row.


We have shown that patterns in dominated solutions throughout the feasible region in biobjective problems can often give more information than is apparent from patterns along the non-dominated front alone. By visualizing the inherent self-organization in design parameters and other variables of interest across the feasible region, in relation to the non-dominated and fully-dominated fronts, we can obtain sound insights into underlying design principles that are valid throughout the entire design space.

With simple examples from a two-member truss design problem and a welded beam design problem, we demonstrated that the information on the non-dominated front can sometimes be misleading and that information from dominated solutions can provide additional useful design principles. We also showed how these principles could be used to modify and improve an existing design, and to improve a previously non-dominated solution. Furthermore, in the welded beam problem, we discovered that dominated solutions a short distance inside the non-dominated front exhibited markedly reduced sensitivity of sheer stress to increased loading, with only minor degradations to the solution quality with respect to the original objectives.

We also visualized patterns in dominated solutions of a complex watershed management plan design problem. This helped us to discover that, for the same overall treatment fraction, placing detention ponds in areas of the watershed with shallower slopes reduces pollutant load at a lower cost. We also discovered that some solutions to the watershed management problem, which were only slightly suboptimal with respect to the original two objectives, were much more robust to increased intensity of precipitation. Further visualizations revealed that this increase in robustness was due to a small increase in the relative proportion of treatment by infiltrating rain gardens, relative to non-infiltrating detention ponds. Such insights provide valuable guidance to watershed managers who are required to develop management plans that can meet today's contaminant requirements, but will hopefully also remain as effective as possible as the climate changes.

From the insights gained in these examples, we anticipate that visualization of dominated solutions could lead to useful innovizations in a variety of biobjective engineering design problems. Since the entire feasible region is included, identified design principles remain valid across this entire region, thus enabling the improvement of existing designs and ruling out apparent relationships that are artifacts of the non-dominated front. In addition, we found that strictly non-dominated solutions are often fragile, in the sense that they are only optimal with respect to the specific forcing conditions for which they were evolved. Visualizing sensitivities to changes in these forcing conditions across the feasible region can often help one identify solutions that are much more robust to uncertainties in these assumptions.


K. J. Chichakly and M. J. Eppstein are with the Department of Computer Science, University of Vermont, Burlington, VT 05401, USA, Corresponding author: K. J. Chichakly (

Color versions of one or more of the figures in this paper are available online at


No Data Available


Karim J. Chichakly

Karim J. Chichakly

Karim J. Chichakly received the B.A. degree from Skidmore College, Saratoga Springs, NY, USA, in 1980, the B.E. and M.E. degrees in electrical and software engineering from Dartmouth College, Hanover, NH, USA, in 1981 and 1986, respectively, the M.S. degree in system dynamics from the Worcester Polytechnic Institute, Worcester, MA, USA, in 2010, and the Ph.D. degree in computer science from the University of Vermont, Burlington, VT, USA, in 2013.

He is the Director of Product Development with isee systems, inc., Lebanon, NH, USA. He is an Adjunct Lecturer of system dynamics with the Worcester Polytechnic Institute and an Adjunct Professor in computer science with Capitol College, Laurel, MD, USA. His current research interests include evolutionary computation, system dynamics, and computer language design.

Dr. Chichakly is a member of the System Dynamics Society.

Margaret J. Eppstein

Margaret J. Eppstein

Margaret J. Eppstein is an Associate Professor and Chair of the Department of Computer Science, University of Vermont, Burlington, VT, USA. She received the B.S. degree in zoology from Michigan State University, East Lansing, MI, USA, in 1978, and the M.S. degree in computer science and the Ph.D. degree in environmental engineering from the University of Vermont, Burlington, in 1983 and 1997, respectively.

She has been a Faculty Member since 1983, a Lecturer from 1983 to 2001, a Research Assistant Professor from 1997 to 2002, an Assistant Professor from 2002 to 2008, and has been an Associate Professor since 2008. She was the Founding Director of the University of Vermont Complex Systems Center from 2006 to 2010.

Her current research interests include forward and inverse modeling and analysis of complex systems in a wide variety of application domains, including biological, environmental, technological, and social systems.

Cited By

No Data Available





No Data Available
This paper appears in:
No Data Available
Issue Date:
No Data Available
On page(s):
No Data Available
INSPEC Accession Number:
Digital Object Identifier:
Date of Current Version:
No Data Available
Date of Original Publication:
No Data Available

Text Size

Comment Policy
comments powered by Disqus