In this age of scientific computing, the simulation science pipeline of mathematical modeling, simulation and evaluation is a rendition of the scientific method as commonly employed as the traditional experimental pipeline. Critical to this simulation approach is the evaluation stage in which numerical data are post-processed, visualized and then examined in order to answer the original queries that instigated the investigation. In fact, visualization of scientific results has become as much a part of the scientific process as mathematical modeling or numerical simulation.

Despite its growing importance in the scientific computing process, visualization has not fallen under the same rigorous scrutiny as other components of the pipeline like mathematical modeling and numerical simulation. Unlike in traditional computational science and engineering areas, there is no commonly accepted framework for verifying the accuracy, reliability, and robustness of visualization tools. This precludes a precise quantification of the visualization error budget in the larger scientific pipeline. Furthermore, very few investigations have focused on how the error originating from other components of the computational pipeline impacts visualization algorithms.

In this paper, we advocate the use of techniques from the *verification and validation* process used in the engineering community (Section 2 presents V&V in more detail). While the lack of a well-established framework for verifying visualization tools has meant that a variety of analysis techniques have been developed [26], [29], we believe that visualization has achieved sufficient importance to warrant investigation of stricter verification methodologies. Several authors have already asserted this need [8], [12], and in this paper we present techniques that are a concrete step towards reducing the gap between best practices in simulation science and visualization.

The purpose of this work is to present a verification methodology for visualization tools that has comparable rigor to that of other components of the computational scientific pipeline. More specifically, we set out to define a verification methodology in the mold of the area of V&V, in the context of visualization. Furthermore, we illustrate our proposal by testing several publicly available isosurface extraction codes to the verification procedure, giving a detailed description of the steps involved in the process.

It is important to emphasize that the point of verification procedures is not to compare algorithms to one other with the hope of finding the best alternative. This procedure equips developers of new algorithms and/or implementations with a process that provides a systematic way of identifying and correcting errors in both algorithms and implementations. The goal is to provide users with a methodology that will give them a more concrete model for the behavior of the implementation, which will increase confidence in the visualization tools. As we will show, a fair verification analysis can bring out unforeseen behavior, and quickly detect implementation problems that might not be caught through standard comparisons or visual inspection.

The contributions of this work are threefold. To the best of our knowledge, we, for the first time, apply the framework of verification to assess the correctness of visualization tools. Furthermore, we provide a detailed description of how to accomplish the verification procedure by subjecting different isosurfacing tools to the battery of tests comprising the V&V process. Our second contribution is the underlying mathematical analysis and associated manufactured solutions developed to analyze the isosurfacing methods. We should clarify that when applying MMS for other techniques (even in the case of iso-surface extraction), the theoretical analysis should be tailored to the particular features of these algorithms. The manufactured solutions presented here are simple but general enough to be promptly employed for evaluating other visualization tools besides isosurfacing. Our third contribution is a comprehensive set of results obtained using the technique, including the finding of implementation errors in two publicly available isosurface extraction codes.

SECTION 2

## Validation and Verification

In this section we provide a brief introduction to the idea of *verification and validation* (V&V), and in particular, its application in visualization scenarios. We will also review efforts to verify implementations in the specific context of isosurface extraction.

Babuška and Oden define verification as "the process of determining if a computational model obtained by discretizing a mathematical model of a physical event and the code implementing the computational model can be used to represent the mathematical model of the event with sufficient accuracy [1]. Although they review the concept only in the context of computational science and engineering applications, it is important to appreciate that the same idea applies to scientific visualization *Verification is about investigating to what extent a (numerical) approximation scheme − both in algorithm and corresponding implementation − represents the desired mathematical model*. Validation, on the other hand, is about ensuring that the model represents physical reality. In this paper, we will concern ourselves only with verification, under the assumption that the model has been validated by the user of the technique. This is the perspective Kirby and Silva suggest for "Verifiable Visualization [12].

One of the main requirements for verifiable visualization is to have a rigorous analysis which predicts the results of the algorithm and its implementation when evaluating it on a known model problem. The more complete this analysis is, the more thorough the testing procedures can be. This *continuous process* of verification through refinement of key controllable input parameters of the method (such as grid spacing) and testing is different from a one-shot process. The verification process should involve a suite of tests with corresponding results from which one can progressively increase reliance on the method under analysis. When appropriately applied, verification provides ways to appreciate the nuances of the applicability of the method. As we will see in this paper, writing down the analysis for the expected result of isosurface extraction gives us concrete bounds on what features we can expect the resulting surface to have, and these are extremely important for users.

A common practice in the visualization community is to test implementations by using complicated, "real-world" datasets. The value of these tests is that they provide evidence of the algorithm's applicability. We advocate a complementary approach: developers should carefully manufacture test cases that can be mathematically modeled and analyzed, called *manufactured solutions*. These manufactured solutions can then be used to test the implementations. In this paper, we present analysis that describe the expected rate of convergence of several isosurface features, and test implementations acting on our model problems using simple analytical volumes. As we show in Sections 3 and 4, this method helps pin down the mathematical characteristics of the technique, and, more practically, it is quite effective at uncovering implementation bugs.

The challenge behind manufactured solutions is to construct them in a way that allows us to predict the expected behavior of the method under investigation. Moreover, the manufactured solutions should tax the method vigorously, bringing out potential problems. In Section 5, we will present some situations where incorrectly chosen manufactured solutions have a big impact in the results. We do so to emphasize that all components of the pipeline, even the construction and evaluation of the manufactured solutions, must be meticulously handled to maintain the rigor of the verification process.

### 2.1 Isosurface Extraction Algorithms

Isosurface extraction is a popular visualization technique, being a tool currently used in science, engineering, and applications. This popularity makes it a natural target for this first application of verification mechanisms in the context of visualization. This same popularity has also driven a large body of work comparing different isosurface extraction algorithms.

Previous researchers have examined topological issues [14], [18], mesh quality [6], [23], accuracy [19], [29], and performance [24]. The influence of different reconstruction schemes and filters in scalar visualization has also been examined [3], [20]. In this paper, we focus on techniques to verify the correctness of algorithms and their corresponding implementations. In particular, we provide mathematical tools that other researchers and developers can use to increase their confidence in the correctness of their own isosurface extraction codes. A traditional way to test implementations in scientific visualization is to perform a visual inspection of the results of the Marschner-Lobb dataset [16]. In the context of isosurface extraction, researchers routinely use tools such as Metro [4] to quantitatively measure the distance between a single pair of surfaces. We argue that the methodology presented here is more effective and more explicit at elucidating a technique's limitations. In particular, our proposal pays closer attention to the interplay between a theoretical convergence analysis and the experimental result of a *sequence* of approximations.

Globus and Uselton [8] and more recently Kirby and Silva [12] have pointed out the need for verifying both visualization techniques and the corresponding software implementations. In this paper, we provide concrete tools for the specific case of isosurface extraction. Although this is only one particular technique in visualization, we expect the general technique to generalize.

It is important to again stress that verification is a *process*: even when successfully applied to an algorithm and its implementation, one can only concretely claim that the implementation behaves correctly (in the sense of analyzed predicted behavior) for all test cases to which it has been applied. Because the test set, both in terms of model problems and analyzed properties, is open-ended and ever increasing, the verification process must continually be applied to previous and new algorithms as new test sets become available. This does not, however, preclude us from formulating a basic set of metrics against which isosurface extraction methods should be tested, as this is the starting point of the process. This is what we turn to in the next section.

SECTION 3

## Verifying Isosurface Extraction Algorithms

In this section, we describe the technique we use for verifying iso-surface extraction algorithms, namely the *method of manufactured solutions* (MMS). We illustrate a possible implementation of MMS in Figures 2 and 3. This technique requires us to write down the expected behavior of particular features of interest of the object (or model problem) being generated. In our case, we are generating triangular approximations of smooth isosurfaces, and the features of interest are geometric surface convergence, convergence of normals, area and curvature.

To use MMS, we first accomplish a mathematical analysis of the expected convergence rate of the features (or characteristics) of interest, known in the numerical literature as the *formal order of accuracy* of the characteristic. This analysis is done for solutions of the problem that can be conveniently described and analyzed (these are the manufactured solutions). Then, the code is executed with progressively refined versions of the data that is used in the generation or sampling of the manufactured solution. Finally, the empirical convergence rate is compared to the one predicted by the analysis. When the convergence rates are comparable, we increase our confidence in the algorithm. If the realizable behavior disagrees with the analysis, either (1) the analysis does not correspond to the correct behavior of the algorithm, (2) the assumptions upon which the analysis was build were violated by the input data and hence the predicted behavior is not valid for the circumstances under investigation, or (3) there are issues with the algorithm or with the implementation of the algorithm (depending on access to source code and algorithmic details, one may not be able to distinguish between these two − algorithmic or implementation − and hence we in this work always consider them together. Given sufficient information, the verification process can help further delineate between these two issues). Notice, however, that all three situations warrant further investigation. In the following sections, we will discuss these issues in more detail. Let us first clarify how we will arrive at theoretical and empirical convergence rates.

For a fixed grid size, we will strive to write the approximation error between the desired isosurface property and its approximation by:

*E* = *|u*_{approx} − uexact |_{u} = *O*(*h*^{p}) = *αh*^{p} (1)

where *u*_{approx},uexact are the approximated and exact values of a property *u* *| · |*_{u} is the norm used to compare the approximate and exact property *p* is the order of accuracy and *α* is a constant.

Practically speaking, the polynomial expression (1) is not very convenient for numerical experimentation, as it is hard to find the value of *p* from the direct plot of *h* against *E*. The standard technique to estimate *p* is to linearize by working on a log-log scale:

log *E* = log(*αh*^{p}) = log *α* + *p*log *h*. (2)

Using this linearized version, we estimate *p* from the slope of the line that best fits the points (log *h,*log *E*) in a least-squares sense. We use this technique in Section 4 when testing the isosurface codes.

MMS critically depends on an analysis of the order of accuracy of expected solutions. Although this seems quite simple, the order of accuracy under a sensitive norm like *|| · ||*
_{∞} has shown in practice to be very effective in bringing out implementation errors in numerical approximation schemes [1], [22]. In this paper, we show that this analysis is just as effective for isosurface extraction. In addition, we believe the convergence analysis required by MMS is interesting in its own right. As we will discuss in Section 5, it helps to shed light on the consequences of implementation choices.

In the context of isosurface methods, manufactured solutions can be built by specifying a "solution surface" to be the exact solution and deriving a scalar field that contains such a solution surface as a level set. The verification methodology then proceeds as following: (1) use the manufactured scalar field as input for the isosurfacing methods, (2) run the methods, and (3) check the output surface against the solution surface (sometimes called the *ansatz solution* within the mathematical verification literature). In many cases, the manufactured scalar field can be derived analytically, making the observed order of accuracy tractable (we give examples in next section).

In what follows, we will derive expected orders of accuracy for several features of surfaces produced by isosurface extraction codes. We keep our assumptions about the actual algorithms to a minimum to maximize the applicability of the arguments given. We essentially only assume that the maximum triangle size can be bounded above at any time, and use Taylor series arguments (under assumptions of smoothness) to derive convergence rates. It is important to point out that order of accuracy analysis of polyhedral surfaces has been studied by many researchers [9,17,27,28]. In fact, the results presented below are in agreement with the ones reported in the literature. However, because we are considering isosurface extraction, some of our arguments benefit by being able to be condensed to simpler statements.

### 3.1 Convergence of Vertex Position

We start our analysis of isosurface extraction by studying the convergence of vertex positions. We analyze this convergence indirectly by relating the values of the scalar field at the vertex points and the distance between the vertices and the correct isosurface. Given a value *λ* such that the exact isosurface *S* is defined by , the *algebraic* distance of to *S* is defined as [25]. Notice that algebraic distances only makes sense for implicit surfaces: it requires a scalar field. In addition, we restrict ourselves to *regular* isosurfaces, ones where for every point *x* in *S*, |▿ *f* (*x*)| exists and is nonzero. Then, the geometric distance between and *S* is approximated by [25]. We illustrate this relation in Figure 4. Since, by assumption, |▿ *f* (*x*)*| > k* for some *k* > 0, and all *x* in *S*, convergence in algebraic distance implies convergence in geometric distance. Convergence in algebraic distance, however, is much more tractable mathematically, and this is the item to which we turn our focus.

Many isosurface methods estimate vertex positions through linear interpolation along edges of a grid. Let *f* : *U ⊂ ℝ*^{3} → ℝ be the a smooth real function defined in a subset *U* = [*a*_{x},b_{x}] *×* [*a*_{y},b_{y}] *×* [*a*_{z},b_{z}], where [*a*_{i},b_{i}] *i* ∈ *x,y,z* are real intervals. We assume the intervals [*a*_{i},b_{i}] have the same length and let *a*_{x} = *x*
_{0}*,…,x*_{n} = *b*_{x} *a*_{y} = *y*
_{0}*,…,y*_{n} = *b*_{y}, and *a*_{z} = *z*
_{0}*,…,z*_{n} = *b*_{z} be subdivisions for the intervals such that *x*_{i} = *x*
_{0} + *ih* *y*_{i} = *y*
_{0} + *ih* *z*_{i} = *z*
_{0} + *ih, i* = 0 *,…,n*, where *h* is the grid size and *c*_{i} jk = [*x*_{i},x_{i}
_{+1}] *×* [*y j,y j*+1] *×* [*z*_{k},z_{k}
_{+1}] is a grid cell. Through a Taylor series expansion of *f*, one can evaluate *f* at a point as:
TeX Source
$$f(\vec p){\rm{ = }}f_{ijk} + \nabla f_{ijk} \cdot \vec \delta + {1 \over 2}\vec \delta ^T H(\vec \xi)\vec \delta $$
where *f*_{i}jk = *f* (*x*_{i},y j,z_{k}), ▿ *f*_{i}jk is the gradient of is the Hessian of *f* at a point connecting (*x*_{i},y j,z_{k}) and , and is such that .

Let the linear approximation of *f* in be defined by
TeX Source
$$f(\vec p){\rm{ = }}f_{ijk} + \nabla f_{ijk} \cdot \vec \delta $$
and consider a point such that , that is is a point on the isosurface *λ* of .

The algebraic distance between the exact isosurface *f* (*x,y,z*) = *λ* and the linearly approximated isosurface can be measured by . From Equations 3 and 4 one can see that
TeX Source
$$\eqalign{{\rm{|}}f(\vec x_\lambda) - \lambda |{\rm{ }} = {\rm{ }}f_{ijk} + \nabla f_{ijk} \cdot \vec \delta + {1 \over 2}\vec \delta ^T H(\vec \xi)\vec \delta - \lambda | = \cr |\tilde f(\vec x_\lambda) + O(h^2) - \lambda | = O(h^2) \cr} $$
thus, the linearly approximated isosurface is of second-order accuracy.

### 3.2 Convergence of Normals

Assume, generally, that the scalar field *f* (*x,y,z*) = *λ* can be locally written as a graph of a function in two-variables *g*(*x*(*u,v*)*,y*(*u,v*)) = *l − f* (*x*(*u,v*)*,y*(*u,v*)*,z*_{k}), as illustrated in Figure 5, where *x*(*u,v*) = *x*_{i} + *uh* and *y*(*u,v*) = *y j* + *vh*. This is acceptable because we have already assumed the isosurface to be regular. Still without losing generality we write *g*(*x*(0,0)*,y*(0,0)) = 0, that is, the isosurface contains the point (*x*_{i},y j,z_{k}). Let (*u,v*) = (*x*(*u,v*)*,y*(*u,v*)*,g*(*x*(*u,v*)*,y*(*u,v*))) be a parametrization for the isosurface *f* (*x,y,z*) = *λ* in *c*_{i}jk and
TeX Source
$${{\partial \vec \Phi } \over {\partial u}} \times {{\partial \vec \Phi } \over {\partial v}} = h^2 (- {{\partial g} \over {\partial x}}, - {{\partial g} \over {\partial y}},1)^T = h^2 \vec n_0 $$
be the normal vector in (0,0) = (*x*_{i},y j,g(*x*_{i},y j)) (the partial derivatives of *g* are evaluated at (*x*(0,0)
*,y*(0,0)) = (*x*_{i},y j)).

Consider now the triangle defined by the points , and approximating the isosurface *f* (*x,y,z*) = *λ* in the grid cell *c*_{i}jk (see Figure 5). Let be the grid point (*x*_{i},y j,z_{k}), so , and . Using the cross product in ℝ^{3}, the normal of the triangle *p*
_{1} *p*
_{2} *p*
_{3} can be computed by:
TeX Source
$$\eqalign{np_1 \vec p_2 p_3 = (\vec p_2 - \vec p_1) \times (\vec p_3 - \vec p_1) = \cr \left(\matrix{h(v_2 g(x(u_3,v_3),y(u_3,v_3)) - v_3 g(x(u_2,v_2),y(u_2,v_2))) \hfill \cr h(u_3 g(x(u_2,v_2),y(u_2,v_2)) - u_2 g(x(u_3,v_3),y(u_3,v_3))) \hfill \cr h^2 (u_2 v_3 - u_3 v_2)) \hfill \cr} \right). \cr} $$
Expanding *g*(*x*(*u*_{i},v_{i})
*,y*(*u*_{i},v_{i}))
*, i ∈ {*2,3 *}* in a Taylor series, some terms cancel and the normal *n*_{p}
_{1}
*p*
_{2}
*p*
_{3} becomes:
TeX Source
$$np_1 \vec p_2 p_3 = rh^2 \left({ - {{\partial g} \over {\partial x}} + o(h), - {{\partial g} \over {\partial y}} + o(h),1} \right)^T $$

where *r* = *u*
_{2}
*v*
_{3} *−u*
_{3}
*v*
_{2}. Comparing the exact normal vector *n*
_{0} in Equation 6 with *n*_{p}
_{1} *p*
_{2} *p*
_{3} above, we recover first-order of accuracy for normals. In addition, notice that the usual scheme of estimating vertex normals by the arithmetic mean of triangle normals does not decrease the order of accuracy; that is, vertex normals (computed by arithmetic mean) are at least first-order accurate.

### 3.3 Convergence of Area

Currently, much less is known about convergence in area, compared to convergence of vertices or normals. To illustrate the difficulty involved in approximating lengths and areas, consider the sequence of approximations to a straight line shown in Figure 6. Even though the function sequence converges uniformly to the line, the length of the approximation stays constant.

To the best of our knowledge, the only relevant results establish convergence in area given convergence in vertex positions *and* convergence in normals, such as in Hildebrandt *et al.* [9]. However, the authors only establish asymptotic convergence, with no order of accuracy associated with it. The argument is more mathematically involved than space allows here, so we refer the reader to that paper. Currently, this means that the only information the observed order of accuracy provides to us is that if we expect convergence in normals, we should also expect convergence in area, and vice-versa.

### 3.4 Convergence of Curvature

The following formula gives an estimate of the curvature at a vertex *p*:
TeX Source
$$K(p) = {{2\pi - \sum {\theta _{ii} + 1} } \over {{1 \over 3}A_{ii + 1} }}$$
where *θ*_{ii+1} and *A*_{ii+1} are the angle ∠_{p}*i p*_{pi+1} and area of the triangle *p*_{i} *pp*_{i+1} respectively (summation is over all triangles comprising the star of *p* [17]. Meek and Walton [17] showed that the curvature computed via Equation 9 does not converge in general; that is, if the vertices of the star of *p* are arbitrarily distributed around *p*, one cannot expect curvature convergence. In fact, they described a more general result stating that *O*(*h*) accuracy can only be obtained if the normals are known to have accuracy *O*(*h*^{2}). Subsequently, Xu [27] presented a very particular distribution of vertices around *p* under which the curvature estimated by Equation 9 has accuracy *O*(*h*^{2}).

Curvature discretization schemes other than the one given in Equation 9 such as the quadratic-fit and spherical-image method (see Meek and Walton [17] for details) also demand particular vertex distributions to ensure convergence. In our context of keeping the analysis applicable for many isosurfacing algorithms, this means we cannot use the lack of observed curvature convergence as an indication of problematic behavior. Based on the results mentioned above, one should actually expect curvature not to converge for most isosurface extraction algorithms. More generally, this indicates a weakness of MMS, namely that some features of interest (such as curvature) will not have sufficient theoretical order of accuracy to be used in numerical measurements. Notice, in addition, that if we had not written down the theoretical model for curvature convergence, we might have expected some sort of curvature approximation. Even a negative result such as the one presented in this section can increase the confidence in the results generated by an implementation.

SECTION 4

## Experimental Results

In this section we present the results of applying the afore-described methodology. We use the framework to verify six different isosurface extraction codes, namely: VTK Marching Cubes [15], SnapMC [21], Macet [6], Dual Contouring [11], Afront [23], and Dellso [5], All these implementations are open source and/or publicly available^{1}. Before presenting the actual results of subjecting these implementations to the verification process, we briefly review their salient features.

**VTK Marching Cubes:** Marching Cubes [15] (MC) is arguably the most popular isosurface extraction algorithm. It reduces the problem of generating an isosurface triangulation to a finite set of cases by considering the *signs* of how the isosurface intersects each cell of a regular background grid. As there are only 256 different types of intersections between the isosurface and a regular Cartesian 3D cell, a template of triangles is set to each case, making the implementation quite simple through a look-up table. The vertices of the triangles lie on the edges of the cubic cells, and they are computed by linearly interpolating the implicit function values stored at the corners of the grid cell.

**SnapMC:** SnapMC [21] is a recently proposed algorithm that extends the original Marching Cubes look-up table to cases where the isosurface goes exactly through the corners of the background grid. The new look-up table is automatically built by an adaptation of the convex hull scheme proposed by Bhaniramka *et al.* [2], Even though the traditional Marching Cubes algorithm can easily handle these cases by some kind of symbolic perturbation, SnapMC *perturbs the scalar field* to avoid edge intersections close to grid corners. In particular, it changes the values on the grid so that the surface is "snapped" to the grid corners.

**Macet:** Macet [6] is another variant of Marching Cubes that tries to improve the shape of the triangles in a mesh. Unlike SnapMC, it *perturbs the active edges* of Marching Cubes cases by moving the vertices before the triangulation step. The motivation behind Macet is that poorly-shaped triangles tend to be generated when the intersection between the isosurface and a grid cell is approximately parallel to an edge of the grid cell. Therefore, some corners of the background grid are displaced so as to avoid the parallel-like intersections.

**Dual Contouring:** Dual Contouring [11] is a feature-preserving iso-surfacing method to extract crack-free surfaces from both uniform and adaptive octree grids. This technique can be seen as a combination of Extended Marching Cubes [13] and SurfaceNets [7] as it makes use of Hermite data and quadratic error function minimization to position the vertices of the surface mesh (as Extended Marching Cubes) and the dual topology to connect such vertices (as SurfaceNets). Dual Contouring tends to generate better quality triangles than Marching Cubes while still being very effective in representing sharp features, rendering this implicit polygonalization method a good alternative to the popular Marching Cubes.

**Afront:** Afront [23] is an advancing-front method for surface extraction. Although we focus on applying Afront to isosurface extraction, it can also be used for remeshing and triangulating point-set surfaces. The outstanding feature of Afront is that it generates triangles adapted to the local details of a surface, namely its maximum absolute curvature. In this sense, Afront is fundamentally different from the other algorithms we analyze. In lieu of grid refinement, we will use its *ρ* parameter to control triangulation size. Because the manufactured solution we use is a sphere, reducing *ρ* by half is roughly equivalent to reducing the maximum triangle size by half. A full analysis of Afront (and, in particular, the influence of the other main parameter η) warrants further investigation, but is beyond the scope of this paper.

**Dellso:** Dellso [5] is a Delaunay-based approach for isosurfacing. It computes the restricted Delaunay triangulation from a 3D Voronoi Diagram. We run our tests on a customized version of Dellso 16 bit, and our examples use the default set of parameter.

In what follows, we present the results of applying the verification process to these algorithms. We will describe the manufactured solutions we use and their observed convergence rate on the isosurface extraction algorithm.

### 4.1 Observed order of accuracy

We start by investigating the behavior of the algorithms under the manufactured solution given by the scalar field *f*(*x* *y* *z* *= x*^{2} + *y*^{2} *+ z*^{2} – 1 and isosurface *f*(*x* *y* *z*) = 0 in the domain *D =* [–4,4]^{3}. Let be a simplicial complex that approximates *S* for a given discretization parameter *k* (cell size *h* for marching cubes-based methods, accuracy *ρ* for Afront and maximum edge size *i* for Dellso).

The order of accuracy for VTK Marching Cubes, SnapMC, Macet and Dual Contouring depends on the cell size *h.* We run our tests with grid refinement *h*_{i}
_{+1} = *h*_{i}
_{+2} and initial condition *h*
_{1}
*.* For Afront, the order of accuracy depends on parameter *ρ* thus the refinement is given by *ρ*
_{i}
_{+1} *= **ρ*
_{i}/2 with initial condition *ρ*
_{1}
*.* Our customized version of Dellso has an additional parameter *i* that controls the largest edge on the output mesh. In this case, the refinement formula is *λ*
_{i+1} = *l*_{i}/2. In the particular case of SnapMC, we set the snap parameter γ to its maximum value (*γ* =1/2). Even though the manufactured solution we selected is about as simple as can be imagined, comparing the formal order of accuracy with the observed one was enough to suggest bugs in two implementations. The observed order of accuracy of the examined properties is presented on Table 1.

#### 4.1.1 Algebraic distance

Section 3.1 shows that one expects second-order convergence for function value on vertices if linear interpolation is used. We define the following approximation error on *λ*
_{∞} norm:
TeX Source
$$E_k = \mathop {\max }\limits_{j = 1 \cdots n} |\lambda - f(v_j)|$$
where *λ* is the isovalue of interest *v*_{j} is a vertex of and *n* the number of vertices Figure 7(a) shows the vertex observed order of accuracy. VTK Marching Cubes, SnapMC have nearly quadratic convergence rates as shown in Figure 7(a). Afront shows a zero-order of accuracy though it presents very low error (in fact, the lowest in Figure 7(a)). This is due to the Catmull-Rom spline that is being used for surface approximation on the voxelized grid. Since it has cubic-order of accuracy, even for large values of p it can approximate with high precision the manufactured solution *f*. Next section shows that this is due to a poor choice for a manufactured solution. Dellso implementation has non-zero order of accuracy due to an outlier. Large values of *λ* causes bad approximations of the manufactured solution.

The Macet and Dual Contouring curves suggest that the algorithms converge to a fixed value. In fact, there was indeed a problem in the implementation that was affecting the convergence of Macet and Dual Contouring (specifically, we found a hard-coded limit in the number of steps in a root-finding procedure that was being triggered by the high resolution of the volume). Once fixed, we obtain the results shown in Figure 8(a). Macet and Afront now have similar behavior in the observed order of accuracy of vertex position (Figure 8(a)). This is because both methods use high-order interpolation with splines, not linear interpolation as assumed before (see Section 5).

#### 4.1.2 Normals

Section 3.2 shows that one expects first-order of accuracy for normal computations. We define the following approximation error using *L*_{∞} norm:
TeX Source
$$E_k = \mathop {\max }\limits_{j = 1 \cdots n} |\theta _{\sigma _j } |$$
where is the angle between the normal of the triangle *ρ*
_{j} and the normal of the point in *S* closest to the centroid of As shown in Figure 7(b), VTK Marching Cubes, Afront, SnapMC and Dellso have good observed order of accuracy above 0.8. However, only VTK Marching Cubes and Dellso present close proximity to linear. Macet and Dual Contouring once again do not present a consistent order Figure 8(b) shows the results after fixing both codes.

#### 4.1.3 Area

Although there is no formal order of accuracy for area, one expects *some* convergence for it (Section 3.3). We define the following approximation error:
TeX Source
$$E_k = |A(S) - A(\hat S_k)|$$
where *A* is the area function of a continuous or piecewise-linear surface. The results are shown in Figure 7(c). VTK Marching Cubes, Afront and Dellso present second-order of accuracy as shown in Figure 7(c). SnapMC accuracy is slightly better than quadratic due to poor approximation for large *h.* The error dropped faster than quadratic when the grid was refined for the first time. Macet and Dual Contouring exhibit once again unexpected behavior. Unlike the previous time, the curves now seem to diverge when *h* is too small. Once the bug is fixed the convergence curves changes, and they become quadratic (Figure 8(c)).

### 4.2 Detected Bugs

We were able to find and fix bugs in two of the implementations under verification, namely, Macet and Dual Contouring, using as manufactured solution a sphere centered at origin with radius 1. The new result curves are shown in Figure 8. The observed order of accuracy for Dual Contouring is quite satisfactory for all manufactured solution. In particular, the normal order of accuracy has the best rate among the methods. Macet improved for its results for area. On the other hand, it still has some issues related to normals, which perhaps indicates a need for more tests and verification. The new order of accuracy for algebraic distance (Figure 8(a)) does not tell us much about the correctness of the code because of the zero-th order of accuracy (same for Afront).

The zero-th order of accuracy might happen if the formal order of accuracy is zero-th order, in which case the observed order matches the formal order. It might also happen due to a poor choice for manufactured solution. If it is not complex enough, the implementation being tested may approximate exactly the solution and therefore there is no error within the approximation although another error source (truncation error, for instance) may show up. The next section presents a detailed discussion concerning MMS.

Although we managed to fix the Macet convergence problem, we were not able to do so in a way that preserves triangle quality (Figure 1). Two were the problems we found in the source code, and we proposed two solutions for one of them Table 2 shows that we could not find any combination that both fixed the convergence problem and preserved the triangle quality simultaneously. This sort of behavior raises the question if there is a theoretical problem that prevents both from being satisfied simultaneously, or it is just a matter of finding a better algorithmic fix. In both cases, further study and subsequent tests must be accomplished.

As we have shown, MMS is an effective means of diagnosing problems within the algorithms and implementations of isosurface extraction algorithms. In this work we have considered the two - algorithm and implementation − as one unit as one cannot always distinguish between the two if only limited information (source code and algorithmic details) is available. In this section, we present a more thorough discussion of the use of MMS, particularly for isosurface extraction.

**On the implementation and use of MMS.** One of the primary advantages of verifying simulation codes using MMS is that it is a nonintrusive method. MMS treats the code being verified as a blackbox, and so can be easily integrated into an existing test suite with little to no impact. However, MMS does not "see" the implementation, and so provides little direct information about where a particular bug might be when there is a discrepancy between the formal and observed orders of accuracy. In our experience, there are three main places where mistakes can happen: (1) in the design and construction of the manufactured solution, (2) in the coding of the algorithm being tested, and (3) in the evaluation and interpretation of the results. Mistakes on the evaluation of results have two flavors: misinterpretation or poor formal order of accuracy. The first heavily depends on testers' and experts' experience and ability to judge what a good result is. For example, should the normal observed order of accuracy for Afront and Macet on Figure 7(b) be considered linear (*p* = 0 *.*80 and *p* = 0.75 respectively)? The latter depends on a rigorous formal order of accuracy analysis of the algorithm considering all sorts of errors; even round-off errors may be significant. In fact, we spent more time on writing out rigorously the analysis of the formal order of accuracy and on searching for possible sources of error than on the tests themselves. This again highlights the fact that verification using MMS is a process: it is typical to go back to the white board and refine formal analyses before arriving at conclusive answers. Although the formal order of accuracy analysis might be a painful process, the literature has many results that can be promptly used. As a consequence, if one wishes to writes his own MC technique, for instance, his only concern is to write a test which exploits the results available within the literature.

**On the complexity of the manufactured solution**. The complexity of the manufactured solution can have a large influence on the effectiveness of verification. Suppose one chooses the manufactured solution to be *f*(*x* *y* *z*) = *x* + *y* + *k* *k* constant, instead of a sphere. Since MC-based techniques use linear interpolation, one expects the approximation to be exact regardless of any discretization parameter *h* *i.e.* *p* = 0 (notice that the evaluated error might be non-zero, implying there is some other error source that does not depend on *h*). Since such a function *f* is extremely simple, it might not trigger bugs that would otherwise reduce the observed order of accuracy. In our experiments, the (problematic) implementation of Dual Contouring achieved the formal order of accuracy for this particularly simple function (*p* = 0).

Another example on the influence of manufactured solution arose with in our examination of Afront. Because Afront uses Catmull-Rom splines, some simple isosurfaces will converge to within numerical error for very rough volumes, and the numerically observed order of accuracy will be much lower than expected. With an implicit function whose isosurfaces are spheres, we observed zero-th order of accuracy for Afront for algebraic distance. With a modified implicit function that included transcendental functions, MMS reveals that Afront does not have the expected convergence rate on the full interval, as shown in Figure 9. Notice that Macet has similar behavior. Additional tests are needed to determine the source of this behavior within both codes.

**On the order of accuracy**. In this paper, we have chosen to make our formal analysis as generic as possible to accommodate as many implementations under verification as possible. Although we are able to evaluate many codes using the same manufactured solution, when using MMS for a particular code, it is best to exploit as much detail about the algorithm as necessary. If the goal is to design a manufactured solution for verifying Marching Cubes-based techniques the manufactured solution should exercise all possible cases. Additionally, particular aspects of the manufactured solutions can be incorporated into the formal analysis. For example, the analysis for Afront becomes much more complicated if curvatures are not constant over the surface (in that case, its additional parameter *η* comes into play [23], and accurately bounding the triangle size is not practical).

The errors in Section 4.1 were measured at different locations on the mesh. Vertex convergence and Gaussian curvature were measured on triangle vertices, while normals were measured on the triangle centroid. More importantly, measurements performed at different locations may have different orders of accuracy. For example, Macet has cubic formal order of accuracy on vertices due to the spline approximation but quadratic formal order of accuracy on centroids.

In Section 3, we define the error using a pessimistic *λ*
_{∞} norm. This makes MMS a very sensitive technique. In fact, it can detect subtle off-by-one mistakes in grid sizes and interactions between node-centric and cell-centric reconstructions, even for simple manufactured solutions. In these cases, it is important not to infer incorrect conclusions.

The numerical estimates for MMS should be performed on as wide a range of parameter values as possible. In our tests, we used *h* ∊ (0.001,1.0) and observed that both faulty implementations performed appropriately for large values of *h*. Just as the implementations might only enter the asymptotic regime and achieve the formal convergences for small values of *h*, it might be that (as we have experienced) bugs only manifest themselves on sufficiently small values of *h*.

**On the limitations of the test**. MMS does not cover every aspect of verification for isosurface extraction. For example, an important aspect we do not know how to test with MMS is the topological correctness of an extracted mesh. This is challenging because there does not seem to be a good measure of convergence for topological properties such as the Euler characteristic or Betti numbers. A proper study of these issues is a natural avenue for future work.

SECTION 6

## Conclusions and Future Work

Because of its simplicity and effectiveness, we believe MMS could become a standard tool in the verification of scientific visualization software in the same that it has been adopted by the scientific simulation community as a trustworthy tool for assess code correctness. Using a simple manufactured solution, we were able to reveal bugs that prevented the convergence of some mesh properties of two publicly available isosurfacing codes. In particular, the by-products of the verification process, namely a continuous refinement of mathematical analysis of the algorithm's behavior and a numerical comparison of the results of the implementation against a known solution are valuable in their own right, and should be published together with new algorithms.

We are investigating the applicability of MMS to other visualization techniques such as streamline generation and volume rendering. In particular, MMS should clarify assumptions and errors intrinsic in these visualizations, a topic that has received recent attention [10]. More importantly, we hope the examples presented here will encourage the adoption of MMS by the visualization community at large, increasing the impact of its contributions to a wider audience.

### Acknowledgments

We thank Lauro Lins and Tom Peters for help with the paper, João Comba for help with the Macet code, and Tamal Dey and Joshua Levine for the customized version of DelIso used in this work. This work was supported in part by grants from NSF (grants IIS-0905385, IIS-0844546, ATM-0835821, CNS-0751152, OCE-0424602, CNS-0514485, IIS-0513692, CNS-0524096, CCF-0401498, OISE-0405402, CCF-0528201, CNS-0551724), DOE, IBM Faculty Awards and PhD Fellowship, the US ARO under grant W911NF0810517, ExxonMobil, and Fapesp-Brazil (#2008/03349-6).