Mesh Simplification With Appearance-Driven Optimizations

In this work, we propose a novel solution for simplifying texture-mapped three-dimensional (3D) meshes. It simplifies a 3D triangular mesh to optimally preserve the visual appearance of the original texture-mapped model at reduced vertex budgets. While taking the prevalent strategy of iterative edge contraction, the proposed scheme is novel in that it takes into account the local texture image characteristics when prioritizing local mesh simplification operators, and conducts closed-form optimization when computing the texture coordinates of each replacement vertex. Outstanding performance of the proposed scheme is demonstrated both qualitatively and quantitatively by experimental results. Further, validity of the proposed algorithmic components is well proved through ablation study.


I. INTRODUCTION
While high realism is increasingly demanded in many graphics applications including film production, video gaming, digital city, scientific modeling and visualization and so forth, there still exists a gap between high volume of data and limited resources of storage, bandwidth and/or computation in many contexts. For instance, graphics applications on mobile terminals may only afford to use lightweight three-dimensional (3D) models for efficient storage, transmission, rendering and editing. In order to reduce this gap, we focus on the simplification of 3D triangular meshes with texture attributes. Three-dimensional model simplification techniques adapt a 3D model created in high resolution to different levels of detail (LODs) for different application scenarios. This field has been intensively investigated in the past, with many good algorithms published. Early 3D model simplification algorithms strived to maintain the geometric fidelity at a reduced vertex count. Later, simplification algorithms were proposed to minimize both the geometric error and the texture deviation The associate editor coordinating the review of this manuscript and approving it for publication was Walter Didimo . in order to preserve the appearance of the texture-mapped model. In general, the texture deviation was controlled less well than the geometric error by those algorithms.
In this work, we propose a novel and complete solution for simplifying texture-mapped 3D meshes. In order to maximally preserve the appearance of the original texture-mapped model, the proposed scheme considers both mesh and texture features to prioritize the local simplification operators and minimizes texture deviations over the simplified surface. Key features of the proposed scheme include: • Appearance-driven mesh simplification. Texture dependent factors are introduced to the simplification cost metric to modify the global vertex distribution such that less texture deviation will be produced in local regions with more texture details. Further, closed-form optimization is proposed for each replacement vertex to derive its optimal texture coordinates which minimize the local texture deviation on the surface.
• Generality of framework. Though the proposed scheme uses iterative edge contraction for the mesh simplification, it is generally extensible for other simplification operators (e.g., vertex decimation, vertex clustering) with minor adaptations.

II. RELATED WORK
The field of 3D model simplification has been researched for more than two decades with many algorithms published. Due to the space limit, we briefly review some of those algorithms. For comprehensive surveys of algorithms published in early years, we refer to several good works [1]- [4]. Most early 3D model simplification algorithms focus on preserving geometric fidelity of the simplified model, as reviewed in Section II-A. Attributes other than geometry, especially texture attributes, have also been taken into account by 3D model simplification algorithms for better appearance preservation, as reviewed in Section II-B.

A. GEOMETRY-CENTRIC SIMPLIFICATION
Although some algorithms simplify 3D models through global optimal point sampling and redistribution [5] or surface partitioning and approximation [6], most existing algorithms work by repeatedly applying simplification operators until a distortion threshold or a desired decimation rate is reached. The most commonly used simplification operators include vertex decimation, vertex pair contraction and vertex clustering. Vertex-decimation-based algorithms [7], [8] remove a vertex and re-tessellate the resulting hole at each simplification step. Vertex-pair-contraction-based algorithms [9]- [14] contract an edge or even a pair of unconnected vertices to one vertex and update the local topology correspondingly at each simplification step. Vertex-clusteringbased algorithms [15]- [17] merge a cluster of vertices to one and update the local topology correspondingly at each simplification step.

B. APPEARANCE-CENTRIC SIMPLIFICATION
The majority of these algorithms use iterative edge contraction (or named edge collapse) to simplify a mesh and determine the texture coordinates of each replacement vertex with care. Garland and Heckbert [18] extend their original algorithm [12] by incorporating attributes into the quadric error metric (QEM). Hoppe [19] proposes an improved quadric error metric leading to improved storage use, computational efficiency and resultant model quality. Cohen et al. [20] build successive mappings between LODs and convert the 3D positioning to two dimensional (2D) positioning plus one-dimensional (1D) offsetting of each replacement vertex. Cohen et al. [21] further improve their work by introducing a texture deviation metric and locally finding the new vertex' texture coordinates based on this metric. Williams et al. [22] prioritize the edge collapses based on a perceptual model taking into account texture deviation, contrast of illumination, dynamic lighting and so forth.
There are other algorithms that refer to the texture images or the rendered images during the mesh simplification. Lindstrom and Turk [23] perform image-driven simplification that utilizes the rendered model images to compute the edge collapse cost. When computing the edge or half edge collapse cost, Shao et al. [24] consider the local texture error and Gonzalez et al. [25] penalize collapsing edges that cross borders in the texture image or have large neighborhood surface areas. Gonzalez et al. [26], [27] define the cost of collapsing an edge as the resultant change in structural appearance of the textured model. The methods of Lindstrom and Turk [23] and Gonzalez et al. [26], [27] require surface rendering at multiple viewpoints for the simplification cost computation, while it may be hard to make a fair, concise and sufficient view sampling that is universally applicable to models of arbitrary geometrical complexity. Moreover, the iteratively conducted 3D model rendering may cause intensive computation. It is also worth noting that, in the methods of Shao et al. [24] and Gonzalez et al. [25]- [27], the texture coordinates of a replacement vertex have not been sufficiently optimized yet.
Other strategies have also been proposed for appearance preservation in texture-mapped model simplification. García and Patow [28] propose a texturing technique named Inverse Geometric Textures (IGT), which defines texture coordinates for all the vertices in a simplified model to preserve texture details from a high resolution reference model. Chen and Chuang [29] and Coll and Paradinas [30] modify the texture image to minimize the texture distortion caused by each edge collapse. It is worth noting that the methods of García and Patow [28], Chen and Chuang [29] and Coll and Paradinas [30] are not well suited for embedded LOD construction, as the texture coordinates of inherited vertices or the texture image content keeps changing during the iterative simplification process.
Cheng and Boulanger [31] simplify a mesh and adaptively compress its texture image based on scale-space filtering (SSF). They parameterize the mesh into an image and, through SSF on this image, identify feature points to be included in the simplified mesh. The texture image is divided into blocks and each block is compressed to a quality level determined by the number of feature points falling inside. The mesh parameterization conducted in this algorithm is quite restricted, limiting its applicability on geometrically complex models.

III. ALGORITHM OVERVIEW
Given a texture-mapped model, we simplify the mesh to maximally preserve the model appearance at reduced vertex budgets. As the distortion in model appearance comes from geometry approximation error and texture deviation on the surface, we strive to minimize both during the simplification process. The pseudo-code is given in Alg. 1 and further explained in the following.
The mesh simplification is based on iterative edge contraction. The contraction cost is computed for every edge and the edge with the minimum cost is contracted to a replacement vertex at each iteration. Following Garland and Heckbert [12], we compute the geometric position of a replacement vertex by minimizing a quadric error metric. Beyond that, we make particular efforts to keep the quality of surface parameterization or, equivalently, minimize the texture deviation on the surface. Specifically, we propose M .contract_Edge_And_Update_Queue(e, Q, I ); 7: end while to 1) incorporate the local textural richness as a factor into the edge contraction cost metric, favoring better preservation of triangles in surface regions with more texture details, and 2) derive the texture coordinates of a replacement vertex using closed-form optimization.
It is worth noting that, while we focus on mesh simplification in this work, the proposed method may directly substitute into edge-contraction based progressive mesh encoders (as surveyed by Peng et al. [32] and Maglo et al. [33]).

IV. ALGORITHM DETAILS
As stated in Section III, we use the same method for computing each replacement vertex' position as many of the others (e.g., Garland and Heckbert [12]), except that special processing is needed when one or two of the end vertices fall on a boundary between two texture patches (see Section IV-A). Uniquely, we control the texture deviation by an optimized quadric error metric based on both mesh and texture image features and a closed-form local mapping optimization method to determine the texture coordinates for each replacement vertex. Note that, in this work, we represent all texture image colors in the CIELAB (or L * a * b * ) space, a perceptually uniform color space.

A. OPTIMIZED QUADRIC ERROR METRIC
A standard mesh simplification algorithm based on the quadric error metric works in the following way.
Initially, it associates with each vertex in the original mesh the list of facets incident on that vertex. Later on, whenever it contracts an edge, (v 1 ,v 2 ), to a replacement vertex, v, it forms the associated facet list of v by combining those of v 1 and v 2 .
If it contracts an edge (v 1 ,v 2 ) to v, the approximation error, ξ 0 (v), at v is computed by a quadric error metric [12] that sums up the squared point-plane distances from v to the associated facets of v, i.e., where The edges are always ordered on their contraction costs, and the edge with the least contraction cost is always picked for contraction at each iteration.
A triangular mesh is usually a piecewise linear approximation to a smooth or piecewise smooth surface, and the approximation accuracy is often positively related to the sampling density. With the QEM described above, denser vertex populations, and correspondingly higher geometric approximation accuracies, will be produced for regions with more shape varieties. This is preferable since human eyes are usually more sensitive to distortions in sharp features than in smooth areas.
Following a similar rationale, texture coordinates at mesh vertices provide a piecewise linear approximation to a smooth or piecewise smooth surface parameterization, and we should retain denser samplings for surface regions with more color varieties (texture details) to achieve better parametric approximation accuracies and therefore less texture deviations for those regions. For this purpose, we propose to incorporate into the QEM (defined in (1)) two factors, d c and d e , which measure the color variety and the edge saliency, respectively, for a local area in the texture image, I . Specifically, we define the local region (or set of pixels), A, in the texture image for an edge, (v 1 , v 2 ), as the parameterized 1-ring neighborhood of this edge (i.e., the union of the parameterized 1-ring neighborhoods of the two end vertices, v 1 and v 2 ), denote the corresponding region in the edge map (obtained by applying the Canny operator [34] on the L * -channel), Y , of the texture image by E, and update the error metric as As d c and d e are constants for this edge contraction, the replacement vertex, v 1 o , and the contraction cost,   (c), respectively, v 1 is a smooth vertex with incident corners' texture coordinates of t , v 2 a crease vertex with t 1 and t 2 , and v 3 a corner vertex with t 1 , t 2 and t 3 .
Since the new cost metric takes into account the texture image characteristics as well, the simplification process leads to a texture-adaptive primitive distribution on the mesh surface. As can be seen from Fig. 1, more vertices tend to distribute in surface regions with more texture details.

B. LOCAL MAPPING OPTIMIZATION
Following Isenburg and Snoeyink [35], we classify vertices based on the configurations of their incident corners' texture attributes. A vertex is a smooth vertex if all its incident corners have the same 2D texture coordinates; a vertex is a crease vertex if its incident corners are divided to two sets each containing contiguous corners sharing the same 2D texture coordinates; a vertex is a corner vertex if its incident corners have more than two different 2D texture coordinates. This classification of vertices is illustrated by Fig. 2 where examples of smooth vertex, crease vertex and corner vertex are shown in Fig. 2 (a), (b) and (c), respectively. It is often the case that a 3D surface is parameterized onto a discrete set of patches in the 2D texture image. On the texture mapped 3D surface, a smooth vertex is inside a texture patch, a crease vertex is on the common boundary of two adjacent texture patches, and a corner vertex is at the intersection of three or more texture patches.
Assume that we contract edge (v 1 , v 2 ) to replacement vertex v o . If both v 1 and v 2 are smooth vertices, v o is also smooth. Its position is computed by the method of Garland and Heckbert [12]. Its texture coordinates are initially assigned the average of v 1 's and v 2 's texture coordinates and further adjusted by our local mapping optimization method, as shown in Fig. 3(a). If one of v 1 and v 2 is a smooth vertex and the other is not, v o inherits the type, the geometric position and the incident corners' texture attributes from the non-smooth one, as shown in Fig. 3(b). If both v 1 and v 2 are crease vertices on the same crease, as shown in Fig. 3(c), the geometric position and the corners' texture attributes of v o are obtained by averaging those of v 1 and v 2 . For other cases, contracting edge (v 1 , v 2 ) may introduce serious texture deviation and is therefore prohibited in our scheme. Details of our proposed local mapping optimization method are given in the following subsections.

1) TEXTURE DEVIATION
We adapt the texture deviation metric proposed by Cohen et al. [21] to measure the mapping distortion caused by an edge contraction. Given a 2D point, t, in the texture space, the Euclidean distance between its 3D positions on two simplified versions of the mesh, M i and M j , gives the texture deviation, E i,j (t), formally defined as: where F i (F j ) is the function mapping the 3D surface of M i (M j ) to the 2D texture domain and F −1 i (F −1 j ) does the reverse. Cohen et al. [21] measure the texture deviation at the vertices of the overlayed parameterizations of M i and M j and take the maximum as the texture deviation between M i and M j . Although this provides an upper bound, it may not completely describe the texture deviation for the whole affected region in the texture domain. Instead, we integrate the texture deviation at all the locations in texture region, R, that is affected by the simplification from M i to M j . Specifically, we propose to compute the texture deviation between M i and M j as Denoting the original model as M 0 , the incremental texture deviation between any two adjacent LODs M i−1 and M i , i > 0, may be non-zero only in the parameterized 1-ring neighborhood of the contracted edge.

2) TEXTURE COORDINATE OPTIMIZATION
Suppose that we are to contract edge e = (v 1 , v 2 ) in M i−1 to produce M i . We denote the k 1-ring neighbor vertices of e (i.e., vertices connected to v 1 and/or v 2 by an edge) as b 1 , b 2 , . . . , b k , and their parameterized positions as r 1 , r 2 , . . . , r k , respectively. The texture domain region, R i−1 , affected by this edge contraction is the polygonal region bounded by r 1 , r 2 , . . . , r k . Further, we assume that v 1 and v 2 are parameterized to t 1 and t 2 , respectively. Cohen et al. [21] confine the texture coordinates of the replacement vertex to the kernel of the polygon R i−1 , which is formed by the intersection of a set of half-planes, each inside a distinct edge of R i−1 . They consider a discrete set of texture domain locations which includes the parameterized positions of the two end vertices, their middle point and the center of three selected corners of the kernel polygon. In contrast, our objective is to find the optimal texture coordinates, t o , for the replacement vertex, v o , from the full region of R i−1 such that In order to make a closed-form formulation, we first construct a fitting parametric surface f i−1 (t) = (x(t), y(t), z(t)) : R 2 → R 3 over the region R i−1 . Each of x(t), y(t) and z(t) is modeled as a second-order polynomial over the parameter domain, which is obtained by the least square fitting to the corresponding coordinate components of v 1 , v 2 , b 1 , b 2 , . . . , b k at the parameter locations, t 1 , t 2 , r 1 , r 2 , . . . , r k , respectively. We are motivated to use second-order polynomials by the fact that polynomial surfaces (e.g., Bézier surfaces and many subdivision surfaces) are prevalently used for global or local mesh parameterization and, in particular, moving least squares (MLS) is frequently used to approximate local surfaces with second-order polynomials over local parameter domains. Correspondingly, we modify the texture deviation metric on R i−1 to: We approximate the computation of ε 1 i−1,i (R i−1 ) by an elaborate sampling on R i−1 . Assuming that the replacement vertex is parameterized to t, we consistently sample each triangle incident on t in the texture domain and its corresponding triangle incident on v o on the surface of M i .
Specifically, we consistently sample every triangle using the same set, L = {l 1 , l 2 , . . . , l m }, of barycentric coordinates where l j = (λ 1,j , λ 2,j , λ 3,j ) gives the barycentric coordinates of the j-th (j = 1, 2, . . . , m) sample. Correspondingly, we modify our texture deviation metric as with % standing for the modulo operator, operator · means dot product between two 3-tuples and a h is the area of h,v o . Finally, we obtain t o as  We solve this constrained nonlinear optimization problem using a quasi-Newton method. We set the initial value of t o to the average of t 1 and t 2 . If no feasible solution was found, i.e., the optimalt found by argmin t ε 2 i−1,i (R i−1 , t) is out of R i−1 , we simply use the initial value.
In order to demonstrate the effectiveness of the proposed local mapping optimization technique, we show partially in Fig. 4 the Panda mesh (see Section V-A) simplified by 50% using our scheme (a) without and (b) with local mapping optimization. For reference, the original is shown in Fig. 4(c). It is evident from Fig. 4 that the proposed local mapping optimization technique significantly reduces the texture deviation, especially around the outer boundaries of the eye regions and the line across the chest.

A. TEST MODELS AND DISTORTION METRIC
Six texture-mapped models are used in our experiments. They are Panda, House, Cartoon-man, Earth, Spiderman and Dragon, whose geometric meshes and texture images are shown in Fig. 5. The original Cartoon-man model is from Intel. The original House model is from http: //www.turbosquid.com, the original Dragon model is from https://graphics.stanford.edu/data/ 3Dscanrep/ and the other original models are from http://www.cgjoy.com. Note that, for our easy use, we have pre-processed the original geometric models and the original texture images and designed a texture for Dragon using commodity software tools. More information about these processed models is provided in Table 1. All the textured models are rendered with bilinear filtering in our experiments.
For the distortion measurement, we render both the original and the simplified 3D texture-mapped models using the same lighting from a randomly selected set of view points. At each view point, we compare the rendered images of the original and the simplified models using the well known SSIM (structural similarity) index [36]. We use the average SSIM index over all the view point samples to measure the quality, which is inversely related to the distortion of the simplified model. For convenience, we call it distortion metric as well.
We run QSLIM, NQM, IMGD, MAYA and our proposed mesh simplification with appearance-driven optimizations (abbreviated as MSADO) method on the test models to simplify the meshes with texture attributes. Selected visual results are shown in Fig. 6, where the corresponding algorithm and decimation rate are marked under each model. For better illustration, both whole model images and enlarged local regions are shown. From Fig. 6, we observe that MSADO leads to better preserved model appearance for all the three test models.
Further, we measure the SSIM indices at various vertex decimation rates for all the test models, which are plotted in Fig. 7. From the curves in Fig. 7, we observe that MSADO leads to much higher SSIM indices than IMGD. Compared with NQM, QSLIM and MAYA, MSADO also produces higher SSIM indices, especially at higher decimation rates. These curves again demonstrate the advantage of MSADO over IMGD, NQM, QSLIM and MAYA in simplifying 3D meshes with texture attributes. It is worth mentioning that the proposed algorithm preserves the original model's surface area very well. For instance, even at a high decimation rate of 80%, the average ratio of area between the simplified and the original models is around 99% for our test models.
The running time for Panda, Spiderman, House, Cartoonman, Earth and Dragon at the rate of 85%, 80%, 80%, 80%, 80%, 80%, respectively, is shown in Table 2. The experiments are conducted on a desktop computer with 12GB memory, 3.40GHz Intel(R) Xeon(R) CPU. Table 2 shows that QSLIM achieves the best efficiency, MSADO runs slower than NQM, while IMGD consumes the most time. The computation cost of MSADO comes mainly from the texture coordinate optimization which involves parametric surface fitting and constrained nonlinear optimization. Note that our current research code for MSADO is not optimized yet.
Finally, we emphasize the effectiveness of our algorithm on the large Dragon model by the visual results in Fig. 8, where the corresponding decimation rate is marked under each model. From Fig. 8, we observe that a fair-quality approximation already shows up at an extremely high decimation rate of 99%, and the simplified model is almost indistinguishable from the original at the high decimation rate of 80%.

C. USER STUDY
We asked 36 subjects to evaluate the visual quality of simplified models. Specifically, we simplified each test model by a couple rates using different methods, and asked the subjects to grade the visual similarity of each resultant model to the original. In our design, the score ranges from 1 to 10 with 1 representing the lowest similarity and 10 the highest. Statistics of the responses to the questionnaire is shown in Table 3. We observe from Table 3 that, for almost all the models and decimation rates, MSADO achieves the highest mean and the lowest variance of the similarity scores. This user study again corroborates the outstanding performance of MSADO.

D. ABLATION STUDY
In this section, we demonstrate the effects of the major novel components in our proposed scheme. As the baseline, we use the basic QEM-based method [12] for geometry simplification and obtain a replacement vertex' texture coordinates by averaging those of the two end vertices. Starting from the baseline, we incrementally add our proposed components to obtain Baseline+W (with d c and d e introduced as weights to the edge contraction cost metric), Baseline+W+TC (with texture coordinate optimization further introduced).
Selected results of Baseline, Baseline+W and Baseline+ W+TC on Panda, House and Spiderman are shown in Fig. 9. Comparing the results of Baseline+W and Baseline, we observe reduced texture deviation around regions with sharp textural features; comparing the results of Baseline+W+TC and Baseline+W, we observe further reduced texture deviation.

VI. CONCLUSION AND FUTURE WORK
In this paper, we have proposed an original scheme to simplify texture-mapped 3D triangular meshes. It strives to maximize the preservation of the original model appearance at any mesh decimation rate.
There are a few most important novelties of the proposed scheme. Firstly, we optimize the edge contraction cost metric to guide the adaptive distribution of 3D primitives on the simplified surface according to both geometric and textural characteristics. Secondly, we propose to optimize the texture coordinates of each replacement vertex using a closed-form formulation of texture deviation on a more complete sampling of a local texture domain. As a result, the proposed scheme achieves outstanding appearance-preservation performance, as experimentally demonstrated. Further, it should be straightforward to extend the proposed scheme for other types of 3D model simplification operators as well.
In the future, we may extend our current scheme for optimal progressive compression of texture-mapped 3D models. It is also interesting to investigate adaptive simplification of texture images in association with 3D meshes. XIANGXU MENG (Member, IEEE) received the B.Sc. and M.Sc. degrees in computer science from Shandong University, China, and the Ph.D. degree in computer science and technology from the Institute of Computing Technology, Chinese Academy of Sciences. He is currently a Professor of computer science with Shandong University. He has published more than 100 articles and four books, covering topics such as photorealistic rendering, geometric modeling, intelligent natural interaction, and service computing. His research interests include computer graphics, computer-aided design, human-computer interaction, and grid computing. He is currently a Professor with the School of Software, Shandong University, China. His research interests mainly reside in digital geometry processing, virtual/augmented reality, and image/video analysis.