Spatial variation in color is arguably the most challenging remaining obstacle towards achieving seamlessness in tiled multi-projector displays. Color is a three dimensional quantity defined by 1D luminance (defining brightness) and 2D chrominance (defining hue and saturation). Variation in color across a tiled display can break the illusion of a single display despite perfect geometric registration. This problem is becoming more pronounced with the advent of commodity projectors that cannot afford to use expensive optical elements like precision filters and fresnel lenses to address the problem optically.

The entire range of color (both chrominance and luminance) that can be reproduced by a display is called its 3D color gamut. The color gamut can also be specified separately by a chrominance gamut (defining the entire range of chrominance that can be reproduced by the device), dynamic range (defining the range of luminance that the device can produce), and the white point (the chrominance of the white). In [9], [7] Majumder et al. show that the spatial color variation in multi-projector displays is due to the spatial variation in the color gamut across the display and classify it in three different categories: intra-projector variation (within a single projector), inter-projector variation (across different projectors), and overlap variation. Existing methods address only parts of these problems and hence cannot provide a general and comprehensive solution. [11], [15], [16], [17] ignore the intra-projector luminance and chrominance variation and propose a gamut matching method to achieve color balancing across the projectors. [1] shows that achieving gamut matching is not scalable for many projectors. Further, gamut matching considerably restricts the final achievable gamut reducing the color quality and resolution of the display [19]. [8] ignores both inter- and intra projector chrominance variations and matches the luminance transfer functions (commonly called gamma functions) to achieve a luminance balancing across the projectors. Blending or feathering techniques usually do not capture the spatial color variation across the display and hence ignore both intra and inter projector chrominance and luminance variations. They ramp the intensity from each projector smoothly from 0 to 1 in the overlap region. Multiple projectors overlapping in the region have complementary blending functions to assure a smooth transition from the color of one projector to another. This can be done either in software [14] or using specialized hardware mounted on the light path from the projector [6]. Since these methods do not address the spatial variation in the color in a comprehensive manner, they result in softening of seams rather than achieving a seamless result (Figure 5). In [10] Majumder and Stevens address the spatial variation of luminance to achieve a perceptually smooth variation across the display. However, since the spatial variation in chrominance is not addressed, color blotches are still visible, especially for flat colors when human is more sensitive to the artifacts (Figure 5).

Commercial vendors who build such displays for entertainment and simulation applications have also been trying to address the color variation problem for a long time. Most earlier systems (e.g. the ones built by Barco and Fakespace Labs) used very high-end projectors, almost a couple of orders of magnitude higher in cost than commodity ones, that assure higher color quality. For example, precision filter manufacturing assures better color matching across the projectors, and the use of Fresnel lens assures significant spatial uniformity in intensity within a projector. Following this, a standard color management system is used to achieve color uniformity. Note that such systems may use camera assistance to achieve geometric registration, but their color registration process is not automated via the camera feedback. Since the color variation is not even captured at a high resolution, these systems cannot correct the spatial variation in color entirely, especially for testing cases of images with white backgrounds. More recently, some vendors have started using automated camera based registration techniques (e.g. Scalable Displays and Mersive). In this case also, to assure better color uniformity, projectors that are at least an order of magnitude higher in cost are used. These systems usually use a combination of gamut matching, edge blending and luminance correction methods [10], [14], [16] to remove the remaining color variations, acceptable for less challenging patterns. However these methods restrict the chrominance gamut of the projectors to be within the common chrominance gamut of all of the projectors, degrading the resulting color quality. Though this yields acceptable results when using higher end projectors, it can be absolutely crippling for commodity projectors. On the contrary, our work uses extremely commodity projectors and demonstrates much superior color seamlessness than when using high-end projectors, in the presence of much more severe spatial color variations. Hence, our work is critical to push the frontier of the technology for entirely seamless and yet extremely affordable multi-projector displays.

In this paper, we present a new method that addresses spatial variation in both luminance and chrominance in tiled projection-based displays. Our method morphs the spatially varying color gamut of the display in a smoothly constrained manner while retaining the white point. We use perception based constraints that automatically assure a visually pleasing appearance. The gamut morphing is achieved by a smooth morphing of the chrominance gamut followed by a smoothing of the luminance. Interestingly, the chrominance morph can also be achieved by changing only the luminance at every pixel, but in a manner that is carefully controlled across different projectors in the overlap region. Together they achieve, for the first time, a true color seam-lessness following the application of our method where the number of projectors making up the display cannot be detected (Figure 1 and 7). The luminance correction proposed by Majumder and Stevens in [10] results in per projector luminance attenation maps (LAM). Since our chrominance morphing is also achieved by modification of the luminance, our method also results in per projector attenuation maps, but achieves chrominance correction also. Hence integration of our algorithm in the online process to a system that has already used the luminance correction is trivial and can be achieved at an interactive rate of 30 frames per second on a GPU. We demonstrate our method on both planar and cylindrical displays made of a large number of projectors and show their superior results when compared to the state of the art methods Table 1 summarizes the different color variations we address when compared to other existing methods. Our assumption of negligible intra-projector chrominance variation is due to earlier empirical studies that show chrominance to be nearly spatially constant within most projectors.

We use the CIE XYZ color space for all of our color computations. A color in the CIE XYZ color space is defined by its 3D coordinates (*X* *Y* *Z*), more commonly called the tristimulus values *Y* is the *luminance* of a color and the *chrominance* of a color is given by its chromaticity coordinates, (*x* *y*), defined as
TeX Source
$$(x,y) = \left({{X \over {X + Y + Z}},{Y \over {X + Y + Z}}} \right).$$

In other words,
TeX Source
$$(X,Y,Z) = (xB,yB,(1 - x - y)B)$$
where *B* = *X* + *Y* + *Z* which we call the tristimulus brightness. The range of (*x* *y*) visible to the human eye is the chromaticity chart.

**Scaling of Color**: All colors lying on a vector (*X* *Y* *Z*) can be represented as *k*(*X* *Y* *Z*) where *k* is a scale factor. From Equation 1, it can be seen that the chrominance of all these colors are the same, but B and Y are scaled to *kB* and *kY* respectively. Conversely, for any two colors with the same chrominance, the ratio of their Y and the ratio of their B are the same.

**Addition of Colors**: The addition of two colors, (*X*_{1} *Y*_{1} *Z*_{1}) and (*X*_{2} *Y*_{2} *Z*_{2}) to create a new color (*X*_{3} *Y*_{3} *Z*_{3}) is given by the vector addition of these two colors in the CIE XYZ space, i.e.,
TeX Source
$$(X_3,Y_3,Z_3) = (X_1 + X_2,Y_1 + Y_2,Z_1 + Z_2)$$
Clearly, Y and B of the new colors are given by
TeX Source
$$Y_3 = (Y_1 + Y_2;{\rm{ }}B_3 = B_1 + B_2.$$
Further, using equations 1 and 2, the chrominance of the new color is given by
TeX Source
$$(x_3,y_3) = \left({{{x_1 B_1 + x_2 B_2 } \over {B_1 + B_2 }},{{y_1 B_1 + y_2 B_2 } \over {B_1 + B_2 }}} \right)$$
Note that and are the barycentric coordinates of the new chrominance (*x*_{3} *y*_{3}) with respect to (*x*_{1} *y*_{1}) and (*x*_{2} *y*_{2}). This result can be generalized to *n* colors, where the chrominance of the new color lies within the convex hull of the chrominance of the *n* constituting colors.

A multi-projector display is a display made of *M* projectors, each denoted by *P*_{j}, 1 ≤ *j* ≤ *M*. We denote the pixel coordinates of each projector with (*p* *q*) and the display coordinates by (*s* *t*). We assume that the relationship between the projector and the display coordinates has been discovered apriori using a geometric registration method like [4], [2]. Hence, the display coordinate (*s* *t*) is related to the coordinate (*p*_{j} *q*_{j}) of projector *P*_{j} by (*s* *t*) = *Gj*(*p*_{j} *q*_{j} *Gj* can be deciphered using any standard geometric calibration techique [4], [18], [12], [2].

In a display, let the color formed by a channel input *i*_{l} = 1 be (*X*_{l} *Y*_{l} *Z*_{l}) and hence *B*_{l} = *X*_{l} + *Y*_{l} + *Z*_{l}. Assuming linear displays, as *i*_{l} changes from 0 to 1, the colors are given by *i*_{l}(*X*_{l} *Y*_{l} *Z*_{l}). Hence, from Section 2, their chrominance remains constant and *B*_{l} is scaled. The three channels of the display form three vectors in the XYZ space. Hence, all of the colors reproduced by the display are given by the parallelepiped spanned by the three vectors (*X*_{l} *Y*_{l} *Z*_{l} *l* ∊ {*r* *g* *b*} where {*r* *g* *b*} are red, green, and blue color channels. This is called the *3D color gamut* of the display. Alternatively, this gamut can also be defined by the 2D chrominance gamut, the tristimulus brightness and the chrominance of the white produced by *i*_{r} = *i*_{g} = *i*_{b} = 1. The 2D chrominance gamut is given by the triangle *T* given by (*x*_{l} *y*_{l} *l* ∊ {*r* *g* *b*}. The tristimulus brightness of the white *B*_{W}, is given by *B*_{W} = ∑_{l} *X*_{l} + *Y*_{l} + *Z*_{l} = ∑_{l} B_{l}. And the chrominance of the white, (*x*_{W} *y*_{W}) is given by
TeX Source
$$(x_W,y_W) = \sum\limits_l {{{B_l } \over {B_W }}(x_l,y_l).}$$
**Single Projector Display**: Empirical data in [9] shows that the color within a projector shows spatial dependency due to the vignetting effect. The vignetting effect is a spatial fall off of brightness from around the center to the fringes of the projector. However, this is a channel independent effect and hence affects the *B*_{l} of all the different channels in the same manner. Hence *B*_{l} at any pixel (*p* *q*) is given by
TeX Source
$$B_l (p,q) = V(p,q)B_{l.}$$
Where *V*(*p* *q*) is the vignetting of the projector at pixel (*p* *q*). Thus, the spatially varying tristimulus brightness of the white is given by
TeX Source
$$B_W (p,q) = \sum\limits_l {V(p,q)B_l .}$$
However, since the vignetting factor is a channel independent scale factor, it does not affect the 2D chrominance gamut *T* which is spatially constant.

**Multi-Projector Display:** Let us now consider the overlap of *N* projectors and a coordinate (*s* *t*) in the overlap region. Thus, the *B*_{l} for each channel in the overlap region is given by
TeX Source
$$B_l (s,t) = \sum\limits_{j \in N} {V_j (p_j,q_j)B_{l_j } .} $$
This indicates a spatial variation in *B*_{W} given by
TeX Source
$$B_W (s,t) = \sum\limits_l {\left({\sum\limits_{j \in N} {V_j (p_j,q_j)B_{l_j } } } \right)}$$
However, the chrominance of each channel will now depend on the proportions of the resulting *B*_{l} from different channels. Since the vignetting is different for different projectors *B*_{l} will change in a channel dependent manner in the overlap region thus creating spatially varying chrominance gamut *T*(*s* *t* *T*(*s* *t*) is defined by the three primaries (*x*_{l}(*s* *t*) *y*_{l}(*s* *t*)) as
TeX Source
$$(x_l (s,t),y_l (s,t)) = \sum\limits_j {{{B_l (s,t)} \over {B_W (s,t)}}(x_{l_j },y_{l_j })}$$
Note that the variation in the *T*(*s* *t*) stems from the variation in the *B*_{l} and *B*_{W} rather than the (*x*_{lj} *y*_{lj}) which are constant for each projector *P*_{j}. From Equations 9 and 10, we see that this variation in turn depends on the relative shape of the vignetting effect *V*_{j}(*p*_{j} *q*_{j}), and the brightness *B*_{lj}, of the contributing projectors, both of which can show a large variation in any tiled display [9] leading marked variation and discontinuities of the chrominance in the overlap region. Similarly *B*_{W} (*s* *t*) also shows visible discontinuities (Figure 4).

Our goal is to remove such discontinuities to create a perceptually seamless appearance. The key to this lies in controlling the *V*_{j} and *B*_{lj} from each projector, by scaling their contributions at each pixel precisely, to realize a smooth change in both *B*_{W}(*s* *t*) and *T*(*s* *t*). Further, we want to constrain this per-pixel scaling based on strong foundations of limitations of the human perception, so that they are not distracting to the human eye. This will result in a smooth morphing of the 3D gamut from one pixel to another ensuring perceptual seamlessness. We present a new algorithm to achieve such a per-pixel scaling that assures a smooth 3D gamut morphing. Our algorithm has four different steps: (a *Per Projector White Point Balancing*; (b *Per Pixel Chrominance Gamut Morphing*; (c *Per Pixel Perceptual Luminance Constraining*; and (d *Per Pixel Bezier Based Luminance Smoothing*.

### 3.1 Per Projector White Point Balancing

It is a well known fact that we humans are very sensitive to the white point of the displays [5]. So, the first step in our method is a white point balancing for each projector. Let (*x*_{D} *y*_{D}) be the desired white point. Using equation 6, we find for each projector a per-channel scale factor α_{l}, 0 ≤ α_{l} ≤ 1 such that
TeX Source
$${{\Sigma _l \alpha _l B_l (x_l,y_l)} \over {\Sigma _l \alpha _l B_l }} = (x_D,y_D),$$
where (*x*_{D} *y*_{D}) is the desired white point. We first fix α_{r} = 1 and then solve the two linear equations resulting from the above equations to find α_{g} and α_{b}. If this leads to an α which is not within 0 and 1, then we repeat the process with either *α*_{g} = 1 or *α*_{b} = 1. Note that since the vignetting effect scales *B*_{l} and *B* similarly at any pixel, the above equation will yield the same α_{l} irrespective of the pixel at which it is computed.

### 3.2 Per Pixel Chrominance Gamut Morphing

In this step we morph the two-dimensional chrominance gamut of one projector to that of another across the overlap region. However, the morph we design is constrained to retain the white point that was achieved in our earlier step. Further, this morph assumes rectangular projections and horizontal/vertical overlaps. An algorithm to assure this in any general tiled display is described in the appendix. Following is the description of our chrominance gamut morphing method.

A chrominance gamut *T* is denoted by a triangle whose vertices are given by *R* *G* and *B*. Let us now consider two chrominance gamuts *T*_{1} and *T*_{2}. We desire to define a morph that takes *R*_{1} to *R*_{2} *G*_{1} to *G*_{2} and *B*_{1} to *B*_{2}. For an intermediate chrominance gamut *t*_{k}, we would like the chrominance of the primaries *r*_{k} *g*_{k} and *b*_{k} to be a *constrained* linear combination of the chrominance of the primaries of *T*_{1} and *T*_{2}. Hence,
TeX Source
$$r_k = (1 - \tau)R_1 + \tau R_2$$
TeX Source
$$g_k = (1 - \tau)G_1 + \tau G_2$$
TeX Source
$$bk = (1 - \tau)B_1 + \tau B_2$$

Note that from Equation 5, 1 − *τ* is given by the proportions of the *B*_{l} of the channel. Hence, we need to find scale factors β_{l1} and β_{l2} between 0 and 1 such that
TeX Source
$${{\beta _l _{_1 } B_{l_1 } } \over {\beta _{l_1 } B_{l_1 } + \beta _{l_2 } B_{l_2 } }} = 1 - \tau$$
TeX Source
$${{\beta _{l_2 } B_{l_2 } } \over {\beta _{l_1 } B_{l_1 } + \beta _{l_2 } B_{l_2 } }} = \tau$$

Clearly, computing β_{l1} and β_{l2} from the above proportions is an underconstrained system. So, we first fix β_{l1} and compute the β_{l2}. If this exceeds 1, we reverse the computation by setting β_{l2} = 1. However, note that this process results in per channel scale factor at every pixel which would ruin the white point balancing achieved in the previous step. So, we make a conscious choice here to retain the white point by computing one common factor for all channels. Hence we seek *β*_{1} and *β*_{2} such that
TeX Source
$${{\beta _1 B_{W_1 } } \over {\beta _1 B_{W_1 } + \beta _2 B_{W_2 } }} = 1 - \tau$$
TeX Source
$${{\beta _2 B_{W_2 } } \over {\beta _1 B_{W_2 } 1 + \beta _2 B_{W_2 } }} = \tau$$

Note that this means the intermediate chrominance gamut *t*_{k} is still a linear combination of *T*_{1} and *T*_{2}.

The next question is how many such *t*_{k}s are required so that the transition between *T*_{1} and *T*_{2} is imperceptible. Let us assume that we make the transition from *T*_{1} to *T*_{2} through *n* steps *T*_{1} → *t*_{1}→ *t*_{2} *t*_{n} → *T*_{2} such that each of the transitions *t*_{k} → *t*_{k}_{+1} is imperceptible. To assure this, we first find the the smallest imperceptible distance *δ* in the chromaticity chart. Then, we choose *n* as
TeX Source
$$n = {{max(|R_1 R_2 |,|G_1 G_2 |,|B_1 B_2 |)} \over \delta }$$
We discuss the value of *δ* is Section 6. This chrominance gamut morphing is illustrated in Figure 2 and 3.

Thus, following the horizontal and vertical morphs as described above, a scale factor for each projector is generated for each pixel in the overlap region. These two are multiplied to create one spatially dependent map of per pixel attenuation factors per projector that defines how the inputs of the projectors at every pixel of the overlapping region should be scaled to achieve a smooth morphing of the chrominance gamut. These maps after each of the horizontal and vertical chrominance gamut morphs are shown in Figure 4.

It is important to note that *n* decides the number of pixels required to achieve an imperceptible morph. If this *n* is greater than the size of the overlap region in pixels, we may not be able to achieve an imperceptible morph. This demands a minimum size of the overlap to achieve chorminance gamut morphing and is discussed in details in Section 6. However, if *n* is less than the size of the overlap region, then we morph in the first and last pixels in the overlap. For the remaining pixels in the center of the overlap, we compute and maintain the chrominance gamut *t*_{k}_{n/2} = 0.5(*T*_{1} + *T*_{2}). This assures that the brightness of the overlap region following the chrominance gamut morph is maximized. Note that, since the scale factors *β*_{1} and *β*_{2} are chosen to be the same across all channels as opposed to different for different channels, the transition from *t*_{k} to *t*_{k}_{+1} will not assure equal changes in the chromaticity coordinates of the primaries, rather a monotonically increasing or decreasing change in the chromaticity coordinates of the primaries. The effect of this in the choice of δ is discussed in Section 6.

Thus, the chrominance gamut morphing step generates for each projector *P*_{j} two attenuation maps and following the horizontal and vertical morphs respectively. These two are multiplied to generate the final map
TeX Source
$$\beta _j (p_j,q_j) = \beta _j^H (p_j,q_j) \times \beta _j^V (p_j,q_j)$$
as shown in Figure 4. Following the application of the above attenuation maps, the new *B*_{W}, denoted by *B*_{WC} (*s* *t*), at every pixel (*s* *t*) on the display becomes
TeX Source
$$B_{W_C } (s,t) = \sum\limits_{j \in N} {\left({\sum\limits_l {\beta _j (p_j,q_j)B_{l_j },q_j)B_{l_j } V_j (p_{j,} q_j)} } \right)}$$
where *N* is the set of projectors that overlap at pixel (*s* *t*) (Figure 4).

### 3.3 Per Pixel Perceptual Brightness Constraining

Our chrominance gamut morphing step assured that the chrominance changes smoothly across the entire display. But no constraints have yet been put on the brightness variation of the display. Thus, the display still shows sharp brightness changes resulting in seams. In this step, we apply the perceptual constraining of Majumder and Stevens 2005 on *B*_{WC} (*s* *t*). By this, a perception based gradient constraint is applied to *B*_{WC}(*s* *t*) resulting in *B*_{WE} (*s* *t*) as shown in Figure 4. To achieve this *B*_{WE}(*s* *t*), the contributions from all channels at that pixel should be multiplied by
TeX Source
$$\xi (s,t) = {{B_{W_E } (s,t)} \over {B_{W_C } (s,t)}}$$
This attenuation map ζ(*s* *t*) in the display coordinates is then broken into attenuation maps for each projector ζ_{j}(*p*_{j} *q*_{j}). Note that in this case *N*_{p} projectors overlapping in a pixel (*s* *t*) all get the same ζ(*s* *t*). Hence, this step retains the chrominance achieved by the chrominance gamut morphing step and changes only the brightness. Both the attenuation map and the brightness constraining are illustrated in Figure 4.

### 3.4 Per Pixel Bezier Based Brightness Smoothing

The luminance obtained after the perceptual luminance constraining still has *C*^{1} discontinuity. This step is designed to seek a brightness *B*_{WS} which is closest to *B*_{WE} but assures *C*^{1} continuity. For this, we fit a higher order *C*^{1} continuous 2D Bezier surface to *B*_{WE}. However, to assure *C*^{1} continuity in the perceptual domain, we apply this operation in log scale. We call the function thus generated *B*_{WS}(*s*,*t*). To achieve the *B*_{WS}(*s*,*t*), the contributions from all channels at the pixel should be multiplied by
TeX Source
$$\eta (s,t) = {{B_{W_S } (s,t)} \over {B_{W_E } (s,t)}}$$
*η* (*s* *t*) is then again broken into attenuation maps for each projector *η
*_{j}(*p*_{j} *q*_{j}). Note that since this is a surface fitting step *η* (*s* *t*) can be slightly greater than 1.0 at places. So, we normalize η(*s* *t*) to assure no saturation artifacts. In this case also, the *N* projectors overlapping in a pixel (*s* *t*) all get the same η(*s* *t*). Hence, this step also retains the chrominance achieved by the chrominance gamut morphing step and changes only the brightness. Both the attenuation map and the result of the Bezier fitting are illustrated in Figure 4.

### 3.5 Image Correction

Let *I*(*p* *q*) be the image generated in the projector coordinate system after applying the geometric function *G*. First, we linearize the image *I*(*p* *q*) using a gamma function of 2. We multiply the different attenuation maps to create a final attenuation map *β*_{j}(*p* *q*) = ζ
_{j}(*p* *q*) × *η
*_{j}(*p* *q*) × *η
*_{j}(*p* *q*). To achieve the color correction, we pixel-wise multiply *I* with the attenuation map *A*_{j} to generate an appropriatedly scaled *I* *I*_{S}(*p* *q*),such that
TeX Source
$$I_S (p,q) = I(p,\dot q) \times A_j (p,q)$$
Note that *A*_{j} is channel independent and hence multiplies all the channels of *I*_{j} similarly. Following this, we apply the channel dependent white point correction to generate the white point corrected image *I*_{W} as
TeX Source
$$I_{W_l } (p,q) = \alpha _l I_{S_l } (p,q)$$
where *l* ∊ {*r* *g* *b*}. Now note, all the above corrections described till now assumes a linear projector. However, in practice the projectors have a channel input transfer function, denoted by *h*_{l}. So, to achieve the desired changes in this non-linear device, the final correction is achieved by applying the inverse of *h*_{l} to *I*_{W} . Hence, the final corrected image *I*_{C} is given by
TeX Source
$$I_{C_l } (p,q) = h_l^{ - 1} (I_{W_l } (p,q)),$$
where *l* ∊ {*r* *g* *b*}.

We have implemented our method on two displays: (a) a planar rear projected display of 3 × 3 array of nine very low-end projectors; (b) a cylindrical front projected display of 2 × 4 array of eight relatively higher-end projectors. Since (a) is a rear-projection system with a screen that deviates considerably from a Lambertian surface and uses old (about 3-4 years) low-end projectors, this shows severe color variation and is an excellent test case for our algorithm. Although (b) uses a diffuse front projection screen with relatively new (1 year) projectors, they also show considerable color variation.

To reconstruct the spatially varying color gamut for the projectors, we use an sRGB camera as a sensor. Ideally, a spectroradiometer should be used to measure the chrominance. But since most projectors have a chrominance gamut within the sRGB gamut [9], using a camera yields sufficient accuracy in measurement. To measure the chromaticity gamut of each projector, we put up for each projector the maximum intensity red, green and blue values. We used a small region near the center of each projector where sRGB measurements are averaged, converted to the XYZ values using standard sRGB to XYZ conversions and then the chromaticity coordinates are computed. The vignetting effect is measured by projecting white from each projector and then capturing it using the high resolution camera. The geometric relationship between the camera, projectors and the display is recovered using methods proposed in [2]. The camera RGB values at corresponding locations for the projectors are then converted to the XYZ values and then to the tristimulus brightness to recover the vignetting effect *V*_{j}. To recover *h*_{l} we use the technique proposed in [10] where we project a uniform sampling of inputs for each channel *l* and find their corresponding tristimulus brightness from the captured RGB value at the center of the projector.

**Real-time Image Correction on GPU:** Our method has an offline calibration step and an online correction step (Figure 4). We used MATLAB for offline computation of the attenuation maps and takes about 5 minutes; this can be easily ported to any other language such as C++ for greater efficiency. This offlice calibration generates three alpha masks (one for each channel) and three inverse transfer functions (one for each channel) for each projector, which are then used for image correction. The online image correction is achieved in real-time (30 fps) using GPUs through Chromium, an open-source distributed rendering engine for PC clusters [8]. A module for Chromium is written that first multiplies the image with the alpha masks and then uses a 1D look-up-table to apply the inverse transfer functions. These are done using a fragment shader on the GPU at interactive rates.

In this section, we show our results on both the planar and the curved screen. Note that the acid test for any color correction method is white and flat colors. Results on such images are not shown in almost any prior work Figure 5 compares our method with traditional RGB blending [14] and the method of photometric seamlessness proposed by Majumder and Stevens [10] on white on our worst display (planar one) made of old low-end projectors on a rear-projection screen. We also compare our results on the much better projectors on the curved screen in Figure 6. In both cases, our results show much superior seamlessness Figure 7 shows the results of our algorithm on many different images both on the planar and the curved displays Figure 8 shows the different steps of our process and how they progressively improve the quality of the seamlessness resulting in the final one where the projectors cannot be detected.

Our method assumes no spatial chrominance variations within a single projector. However, in practice, commodity projectors does show small smooth spatial variations in chrominance [9]. Further, our method *morphs* the 3D color gamut across the display to make local color variations imperceptible. If we attempt to *match* the 3D color gamut at every pixel the brightness of the display will be limited to the brightness of the darkest pixel and the chrominance gamut will be limited to the common chrominance gamut of all of the projectors which is usually very small. Instead, we smoothly morph the 3D color gamut in order to remove the local variations while retaining as much of the display color gamut and dynamic range as possible. Hence, global color variations still exist. Note that these global variations are not the result of applying our method, but were camouflaged in presence of more drastic local variations. Further, human perceptibility of these global variations is closely tied to the angle subtended by the display in the human eye. This angle is significantly smaller when presented as a printed picture on a paper than in reality. So, these global variations, that are not disturbing when seen in person, can become perceptible when presented on a paper.

In this section, we discuss the different system issues, accuracy and performance.

### 6.1 Minimum Size of the Overlap and δ

Consider the overlap of two projectors *P*_{1} and *P*_{2}. For projector *P*_{1}, following the chrominance gamut morph, τ defines the contribution of *P*_{1} to a pixel in the overlapping region. Hence Now, let us consider the implication of this on the change in the proportion of *B*_{l} for a single channel from *P*_{1} and *P*_{2}. Let us consider channel *l*, and let *B*_{l1} = *k*_{1} *B*_{1} and *B*_{l2} = *k*_{2} *B*_{2}. Hence, the proportion of the brightness of one channel *l* for *P*_{1} is given by
TeX Source
$${{k_1 B_{W_1 } } \over {k_1 B_{W_1 } + k_2 B_{W_2 } }} = {{k_1 \tau } \over {k_1 \tau + k_2 - k_2 \tau }}$$
after replacing Taking the derivative of this we find that the speed of this will vary between and Note that usually *k*_{1} and *k*_{2} vary from projector to projector but not much since these signify the proportion of a primary in white. Hence, both these functions are close to 1. Thus, the chrominance of each primary will not see a linear change but a monotonically increasing/decreasing change that is close to linear. This is also evident from the chromaticity plots in Figure 4.

During the chrominance gamut morph, ideally, we want *δ* to be less than 3 or 4 in the CIE LAB space in order for the color morph to be imperceptible. We ran the following analysis to find how much overlap we need to assure such a δ. Consider the channel *l* on a pair of projectors *P*_{1} and *P*_{2}. Knowing the distance between the primary *l* on these two projectors, we plotted the maximum LAB distance between adjacent pixels in the overlap of *P*_{1} and *P*_{2} for varying size of the overlap region. We found this curve for all pairs of projectors (in our pool of 17 projectors) and then found the maximum of the LAB distance for each overlap size. The resulting plot (Figure 9) provides us with a good estimate of the maximum LAB distance that would result between adjacent pixels with varying overlap size. We know that a LAB distance of 3 or less is within the just noticeable distance (JND). In the plot we see that to be within the JND, we need a minimum of 90 pixels overlap for the green primary. As expected, the overlap region required for red and blue to assure the same is much smaller. This is due to the fact that the relatively higher luminance of green makes it easier to perceive chrominance variations. An overlap region of about 90 pixels is around 10% of the projector resolution in any direction and less than 5% of the screen resolution in any direction. It is often required to overlap projectors much more to alleviate the fall-off due to the vignetting effect [9], [19]. Hence, the overlap region required by our method is reasonable in almost all tiled displays.

### 6.2 Difference from the Traditional RGB Blending

Traditional blending methods feather the RGB input of the projectors in a linear or cosine manner in the overlap region [14]. Our chrominance morphing achieves superior results than such traditional RGB blending. The primary reason for our superior results is that we can constrain the chrominance gamut morphing speed to be as close to linear as possible since we use a precise measurement of the projectors' spatial brightness variation. Hence, we constrain the change in chrominance easily to be within the human tolerance (Figure 9). Traditional blending does not consider the brightness falloff of the projectors and just uses a linear or cosine function to blend the brightness of each channel in the overlap region. This does not assure a linear change in the (*x* *y*) space and becomes especially pronounced when the projectors do not have similar brightness falloffs. An additional advantage of our chrominance gamut morphing is that, we may not use the entire overlap region for morphing and hence retain more of the brightness of the display.

### 6.3 Effect on the Display Quality

It is evident that every step of any color calibration method, including ours, imposes some constraints on the spatial color variation that would invariably lead to some reduction in the brightness of the display Table 2 includes an evaluation of this reduction in the dynamic range versus the achieved uniformity through the different stages of our method and a comparison with other existing methods. As a metric, for the dynamic range we use the mean of the brightness of the white at all pixels of the display and for the uniformity we use standard deviation from the mean brightness and standard deviation of the euclidian distance of the chrominance of each pixel from the mean chrominance. Note that our method shows a reduction in the dynamic range when compared to the photometric seamlessness method of Majumder and Stevens [10] and the edge blending method. However, we achieve greater smoothness in both the chrominance and brightness variations as indicated by the reduction in the standard deviations by an order of magnitude.

### 6.4 Is the perceptual brightness constraining required?

In our method, since the last step of the Bezier based smoothing anyway smooths the spatial brightness variation, one may question the utility of the perceptual brightness constraining. We emphasize this in Figure 10, which shows the results with and without the use of this method. Note that the results are smoother and visually more pleasing when the perceptual brightness constraining is applied. Further, the Bezier based smoothing is just a way to fit a smooth function to the spatially varying brightness. If the underlying variation is perceptible to the human eye, the Bezier based smoothing seldom makes it imperceptible. Applying the perceptual brightness constraining step already assures that the spatial variation is within the human tolerance and hence the Bezier smoothing will provide a good fit to that.

In conclusion, we have presented, for multi-projector displays, the first method that addresses the 3D color variations within and across projectors and in the overlap region simultaneously. Our method smooths both the chrominance and the brightness across the entire display resulting in a true color seamlessness where the number of projectors is absolutely invisible in the display. This is achieved by a morphing of the 2D chrominance gamut in the overlap region followed by the brightness smoothing across the entire display, both guided by perceptual parameters to assure that the variation is within the human tolerance and is hence not detectable. Unlike previous methods, our method is successful in achieving seamlessness even for the difficult case of flat colors, especially flat white. Our work demonstrates, for the first time, that inexpensive multi-projector displays can indeed provide similar seamlessness as the cost-prohibitive high-resolution single projector displays.

However, in this work, we assume that the chrominance gamut remains constant across a projector. But, in reality, the chrominance gamut does show some variation within a single projector. We would like to remove this constraining assumption in our future work to improve the results. There has been earlier work in achieving geometric registration in a distributed manner [3]. We would like to explore our method to achieve color seamlessness in a distributed manner in large multi-projector displays.

### Acknowledgments

We would like to thank Epson and Canon for their generous donations of projectors and cameras used in this project. This research is funded by NSF SGER 0743117 and NSF CAREER IIS-0846144.

## Appendix

We describe here an algorithm to achieve rectangular projections and overlaps for our chrominance gamut morphing method. For illustration of the process, please refer to Figure 2. Let us consider a single row of two *rectangular* projectors *P*_{1} and *P*_{2}. Their 2D chrominance gamut in the non-overlapping regions are denoted by the *T*_{1} and *T*_{2} respectively. Now, consider a horizontal scanline *A* through these two projectors. At any point in the overlap region in this scanline, we want to morph the chrominance gamut in the horizontal direction. This is achieved by linear morphing of *T*_{1} to *T*_{2} controlled by the parameter *a*. Details of this morphing algorithm is provided later in this section. Since the overlap between the projectors is rectangular, any vertical line *L*_{12} will have the identical morphed gamut *T*_{12}(*a*). Following the horizontal morph, we can consider *P*_{1} and *P*_{2} together to form a large projector *P*_{12} where every vertical line has identical chrominance gamut. Similarly, horizontal chrominance gamut morphing can be applied independently to another row of projectors *P*_{3} and *P*_{4} to form a large projector *P*_{34} where the same property holds. Now, in a display made of two rows of projectors *P*_{12} and *P*_{34} with a overlap between them every vertical line follows the property that it has constant chrominance gamut in the region with no overlap. Hence, they can be morphed in the vertical direction linearly exactly the same way as the horizontal morph. This method can extend to *n* projectors in each row and *m* rows in the display. Thus, by performing two passes of chrominance gamut morphs in each of the horizontal and vertical direction, we can produce a smoothly morphed chrominance gamut at any point on the display.

However, note that the key aspect of the above method is that the projectors have to be rectangular with rectangular overlap regions.

This is hardly the case, especially since projectors have off-axis projection leading to keystoning effect. Hence, we first turn off some pixel at the boundaries to make each projection rectangular Figure 11 describes this process. First, we find the largest inscribed rectangle for each projector using methods proposed in [13], as shown in Figure 11(a). Next, we consider every row of projectors independently as a single projector and repeat the process of the finding the largest inscribed rectangle in them. All pixels outside this rectangle are again turned off. Multiple such rows of projectors are arranged in a column and the above process is repeated again. The above process ensures a rectangular projection, display, and horizontal/vertical overlaps. This process is illustrated in Figure 11. Note that the keystoning showed for each projector is exaggerated for illustration purpose. In reality, the keystoning is much smaller and this does not lead to much pixel wastage. We measured these wasted pixels for both our set up. On the planar display, where it is easy to achieve a close to rectangular grid, the wastage was about 9.6%. On the curved display, where the keystoning is more severe, we see a wastage of 19.8%.