Projection Mapping with a Brightly Lit Surrounding Using a Mixed Light Field Approach

Projection mapping (PM) exhibits suboptimal performance in well-lit environments because of the interference caused by ambient light. This interference degrades the contrast of the projected images. Consequently, conventional methodologies restrict the application of PM to dimly lit settings, leading to an unnatural visual experience, as only the PM target is prominently illuminated. To overcome these limitations, we introduce an innovative approach that leverages a mixed light field, blending traditional PM with ray-controllable ambient lighting. This methodological combination, despite its simplicity, is effective because it ensures that the projector exclusively illuminates the PM target, preserving the optimal contrast. Precise control of ambient light rays is essential to prevent them from illuminating the PM target while adequately illuminating the surrounding environment. Furthermore, we propose the integration of a kaleidoscopic array with integral photography to generate dense light fields for ray-controllable ambient lighting. Additionally, we present an efficient binary-search-based calibration method tailored to this intricate optical system. Our optical simulations and the developed system collectively validate the effectiveness of our approach. Our results show that PM targets and ordinary objects coexist naturally in environments that are brightly lit as a result of our method, enhancing the overall visual experience.


INTRODUCTION
Projection mapping (PM) is a technology in which computer-generated images are projected onto physical surfaces, seamlessly merging real and virtual worlds.PM offers an immersive experience by allowing users to perceive augmented content without the need for special glasses, facilitating easy sharing of the experience with others and enabling physical interaction with augmented objects.Consequently, it represents a promising avenue in the realm of augmented reality [6,13].
However, PM has limited performance in a bright environment, as illustrated on the left side of Fig. 1.In this scenario, a PM application was employed on the object on the left, originally featuring a white surface.Its appearance was transformed into a colorful texture.As depicted in the figure, the projected image cannot be discerned clearly.The culprit here is ambient light, which bathes the entire scene in brightness.Ambient light not only illuminates non-PM ordinary objects (e.g., the bunny on the right and the board behind the bunnies in Fig. 1) but also spills onto the PM target (e.g., the left bunny in Fig. 1), significantly reducing the contrast of the PM presentation owing to elevated minimum brightness.
Consequently, conventional PM solutions have predominantly functioned in dark environments, as exemplified in the bottom center of Fig. 1 [5,18,25,29,41].However, PM fails to provide a natural visual experience in dimly lit settings for two key reasons.First, it results in a PM scene where only the PM target is well-illuminated while the surroundings remain dark, making the PM target appear overly radiant [27,36].This issue becomes particularly critical in the replication of non-luminous natural surface appearances.Second, ordinary objects not subject to PM intervention appear dark, presenting a discrepancy with the ideal augmented reality scenario where ordinary objects and PM targets coexist under identical lighting conditions.
Our objective is to surmount these challenges and realize PM with well-lit environments while preserving a high level of contrast.Furthermore, we aim to ensure that the appearance of ordinary objects aligns with the lighting conditions of the PM presentation.For instance, if the PM target is augmented to simulate diffused lighting, the ordinary objects must exhibit smooth shading.Another critical consideration is shadow consistency, in which shadows must align with lighting conditions to appear natural, such as soft-edged shadows in cases of diffused lighting.
Such PM has a high demand for practical applications in various fields.These include enhancing the experience of stage productions and attractions [1,5], facilitating the design of clothing and products [9,34], providing support for education [17], trying on makeup [7,47], and providing support in manufacturing and medical operations [30].This paper introduces a novel method that harnesses a mixed light field, combining image projection using a conventional projector with ray-controllable ambient lighting.In this approach, instead of using normal ambient light, ray-controllable ambient light is deployed to avoid illuminating the PM target while maintaining adequate illumination in other areas within the scene, as demonstrated in the central portion of Fig. 1.Concurrently, the projector operates conventionally, projecting images onto the PM target.Consequently, this configuration allows us to achieve high-contrast PM presentations with brightly lit surroundings, as depicted in the right side of Fig. 1.This result is achieved because only the projector directs light towards the target, even when the surroundings are well-illuminated.Additionally, the surface appearance of ordinary objects and their associated shadows appear natural owing to our ambient light replication of the light field within the PM-augmented scene.As described in the aforementioned technical advantages, our simple yet effective methodological combination is capable of resolving conventional, complex problems.
This paper also presents supplementary ideas for realizing these concepts.First, we propose the integration of a kaleidoscopic array with integral photography (IP) to recreate dense light fields conducive to ray-controllable ambient lighting.Second, we advocate for the calibration of complex light field configurations within the illumination units utilizing the kaleidoscopic array through an efficient binary search technique.
The specific contributions of this study are as follows.
• We present our efforts to realize PM with a brightly lit surrounding by leveraging a mixed light field, which is achieved through the combined use of a standard projector and ray-controllable ambient lighting.
• We introduce a kaleidoscopic array for ray-controllable illumination, which aims at enhancing the density of the replicated light field.Additionally, we have devised an efficient and precise calibration method for this optical system, employing binary search techniques.
• The optical simulation results offer valuable insights by allowing for a beneficial comparison of various potential approaches to the implementation of PM with well-lit environments.
• We have developed the system and showcased several augmented scenes in which PM targets and ordinary objects seamlessly coexist, creating a natural visual experience.
• Through the evaluation of the developed system, challenges for future applications have been summarized.Ray-controllable ambient lighting particularly has limitations in the fidelity of lighting reproduction due to artifacts originating from the optical system.

Projection mapping in a bright environment
Fig. 2 illustrates potential approaches to allow for PM with welllit environments.The scene comprises three boards, each initially featuring a white surface.The objective is to transform the appearance of the front-left board into a vibrant, textured display.
In the top-left quadrant of Fig. 2, we observe a configuration that combines conventional PM with standard ambient lighting.As discussed in the preceding section, this approach degrades the contrast in the appearance of the PM target.In theory, increasing the projector's maximum brightness could potentially diminish the perceptual degradation of the PM contrast.Nonetheless, such an adjustment would result in a perceptually darker surrounding environment, which continues to be a concern.Although there are projection-related studies that focused on ambient light estimation for radiometric compensation [3] and robust interaction in ambient light [46], they do not specifically concentrate on enhancing PM contrast.
A recently proposed innovative approach involves the application of PM to the entire scene [19,43].Consequently, irrespective of whether they pertain to PM targets or ordinary objects, all surface appearances are harmonized to exhibit high-contrast with well-lit conditions.However, employing a single projector for this approach is insufficient because of the appearance of unintended shadows in areas that were occluded from the projector's illumination, as illustrated in the top-center quadrant of Fig. 2.
To circumvent these blind spots and expand the coverage area, recent work has developed a system utilizing multiple projectors, as depicted in the top-right quadrant of Fig. 2 [19,43].The implementation of accelerated techniques for controlling multiple projectors can enhance this approach, even in dynamic scenarios [22,32].Nevertheless, this approach demands that all surfaces in the scene possess ideal reflective characteristics for PM, namely diffuse reflection.Achieving the requisite appearance for ordinary objects without diffuse reflections is challenging.Consequently, creating a scenario where PM targets and ordinary objects coexist consistently remains elusive.A system constrained by this limitation deviates from the core concept of augmented reality, which aims to seamlessly integrate the virtual and real worlds.

Lighting reproduction
Lighting reproduction techniques were explored to find alternative approaches for PM with well-lit environments.One such technique involves a lighting system featuring numerous light sources arranged spherically, allowing for the placement of an actor in a studio within various virtual environments [10,23].Another effective method employs large-scale monitors positioned around studios, which can be utilized in virtual video production [21].However, the inherent characteristics of these lighting systems render them incompatible with PM, as they inadvertently illuminate the PM target, leading to issues of low contrast.Furthermore, while they excel in reproducing realistic reflected colors on surfaces, they encounter challenges in faithfully reproducing shadows originating from the light field with high quality.
Still-life photography presents another application that demands intricate lighting setups.While it is conceivable that the professional lighting technique could replicate an ideal ambient light closely resembling the illumination within the augmented scene while avoiding the PM target, this necessitates substantial manual effort each time the objects are rearranged.Various automatic steerable illumination units have been introduced to alleviate the complexities associated with constructing lighting configurations, employing pan-tilt platforms [26,28], robot arms [51], and aerial robots [42].However, these approaches have limitations in terms of lighting variety since they primarily rely on a limited number of spotlights.
The next method leverages IP.IP is used for 3D imaging and displays to capture or present light fields.This capability can be adapted to reproduce lighting environments [24,33].In comparison to the aforementioned methods, the key advantage lies in its precise control of rays within the light field.The system can be configured as a combination of a lens array and backlight.Various implementations have employed LED arrays [44,45] and projectors [55].Moreover, we found a system consisting of mirrors and projectors [4].Among these, our focus centers on the configuration involving a lens array and projectors since it can generate fine rays and increase the number of rays emanating from each lens.Consequently, we gain intricate control over the rays within the light field [38,52,54].
While this particular type of lighting reproduction has not yet been introduced in the context of PM, it holds promising potential.Theoretically, this system has the capacity to reproduce ambient light and can serve as an image projection unit for PM.As illustrated in the bottom-left quadrant of Fig. 2, an example configuration showcases the realization of PM with a brightly lit environment using solely this technique.However, the light rays passing through the lens array exhibit slight diffusion due to lens aberration, resulting in a bokeh effect on the surface of the PM target.Consequently, the resolution of the PM appearance achieved through this method may not be as high as that of conventional direct image projection.However, the ability to manipulate the light field with precision is sufficiently robust to mask areas that remain unilluminated.Our methodology draws inspiration from these insights to selectively leverage the advantages of this lighting reproduction technique.

MIXED LIGHT FIELD 3.1 Overview
We introduce a novel concept: a mixed light field comprising a conventional projector and a ray-controllable lighting unit tailored for PM applications with well-lit surroundings.This lighting unit faithfully reproduces ambient light across the entire scene while skillfully avoiding illumination of the PM target.As a result of this adaptive and spatially varying ambient lighting, the surface of the PM target receives exclusive illumination from the PM projector, enabling a high-contrast PM presentation even with brightly illuminated settings.Furthermore, the ray-controllable lighting unit replicates a light field reminiscent of the one found in scenes augmented by PM technology.Consequently, surrounding objects, apart from the PM targets themselves, assume shading appearances and cast shadows consistent with those of the PM targets.Importantly, this method retains the conventional PM performance by enabling precise manipulation of appearance at high spatial resolutions.Moreover, while it necessitates the PM target to exhibit diffuse characteristics, the surrounding objects can possess arbitrary reflectance and transmittance properties.
The linchpin of this concept lies in the ray-controllable lighting unit.This unit must adeptly govern the rays within the light field to reproduce a range of ambient lighting scenarios while meticulously avoiding illumination of the PM target.The configuration illustrating this requirement is showcased in the bottom-center portion of Fig. 2. To fulfill this, we propose the use of IP-based illumination in conjunction with projectors, as elaborated upon in Section 2.2.
We introduce a novel method centered around kaleidoscopic arrays to enhance the density of the light field.Additionally, we employ an efficient calibration technique, which is essential for reproducing highquality PM with brightly lit environments.Detailed explanations of these two methods are provided in the ensuing sections.

Kaleidoscopic array
In this section, we first provide a concise overview of a standard IPbased lighting unit that does not incorporate a kaleidoscopic array.In this unit, the focal plane of the projector is meticulously aligned with the anterior focal plane of the lenses within the lens array.This precise arrangement ensures that a ray emanating from a pixel projected by the projector emerges from the lens as a parallel beam of light.As shown in Fig. 3(a), the rays exiting each lens converge to a single focal point.Consequently, rays radially emanate from these convergence points, shaping the light field.
However, as shown in Fig. 3(a), the ray density between the central and peripheral regions of the light field varies.The peripheral regions do not possess a sufficiently high ray density for effective ambient light reproduction.If we set the target space (the possible area for PM with a bright environment) only as the central area with higher ray density, there will be a lot of wasted light rays.To mitigate this issue, the ray density can be increased by reducing the pitch of the lens array.This approach also requires a corresponding increase in projector resolution to match the finer pitch.However, manufacturing large lens arrays with fine pitches poses significant challenges.
As an alternative solution, we focus on mirrors as optical components.Historically, mirrors have proven effective in expanding the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.number of viewpoints in bidirectional texture reflectance measurements and enhancing the resolution of 3D displays [14,15,54].In our study, we introduce a kaleidoscopic array to ensure a uniform high-density light field within the target space, crucial for ambient light reproduction.This method is illustrated in Fig. 3(b) and involves placing an array of mirrors behind the lens array.This configuration reflects rays emerging from the lens array, leading to an increased ray density.An example of this configuration is illustrated in Figs.4(a) and (b).
We present the design methodology for the mirror arrays.The mirrors can be positioned on the four sides surrounding each lens, as shown in Fig. 4. Therefore, the number of mirrors is 4N when the number of lenses is N.These mirrors are flat with reflective surfaces only on the side facing the corresponding lens, with the opposite side being non-reflective.Each mirror can adjust its length in a direction perpendicular to the lens surface and its distance from the center of the convergence points.In light of these factors, the design of a mirror array necessitates maximizing ray density within the target space while minimizing mirror length to facilitate ease of fabrication and installation to lens array.Fig. 5 presents an example showcasing the impact of mirror length on performance.
We propose the determination of the configuration of the mirror array corresponding to the given projector, lens array, and target space by solving the following optimization problem: The final output image is overlayed on each step to clarify the explanation.We identify which pixels create light rays that enter the PM target.In each searching step, we turn on each region half by half.If light rays enter the PM target, then the pixels we are looking for exist.On the other hand, if light rays do not enter the PM target, then the pixels we are looking for do not exist.In this case, there is no need to continue this search on this branch. minimize L L L = [l 1 , l 2 ,...,l N m ] represents the lengths of mirrors.In the implementation, we heuristically placed N m (N m < 4N) mirrors only at the red locations shown in Fig. 4(c).In addition, we solve this equation by projecting onto two-dimensional, horizontal and vertical planes.Consequently, we achieve the same length and positions for vertical mirrors in the same column and horizontal mirrors in the same row.This simplification effectively avoids complicated mirror reflections and is helpful in obtaining efficient solutions.|L L L| is the total mirror length.
represents the center positions of the convergence points.f (p p p i ; L L L,U U U) is the number of convergence points behind the lenses that can produce a light ray at point p p p i in the target scene.This number includes the mirror images of the convergence points, which increase with mirror reflections.M is the number of representative points in the target scene for evaluating the reproduced light field.ε l , ε t , and ε u are the maximum mirror length and the minimum and maximum distance from the convergence point, respectively.N k is the set of mirror indices corresponding to k-th lens.

Projection image generation
This section details the procedure for producing ambient lighting.We calculate the light field corresponding to the ambient light in the PM and employ ray tracing to obtain the output of the projector.Subsequently, pixels that directly illuminate the PM targets are set to zero.Although the two aforementioned initial procedures align with conventional methods [44,45,55], we focus on the third procedure.
In standard calibration, rays are treated as linear entities.However, real-world systems introduce various complications, such as ray dispersion, stray light from lenses or mirror arrays, and interreflections  influenced by object configurations and their reflective properties.Consequently, traditional optical calibration and ray-tracing simulations may inadvertently illuminate the PM targets, with even minimal light exposure leading to noticeable contrast degradation.
Therefore, we adopt an image-based approach, directly capturing the actual scene with a camera to identify the pixels from the projector that illuminated the PM targets.We assume knowledge of the PM target's area within the camera image, placed at an ideal viewpoint.One rudimentary approach involves sequentially projecting individual pixels to determine if they illuminate the PM target.However, this brute-force method proves inefficient because of the sheer number of measurements and extended camera exposure time required to detect low-intensity light.An alternative method utilizes the light transport matrix (LTM) between the camera and projector to identify which pixels from the projector reach the PM target area within the camera's view [50].Although obtaining an LTM is time-consuming, efforts have been made to enhance its efficiency using adaptive sampling, sparse modeling, and machine learning [35,37,40,49].However, for these optimized results, obtaining results that match those of the brute-force method in terms of precision remains challenging.
For our specific needs, binary information suffices to determine whether a projector's pixels have reached the PM target.Consequently, we propose a straightforward and accurate approach based on binary search within a projector image.Fig. 6 shows the pipeline of this method where the PM target is a rabbit-shaped one.Our approach involves projecting an image and monitoring it with a camera.If the light is detected within the PM target's range, we proceed to split and continue the search; otherwise, we terminate the search.This significantly reduces the total number of required measurements.Reducing the camera exposure time can further expedite individual measurements when the range of the projector image is substantial.The accuracy of our method matches that of the brute-force approach.

IMPLEMENTATION DETAILS
Fig. 7 shows the developed system.A projector with a resolution of 1024 × 768 (Tokyo Electron Devices, Limited, TB-UK-DYNAFLASH, throw ratio: 1.81, offset : 112.5%) was used for the ray-controllable lighting unit.This synchronizes the projection with an external camera (Basler, ace acA720-520um, resolution of 720 × 540), which is useful for image generation, as described in Section 3.3.Moreover, we used a projector with a resolution of 1920 × 1080 (Vivitek, QUMI Q38, throw ratio: 1.2, offset: 100%) for the PM.
The lens array for the ray-controllable lighting consisted of 108 lenses, arranged in both horizontal (12 lenses) and vertical (9 lenses) configurations.Such lenses, capable of accommodating a wide maximum cone angle for emitted light, effectively broaden the range of ray directions within the light field.Consequently, in this study, we opted for lenses with a focal length of 50 mm, an aperture number of 0.5, and a diameter of 50 mm.These lenses were arranged with a 53 mm pitch.Consequently, the size of this lens array was 636 × 477 mm.While it is feasible to fabricate a lens array with enhanced ambient-light reproduction capabilities at a higher cost, this study omits that consideration, with the primary focus on demonstrating the potential for improving lens array performance using a kaleidoscopic array.
The kaleidoscopic array configuration was determined using the method described in Section 3.2.The constraints in Equation (1) were set as follows: ε l = 70 mm, ε t = 15 mm, ε u = 50 mm.ε l was heuristically determined based on the condition of being the maximum permissible length that is feasible for fabrication and can be accommodated within the system.ε t and ε u were determined based on the specifications of the previously mentioned lens array.The number of evaluated positions M was five.The arrangement was positioned on the two-dimensional plane, as depicted in Fig. 4(d), to simplify the process as described in Section 3.2.Consequently, the optimized mirror lengths L L L were

EVALUATIONS BASED ON SIMULATION 5.1 Comparison with other methods
We compared six methods: PM with normal ambient light (method #0), PM onto the entire scene with a single projector (method #1), PM onto the entire scene with multiple projectors (method #2), PM with a light field (method #3), mixed light field (method #4), and mixed light field with a kaleidoscopic array (method #5).The definitions of these methods are described in Sections 2 and 3.The experiment aims to assess whether each method can achieve a high-contrast PM while maintaining an appearance similar to the surrounding environment using method #0.
The simulated scene is illustrated in Fig. 8.The front surface of the center box target was augmented to have a chessboard texture through PM.Moreover, two ordinary objects were placed to assess the natural coexistence with the PM-enhanced object.The flat plate on the left was Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.Fig. 9: The simulated illuminance of both the PM target and background is presented.In the 3D configuration images, only the surfaces of the box and background board were simulated, excluding other objects.The bottom table illustrates the average illuminances of the white and black areas within the chessboard-textured mapping and the background areas.These values were derived from the red, green, and blue regions in the bottom-right illustration.Additionally, the table includes the PM contrast, which was calculated based on the ratio of the white to black values.
a diffuse object.The sphere was transparent glass, and its refractive index was 1.6.It was assumed that the system possessed the capability to measure the shape and reflectance properties of the flat plate; however, it was not equipped to assess analogous attributes of the glass sphere.Consequently, although methods #1 and #2 might successfully reproduce consistent shadows from the flat plate, they could face challenges in maintaining high-quality reproduction in proximity to the glass sphere.We utilized two 250 × 250 mm LED panel lights for method #0, with the arrangement shown in Fig. 12(a).For PM in methods #0, #1, #4, and #5, we employed a projector with a resolution of 912 × 513, a throw ratio of 0.59, and an offset of 70 %.This projector was also used in method #2 to project an image onto the center box in the scene, while additional four projectors (resolution: 816 × 459, throw ratio: 0.73, offset: 62.5 %) were utilized to apply PM to the surrounding areas.The specifications of the ray-controllable lighting unit for methods #3, #4, and #5 are described in Section 4, except for the offset of the projector, which was modified to 100 % 1 .
The conditions of the light intensity were determined as follows.
First, the target average illuminance on the surface of the center PM box was set to 1400 rays/mm 2 .In this simulation, illuminance was defined as the number of rays.Second, the target average illuminance for the background board was set to 1400 rays/mm 2 in method #0 and 5600 rays/mm 2 in other methods, respectively, when the scene consisted only of the background board and the maximum number of rays was cast.
To achieve the aforementioned target illuminances, the PM projector cast up to 1200 rays/pixel with a resolution of 912 × 513 for all methods except for method #3.However, the projector in method #1 adjusted to cast up to 6000 rays/pixel if the corresponding pixel illuminated the surrounding area.In contrast, the additional projectors in method #2, with a resolution of 816 × 459, cast up to 1600 rays/pixel.For method #3, the projector for the ray-controllable lighting unit cast up to 1840 rays/pixel and 8400 rays/pixel in the pixels for the center PM box and the surrounding area, respectively.The projectors for the lighting units in methods #4 and #5 cast up to 8400 and 3600 rays/pixel, respectively.
Based on these conditions, we conducted a Monte Carlo ray-tracing optical simulation using our own originally implemented codes.In this simulation, following the characteristics of a projection lens, rays were cast in random directions from each pixel and traced until they hit a surface.The illuminance at each surface point was calculated by  tallying the number of rays that reached it.
In methods #3, #4, and #5, to reproduce the same light field in the ambient light as in method #0, we searched for similar rays using the following conditions.The first condition checked whether the distance between the points passed by the true ray and the candidate ray at the plane, which was positioned in front of the background board, was less than 2.5 mm.The second condition evaluated whether the angle between those rays was less than 0.1 rad.If both conditions were satisfied, the candidate ray with the narrowest angle was illuminated from the system.
The simulation results are presented in Fig. 9.These results depict the PM appearance on the box surface and the illuminance distribution on the background board.In this study, we omitted the evaluations of the appearance of the left flat plate and right sphere.
Although the contrast for PM was low, method #0 reproduced the ground truth of illuminance distribution for the background board.Methods #1 and #2 achieved high contrast for the chessboard-texture appearance, and bright illuminance in the surrounding area was closer to method #0.However, the shadow and caustics in the background board largely differed from the ground truth.Method #3 had the advantage that the shadow and caustics in the background were similar to the ground truth.However, the surrounding area was darker than method #0.As shown in the upper area of the background, the illuminance was only around half of the ground truth.In addition, method #3 could not achieve high contrast for PM because this method made the projected image blurred.
Methods #4 and #5 achieved high contrast for PM appearance.However, similar to method #3, the illuminance in method #4 was low.On the other hand, we could confirm that the illuminance distribution in the background board was closest to the ground truth due to higher ray density in method #5 than in method #4, as shown in the graph of the illuminance profile.

Efficiency of image generation
We evaluated the efficiency of the method described in Section 3.3.The distance between the PM target and lens array was 500 mm.The targets evaluated were a flat square and a Stanford bunny.Each object was evaluated by using three objects of different sizes.We used 50, 100, and 200 mm squares.The bunny sizes were approximately 50 × 50 × 39, 100 × 100 × 78, and 200 × 200 × 156 mm.Fig. 10 shows the generated mask images where the pixels illuminating the PM target are set to one.Fig. 10 presents the number of measurements to get each mask image.Compared with the brute-force method, whose number of measurements was 786,432 (1024 × 768), all the images could be obtained with fewer iterations.This is because, as shown in Fig. 10, the found pixels were sparsely distributed.

EXPERIMENTS USING THE DEVELOPED SYSTEM
Results in this section are based on the system shown in Fig. 7.

Evaluations
Fig. 11 shows the results of four methods.The PM target is the front face of the white box.The box geometry was 110 × 80 ×75 mm.We placed an ND filter (OD = 0.9) in front of the PM projector.An ordinary object was placed next to the box.The object was a 3D-printed Stanford bunny [48].Behind these two objects, a large flat-textured board was placed.The goal was to change the appearance of the front box surface using a PM and illuminate the surrounding area with diffuse lighting.Two images, a landscape photograph and a chessboard pattern, were projected onto the PM target.We used two 250 × 250 mm LED panel lights for diffuse lighting in method #0.The arrangement is shown in Fig. 12(a)2 .The brightness of the panel lights was adjusted to 215 lux at the center of the front surface of the box.
The irradiance values were estimated at the three points shown in Fig. 12(b) using the chessboard-pattern picture based on the recovered camera response function [11].Moreover, we evaluated two types of contrast: PM contrast C p (the ratio of the irradiance values between white and black) and scene contrast C s (the ratio of the irradiance values between background and black).The comparison of those values using four methods is shown in Table 1.
The first result was the PM under normal ambient light (method #0), as shown at the top of Fig. 11.The results show that both contrasts were low, whereas the shading and shadow were smooth in the surrounding ordinary objects.The second result involved PM onto the entire scene using a single projector (method #1), as shown in the second row of Fig. 11.The results present that the scene contrast C s was low; however, the PM contrast C p was high.The appearance of the surrounding area differed significantly from the intended diffuse lighting.
The third and fourth rows present the results for the mixed light field, with configurations both without and with a kaleidoscopic array (methods #4 and #5).For the ray-controllable illumination unit, the projector pixel values were set to their maximum values except for the pixels projected onto the PM target.Both results were markedly more effective in achieving PM with a bright environment than the two aforementioned results.
In this evaluation, the methods #4 and #5 did not perfectly replicate the light field of the aforementioned panel lights.The ray-controllable illumination unit was larger than that of the panel light.We discussed the results based on these assumptions.First, when comparing with and without the kaleidoscopic array, it was evident from the luminance of the background that the surrounding environment became brighter when the kaleidoscopic array was utilized.Moreover, when the kaleidoscopic array was used, the shadows of ordinary objects were no darker than when they were not used.This result indicates an increase in the light-ray density when using the kaleidoscopic array, leading to more rays that were not obstructed by objects.In addition, the shadow obtained using the kaleidoscopic array is smoother.However, the PM contrast C p was higher when a kaleidoscopic array was not used because of the stray light introduced by the mirror array, which was not effectively eliminated from the PM projection range.Despite this issue with stray light, the scene contrast C s was enhanced in configurations incorporating the kaleidoscopic array.
In addition, we measured the illuminance to compare the performances with and without the kaleidoscopic array.The center of the front surface of the PM target and the point 30 mm away from the PM target were measured.The projector for the PM was turned off during this measurement.We used Konica Minolta T-10A for illuminance measurements.Table 2 lists the results.We confirmed that the kaleidoscopic array increased the intensity of the ambient light.Although the stray light due to the kaleidoscopic array increased the illuminance at the PM target center, the illuminance ratio was still higher with the kaleidoscopic array.We also compared the execution times required to obtain pixels illuminating the PM target in the projector for the ray-controllable illumination unit.Approximately 5.5 hours were required when we captured the projection for every pixel at 40 fps with an exposure time of 25 ms in a brute-force approach.However, with the same capture setting, our binary search method only took 11 min and 40 min for configurations with and without the kaleidoscopic array, respectively.We expected the time to be shorter if we adaptively change the exposure time, as described in Section 3.3.

Demonstrations
This section presents a demonstration of leveraging the system with a kaleidoscopic array.The first example is shown in Fig. 1.Two 3D-printed Stanford bunnies of the same size and shape were arranged Fig. 13: Polka-dot PM using the proposed method was applied to the left bunny.The right picture shows the scene when the projector for PM was turned off.
in the scene.The left bunny was used as the PM target, which was originally white, and the right bunny was used as an ordinary object with a colorful texture.As shown in this result, the PM with a bright environment contributes to the realization of a scene in which the PM and ordinary objects coexist naturally.
The second demonstration results are presented in Fig. 13.The PM was selectively applied to certain areas in a polka-dot pattern on the left bunny.The ray-controllable illumination unit cast ambient light on the scene while avoiding the PM areas.Mixing the ambient light and PM flexibly, even on the surface of a single object, is possible due to the high ray density of the light field.We confirmed that the dot diameter could be reduced to 14 mm, which was the minimum diameter limited by the ray-controllable illumination unit.
The third demonstration results are shown in Fig. 14.The scene consisted of a box (110 × 80 × 75 mm), a mirror, and a board as shown in the bottom right of the figure.PM was applied to the front surface of the box.The mirror and board were used as ordinary objects.In this demonstration, our mixed light field system reproduced the spotlight as ambient light.Similar to the previous demonstrations, rays from the ray-controllable lighting unit illuminating the PM target were removed, as shown in the bottom left of the figure.The image on the top left of Fig. 14 shows PM with the light field of spotlight reproduced by the proposed method.For reference, the PM target was illuminated using an actual spotlight (Hozan L-701) shown in the top right of Fig. 14.We maintained a high contrast in the PM content, even under lighting conditions different from the aforementioned diffuse lighting.Our method successfully reproduced close shading and shadows on the board relative to the reference.In addition, we reproduced the reflections of the spotlight from the mirror 3 .

Limitations in illumination unit
The surface appearances and shadows of ordinary objects were physically reproduced using light from the ray-controllable illumination unit.A significant advantage is the elimination of the need to readjust the output of the illumination unit every time ordinary objects change the reflectance property and arrangement.
However, as shown in the results, block-shaped artifacts were observed in the shaded area because the ray density needed to be increased one step further.For example, arranging smaller lenses with shorter pitches to increase the ray density remains a challenge.Moreover, it is believed that increasing the number of projectors or their resolution can enhance the ray density.On the other hand, these challenges are largely mitigated because of the proposed kaleidoscopic array, which has contributed to the enhanced ray-density performance.
Subsequently, as shown in the spotlighting results in Section 6.2, artifacts occurred owing to the chromatic aberration of the lens.To reduce these artifacts, it is necessary to construct an array of lenses that suppress such aberrations.Recent examples, such as the fabrication of lenses equipped with achromats and apochromats to address chromatic aberrations using 3D printing [39], should be considered cost-effective solutions.
However, because chromatic aberration is an issue in refractive optical systems, the construction of an illumination unit using solely reflective optics can also be considered.Relevant examples include the design of light field displays using reflective optics [8,12,53] and manipulating light in the art field with concave mirror arrays and projectors [2].However, this remains an unexplored area, and there are concerns regarding whether an illumination system that controls high-density rays using only reflective optics can be realized.
As shown in Section 6.1, the kaleidoscopic array succeeded in increasing ray density.However, the luminance of the dark parts of the PM area increased, suggesting a potential increase in stray light.The increase is attributable to the slight diffuse reflection because the mirrors used in this study were manufactured using aluminum vapor deposition.Careful selection of mirror materials is essential. 3Additional demonstrations are shown in the supplementary material.

Limitations in light-ray control
The binary-search-based ray calibration method, as detailed in Section 3.3, effectively obtained a ray distribution that avoids casting illumination onto the PM with high accuracy and in a short timeframe, accurately accounting for these situations.However, this method becomes inefficient when dealing with large PM targets that encompass extensive scene areas.Depending on the size of the target, it may require switching search strategies, indicating room for improvement.
Furthermore, since only one camera and one ray-controllable illumination unit were utilized, the control of the illumination rays projected onto blind spots was limited.In the future, expanding this methodology to incorporate multiple cameras and illumination units will be necessary.
Nonetheless, further acceleration is essential to deploy this method effectively in PM for dynamic scenes.The binary search approach relies solely on observed projection outcomes without prior knowledge of the optical system.It is imperative to integrate prior knowledge and leverage concepts from techniques like compressive sensing and machine learning [35,37,40,49] without sacrificing accuracy.

Consistency between PM and ordinary objects
The surface representation of the PM target is influenced not only by stray light from the ray-controllable illumination unit, as previously mentioned, but also by projection artifacts, such as defocus, color fidelity, resolution, and dynamic range, along with the quality of computer graphics integrated into the projected image.Additionally, any occluding objects positioned between the projector and the PM target cast shadows on the surface, resulting in disparities with the reproduced ambient lighting conditions.We plan to leverage the latest advancements in PM technology specifically designed to address these challenges to achieve a more seamless surface appearance alignment between ordinary objects and the PM target [13,16,20,31,32].

CONCLUSION
This study introduced an innovative method for PM with bright surroundings.Our approach leverages a mixed light field that combines a traditional projector with a ray-controlled illumination unit capable of replicating ambient light while strategically avoiding illumination of the PM target.Consequently, PM carried out exclusively by the projector maintains high contrast with well-lit conditions.Additionally, ordinary objects maintain their natural appearance and shadows when illuminated by ray-controlled ambient light, even with non-diffuse reflectance properties, addressing a challenge that previous studies have not fully solved.
Moreover, we implemented a kaleidoscopic array to enhance the light field density within our ray-controllable illumination system.Moreover, we described a ray calibration methodology that employs binary search, ensuring precise avoidance of illumination on the PM target area within a condensed timeframe.Optical simulations comparing our technique with previous approaches have demonstrated its effectiveness.Our engineered systems provide compelling demonstrations in which both PM targets and ordinary objects coexist seamlessly with well-illuminated environments, creating a significantly enriched visual ambiance.
Nonetheless, PM for brightly lit settings is still in the early stages of development.As highlighted in Section 7, there is a continuing need for persistent efforts to overcome the identified challenges.It is crucial to usher in the use of PM with bright environments in a more natural and seamless manner for practical, everyday applications, such as stage productions, attractions, aiding in product design, makeup try-ons, and support in manufacturing and medical operations.
Fig.2: A set of strategies is explored to realize PM with brightly lit surroundings.The test setup involves the placement of two small boards within the scene: one serves as a PM target, and the other represents an ordinary object.A larger board is positioned behind them.The proposed concept, known as the mixed light field, aims to achieve a high level of contrast in PM presentations while simultaneously reproducing the natural appearance of ordinary objects.

Fig. 3 :
Fig. 3: (a) Light rays from the normal IP-based illumination unit are shown.The ray density in the peripheral regions is not sufficiently high.(b) Light rays from the unit with a kaleidoscopic array are shown.Using the reflection off mirrors, we can make the ray density higher.

Fig. 4 :Fig. 5 :
Fig. 4: Proposed system with kaleidoscopic array.(a,b) The kaleidoscopic array is integrated into the mixed light field.The blue panels are mirrors that consist of each kaleidoscope.(c) Mirror locations, heuristically determined, are indicated in red.(d) The arrangement of representative points (orange) for evaluating Equation (1) against the lens array when viewed from horizontal or vertical directions of the system.

Fig. 6 :
Fig.6: Binary search on the projector image.The final output image is overlayed on each step to clarify the explanation.We identify which pixels create light rays that enter the PM target.In each searching step, we turn on each region half by half.If light rays enter the PM target, then the pixels we are looking for exist.On the other hand, if light rays do not enter the PM target, then the pixels we are looking for do not exist.In this case, there is no need to continue this search on this branch.

Fig. 7 :
Fig. 7: The left picture shows the entire configuration of the developed system.The right shows close-up pictures of the kaleidoscopic array.The light-absorbing sheet was attached to the back side of each mirror against stray light.
target illuminance distribution in a cross-section #1 PM onto entire scene, single projector #0 PM with normal ambient light #2 PM onto entire scene, multi-projector #3 PM with light field #4 Mixed light field #5 Mixed light field with kaleidoscopic array #1 PM onto entire scene, single projector #0 PM with normal ambient light #2 PM onto entire scene, multi-projector #3 PM with light field #4 Mixed light field #5 Mixed light field with kaleidoscopic array white area black area background

Fig. 10 :
Fig. 10: Simulation of image generation for two targets of varying sizes.The white pixels indicate those illuminating the PM target.The PM targets included a flat square and a Stanford bunny.The PM target was positioned at the center of the target space.Numerical values at the bottom of each image represent the number of measurements.

Fig. 11 :
Fig. 11: The actual PM results for four different methods.The PM target was a white box.The left and middle columns show the PM results of a landscape image and a chessboard pattern, respectively.The right two columns show the close-up of the shadow of the Stanford bunny.

Fig. 12 :
Fig. 12: (a) Arrangement of panel lights in the scene.(b) Measured point for the irradiance and contrast evaluations.W hite, black, and background values were measured at the red, green, and blue points, respectively.

Fig. 14 :
Fig. 14: (Top left) PM in the light field of a spotlight using the proposed method.(Top right) PM with an actual spotlight.(Bottom left) PM projector was turned off in the proposed method.(Bottom right) Experimental setup.
watanabe et aL.: PROJeCtIOn MaPPInG wItH a bRIGHtLY LIt SURROUnDInG USInG a MIXeD LIGHt FIeLD aPPROaCH

Table 1 :
Comparison of camera irradiance and contrasts.

Table 2 :
Comparison of illuminance and their ratios.