A New Algorithm for Displaying Images with High Resolution Using a Directional Volumetric Display with Threads and a Projector

Using threads and a projector, a directional volumetric display capable of moving images and full-color representation was developed in our previous work. However, the horizontal resolution of the directional volumetric display could only achieve 20 pixels with 400 threads. Further, the conventional algorithm requires P squared threads per P horizontal pixels of the input image. Because it is difficult to place a large number of threads, thus, a new algorithm for developing projected images to improve the directional volumetric display’s resolution was proposed. It is feasible to display images of P pixels with at least P threads using this technique. However, the higher the resolution, the lower the image quality in the proposed algorithm. Thus, it was verified how many threads can be used to display high-resolution images without degrading the image quality. Further, by representing the horizontal resolution of the input image with 5-6 threads per pixel, it is possible to display high-resolution images while maintaining the image quality. The proposed technique can display 64 pixels per 384 threads, whereas the conventional method can only display 20 pixels input images per 400 threads.


I. INTRODUCTION
S EVERAL three-dimensional (3D) display technologies such as holography [1], [2] have been developed recently. Holography is an ideal 3D display technology that can record and reproduce 3D information in its original form. However, the computational complexity of holograms for capturing three-dimensional information has been a major concern. Custom hardware accelerators such as fieldprogrammable gate arrays (FPGAs) and ASICs (applicationspecific integrated circuits) for high-speed computing [3], [4] have recently been shown to offer a feasible solution to this problem. Another type of 3D display technology is a volumetric display [5]- [7]; it is a technology that uses water drops, bubbles, plasma, and rotating objects as a medium [8]- [11] to show images on a volumetric display. Volumetric displays using photophoretic trapping, which can display full-color images in the air [12], [13], and a volumetric display using a femtosecond laser and two parabolic mirrors facing each other [14], and multimodal acoustic trapping displays, capable of trapping particles acoustically and delivering visual, auditory, and tactile contents simultaneously [15], have recently been developed.
Nakayama et al. [16] proposed simultaneously displaying different two-dimensional information for multiple directions using a volumetric display. A display produced through this technique is called a directional volumetric display because it conveys information in a specific viewpoint direction (Figure 1). Directional volumetric displays are currently being developed using 3D crystals, light-emitting diodes (LEDs), quantum dots (QDs), inkjet printing, and threads [16]- [20]. Figure 1(a) shows a directional volumetric display using 3D crystals, and Figure 1(b) shows a directional volumetric display using threads and a projector. It is now possible to display higher resolution images using threads instead of LEDs.
Further, the directional volumetric display has been researched to develop interactive displays for people and systems. These include systems that transmit images while following the observer and systems that display directional images based on the language and position of the observer [21], [22]. Thus, the directional volumetric display is expected to be used for both practical and entertaining purposes. However, there are limitations, such as the display of low horizontal resolution. Threads must be added to increase the resolution, but the cost rises exponentially as the resolution increases. When the horizontal resolution of the displayed image is P, the thread to the square of P is required. Thus, a new design algorithm to achieve high resolution is proposed in this study.
The main contributions of this study are as follows.
• P threads are required per pixel of the horizontal resolution of the displayed image in the conventional method, but it is confirmed that the number of threads can be reduced to 1 per pixel of the horizontal resolution of the displayed image in this method. • It was shown that the image quality was almost equal to that of the conventional method with 5-6 threads per pixel in the horizontal direction of the displayed image. • Contrary to the conventional method, which displays an image of 20 × 20 pixels with 400 threads, this method displays an image of 64 × 64 pixels with almost the same number of threads.
The technologies for displaying different images depending on the viewing direction are introduced in Section II, the method achieving high resolution of the directional volumet-ric display is described in Section III, and the results and discussions are described in Sections IV and V, respectively. Finally, the conclusion and future work are described.

II. RELATED WORK
Viewpoint-dependent viewing can be accomplished by several techniques. Pjanic et al. [23] and Sakurai et al. [24] developed a technique for creating color patterns with varying hues when viewed from a different perspective. Ikeda et al. also developed a system that uses a high-speed projector at 2,400 frames per second to project a specific image on a flat surface, which displays different images based on movement at a specific speed and direction [25]. However, these techniques have limitations as multiple images can only be displayed on flat surfaces.
The objects on the cover of "Gödel, Escher, Bach: An Eternal Golden Braid" and "The SQRIANCLE" (coined from Square, Triangle, Circle) are 3D objects that can be used to display multiple images in different directions [26], [27]. An example is "Shadow Art" developed by Mitra et al. [28]. Shadow Art creates a meaningful shape by shining light on the object from a specific angle and casting a shadow on the background. However, it is impossible to identify the object by looking at it directly. He developed a system model by inputting multiple target shadow images and projecting them as shadows to create a 3D model. To achieve this process, the user sets the light source's position, size, and orientation, and then uses the input images to hollow out the 3D model in its initial state. This allows multiple shadows to be projected from a single model formed. However, creating this Shadow Art is challenging because it cannot be expressed without a light source to cast the shadows. Further, Suzuki et al. proposed a method that uses wires to form user-specified line drawings from different viewpoints using two-line drawing images and a user-provided viewpoint as input [29]. Hsiao et al. recently developed a computational framework to automatically create multi-view 3D wire sculptures using two or three user-specified line drawings and their related viewpoints as input [30]. These methods do not require the creation of shadows because the shape can be determined by the wire itself; they do not illuminate the light source and it is easier to modify the sculpture than Shadow Art. The twodimensional images projected from these three-dimensional objects are restricted to binary images, and the number of images displayed is severely restricted. It is also challenging to represent moving images. Thus, only simple objects such as characters and character silhouettes can be displayed.
Sugihara [31] realized topology-disturbing objects using optical illusion. Topology-disturbing objects are objects that appear to disturb topology when viewed from two specific viewpoints, such as two objects that appear to be separated from each other when viewed from one viewpoint but appear to cross each other when viewed from another viewpoint. In addition, Sugihara et al. [32] realized a 3D object that was initially incomplete but became complete when it was reinforced with specular reflection. The two-dimensional images projected from these three-dimensional objects are restricted to binary images, and the number of images displayed is severely restricted. It is also challenging to represent moving images. Thus, only simple objects such as characters and character silhouettes can be displayed.
In our directional volumetric display, each voxel, which is a volume element consisting of a three-dimensional structure, has color information and gradation that can be displayed in full color. Multiple full-color movies can currently be displayed in different directions using threads and a projector. Further, a system that employs a directional volumetric display using threads and a projector to identify spoken languages and transmit the corresponding language has been developed. Thus, this can be a new information transmission system not only in the fields of art and entertainment but also in various scenarios because it can alleviate conventional limitations.

A. CONVENTIONAL METHOD
As voxel generation method, consider a P × Q × R (horizontal × vertical × horizontal) virtual space, as shown in Figure  2. Using Shiraki et al's method [33], the voxel value V ijk of the volumetric display shown in the blue box is determined by adding the pixel values a ij and b kj of the input images to be displayed. Thus, the voxel value is given in Equation (1), where i, j, k are arbitrary coordinates of the system (x, y, z), and λ is a constant for normalizing the voxel values.
In the directional volumetric display using 3D crystals, the voxels are overlapping on the x-and z-axes, so adding them together from on the z-axis represents 1 pixel of the front image, and adding them on the x-axis represents 1 pixel of the side image. Therefore, the pixel value A ij of the image observed from the front is expressed using Equation (2), and the pixel value B kj of the image observed from the side is expressed using Equation (3).
The correspondence between voxels and threads in a directional volume display using threads is explained. Figure 3 (a) depicts a view of Figure 2 from the y-axis direction, with voxels overlapping in the y-axis direction. For each voxel, a thread is placed. Then, the voxel value calculated using Equation (1) is assigned to the corresponding height of the corresponding thread. Owing to the constraint of the thread arrangement, they do not overlap on the x-and z-axes, so 1 pixel of the front image is represented by binding R threads together from the x-axis direction. Similarly, a pixel in the side image is represented by binding P threads from the zaxis direction [20]. Figure 3 (b) shows the correspondence between the threads and displayed images. Four threads are bundled and observed to display one pixel in the horizontal direction of displayed images.
When displaying an image with a resolution of P × Q (horizontal × vertical) in the conventional method, P threads are used for each horizontal resolution of 1 pixel, and a total of P 2 threads are required for the entire display. As shown in Figure 3, four threads are used to display 1 pixel of the image,  so only 4-pixel images can be displayed. Thus, the higher the resolution of the image, the higher the difficulty to fabricate the display. (1) VOLUME 4, 2016

B. PROPOSED METHOD
To achieve a higher resolution of the displayed image, reducing the number of threads required for 1 pixel in the horizontal resolution by a factor of 1/N and allocating the reduced number of threads to a new horizontal resolution was proposed.
Equation (4) is used to determine the voxel value V of size P × Q × P generated from images PN × QN, which is N times expansion of the images displayed in the front and side views; where a i ′ j and b k ′ j are the pixel value of the image in the front and side views, respectively. Further, i, j, k is an arbitrary coordinate of the system (x, y, z), and λ is a constant for normalizing the voxel value.
The coordinates i ′ and k ′ of each image are calculated from Equations (5) and (6), where t x and t z are the coordinates of the thread placement in the x-z plane corresponding to each voxel. The range of the constant N depends on the number of threads required to display 1 pixel in the horizontal resolution of the images, and is expressed in Equation (7).
When the image is projected onto threads, a pixel of the front image is represented by binding P/N threads from the x-axis direction. A pixel of the side image is also represented using P/N threads bound from the z-axis direction. Figure  4 (a) depicts a view of Figure 2 from the y-axis direction, with voxels overlapping in the y-axis direction. Each thread is placed in each voxel that contains the coordinates calculated using Equations (5) and (6). Then, the voxel value calculated using Equation (4) is assigned to the corresponding height of the corresponding thread. Figure 4 (b) shows the correspondence between the threads and displayed images. Two threads are bundled and observed to display one pixel in the horizontal direction of the displayed images. In the conventional method (Figure 3), four threads are used to display 1 pixel of the image, so only 4-pixels images can be displayed. In this method, two threads are used to display one pixel of an image (Figure 4), so the image with eight pixels can be displayed.
This method allows the representation of images with a horizontal resolution of P pixel using P threads. However, reducing the number of threads reduces the projected image quality. Thus, how many threads per 1 pixel can be used to achieve high image quality while maintaining the input image quality is examined.

A. EXPERIMENT
To verify the effectiveness of the proposed method, experiments were conducted. Figure 5 shows the relationship between the projector, threads, and observers. In Figure 6, the input images used in this study are 64 × 64 images. Input images were collected from "Standard Image / Sample Data" [34]. These can be used as standard images for simulation analysis. To obtain the simulation and projection results, the personal computer (PC) used was a Mac book pro (Apple Inc.). Table 1 shows the specifications of the PC, and Table 2 shows the information of the software used. Figures 6 (a) and (b) are used to form a projection image, which is then projected onto the threads to display Figure  6 (a) and (b) from the front and side views, respectively. Similarly, the projection image is created using Figures 6 (c) and (d), which are displayed from the front and side views, respectively. The simulation in this study is performed using multiple data of the number of threads, which is N times the resolution of the input images. In addition, although the simulation of threads would normally have a gap, the simulation image was generated with an ideal arrangement of threads with no gaps to facilitate the evaluation of image quality. The  image quality of simulated images was evaluated using the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). PSNR is an image quality index that indicates how much noise is contained in a comparison image compared with the original image. SSIM is an image quality evaluation index that compares the luminance, contrast, and structure of a comparison and original images, and the higher the value, the higher is the image quality. The evaluation using SSIM is said to be close to human perception. Mean absolute error (MAE) and mean square error (MSE), which are used in evaluating regression models, are used to analyze the error between the input image and simulation result. The closer MAE and MSE are to zero, the smaller the error and the closer the comparison image is to the input image. Further, the projected image was projected onto 64 threads and 384 threads, and the displayed image was observed. Figure 7 shows the simulated image of the projection result.  Figure 7 are the conventional method's results. Table 3 shows the results of the evaluated output images using PSNR. Table 4 shows the results of the evaluated output images using SSIM. Table 5 shows the results of the evaluated output images using MAE. Table 6 shows the results of the evaluated output images using MSE.  From Figure 7, it is confirmed that this method can output high-resolution images. Further, from the results of the PSNR, SSIM, MAE, and MSE values, it can be confirmed that the image quality is poor when 64 threads are used, and improves as the number of threads increases. Furthermore, it is confirmed that the image quality when using more than 320 or 384 threads is almost equal to the image quality of the output image using 4,096 threads, which is the conventional method. Thus, it is feasible to display the image while maintaining the image quality similar to the output images of the conventional method using 5 or 6 threads per 1 pixel of the original image's horizontal resolution.

C. RESULTS OF PROJECTION
Based on the simulation results, two types of directional volumetric displays, 64 and 384 threads, were fabricated, and projected images were projected onto the threads. The overview of the projection system for the directional volumetric display is shown in Figure 8. Images were projected using an MH550 projector (BenQ Japan Co., Ltd., Japan). The specifications of the projector are listed in Table 7. The projector was installed at a distance of 1.48 m from the directional volumetric display and 0.70 m above the floor. Vinymo MBT (Nagai Yoriito Co., Ltd., Japan) was used as the thread. The number of threads that constructed the directional volumetric display was 59 of 64 and 315 of 384. The failure to place the threads was that the constraints on thread placement of Shiraki et al. [20] could not be met. Figure 9 shows the result of the projection onto 64 threads and Figure 10 shows the result of projecting onto 384 threads.     Figure 7 (a) Figure 7 ( Figure 11 that the images could not be observed from other directions. The conventional method can only display images of 20 pixels (simple symbols and characters) with 400 threads [20], but the proposed method can display images of 64 pixels (more complex images) with 384 threads in Figure 10 and visualization 1.

V. DISCUSSION
This study compares the conventional and proposed methods, a comparison of input and simulated images, a comparison of projected and simulated results, and a discussion of the limitations of the proposed method were discussed.

A. COMPARISON OF THE PROPOSED METHOD AND THE CONVENTIONAL METHOD
To display multiple images, the pixel values of images displayed in different directions are added to the voxel calculation. The pixel values of images to be displayed in the other direction are errors and are included in all voxels. In the conventional method, all voxels, including the noise, are added in the depth direction to obtain the pixel values of displayed images. The noise is smoothed via adding voxels in the depth direction. This method reduces the number of voxels added in the depth direction and allocates the voxels to other pixels in the displayed image to achieve higher resolution.
Although there are differences in the results depending on the image used, Tables 3, 4, 5, and 6 show that the values of PSNR, SSIM, MAE, and MSE are low when the thread count is 64 because no smoothing is performed, and these values improve as the thread count increases. It was verified that using 5 to 6 threads per 1 pixel of the input image can be displayed while maintaining the image quality similar to the conventional method.
Finally, a comparison of the computational complexity of the conventional and proposed methods is presented. Onotation denotes the amount of computation. The computational complexity of the conventional and proposed methods depends on the number of voxels to be computed. In the conventional method, all voxels are computed, so the computational complexity is O(n 3 ). In this method, the number of voxels to be computed depends on the value of N in Equations (5) and (6), and the computational complexity changes. When N = 1, the computational complexity is O(n 3 ); however, the process is the same as the conventional method. When N ̸ = 1, the computational complexity is O(n 2 ), which confirms that the proposed method is superior to the conventional method.

B. COMPARISON OF INPUT AND SIMULATION IMAGES
The values of PSNR, SSIM, MAE, and MSE improved as the number of threads increased, and it was confirmed that the image quality was equivalent to that of the conventional method at 320 or 384 threads. However, from Tables 3 and  4, it is confirmed that the PSNR and SSIM values do not increase, even when the number of threads increases. For example, in Figure 7 (a) in Tables 3 and 4, the values of PSNR and SSIM are larger when 384 threads are used than when 512 threads are used because the threads corresponding to noisy voxels are removed, which improve the image quality.  Tables 3 and 4, it was observed that PSNR and SSIM differed depending on the combination of the two input images. For example, the front image in Figure 7 (b) has a lower SSIM value than the front image in Figure 7 (a) ( Table 4). Further, Tables 5 and 6 show that the MAE and MSE values of the front and side images in Figure 7 (b) are larger than those of the frontal and side images in Figure 7 (a), and the errors are larger. This is because the current formulas used for voxel generation do not consider features such as the color tone and edges of the image; furthermore, the strength of the noise in the image that can be confirmed from the generated voxels depends largely on the combination of the images used for generation. Therefore, the image quality of the displayed image would depend on the combination of the thread arrangement and the input images.

C. COMPARISON OF PROJECTION RESULT AND SIMULATION IMAGES
The displayed images of the projection results in Figures 9  and 10 were close to the results of the simulation with 64 and 384 threads in Figure 7. It was also confirmed from Figure  11 that the image could not be observed from any direction other than the display direction. However, there is some error between the projection results in Figures 9 and 10 and the results of the simulation with 64 and 384 threads in Figure 7.
The following are two reasons for the discrepancy between the projected and simulated results: (1) The thread is placed manually, resulting in an error between the placement of the thread in the simulation and the placement of the actual hanging thread. Thus, threads obstruct the rays, which should not hit it, resulting in noise. To address this issue, it is crucial to place the threads accurately, but this is not practical. (2) The brightness of the projected and the simulated images are similar, thus when the image is projected, the brightness value changes depending on the reflectance of the thread, which is assumed to be different from the actual simulation result. To address this issue, it is important to consider the reflectance of the projected object. Huang et al. [35], [36] previously succeeded in projecting an image similar to the actual input image by correcting the input image using the information on the projection surface captured by a camera and an ambient light. The camera used in this study was only for capturing the projection results. In the future, it is necessary to use a camera to obtain projection surface information and realize projection based on the information of the projection surface, thereby reducing the error with the simulation results.

D. LIMITATIONS
It was confirmed that by representing each pixel of the input image with 5-6 threads of horizontal resolution, the image quality of the displayed image was almost equal to that of the conventional method. A directional volumetric display capable of displaying (256 × 256)-pixel images is considered to be fabricated using approximately 1,500 threads. Currently, images of 4-5 pixels per thread are projected using a projector. Therefore, when using a projector capable of projecting an image of 1,920 × 1,080 pixels, the number of threads that can project an image is approximately 400. So, to project images onto 1,500 threads, it is necessary to reduce the number of pixels of the image projected onto each thread or increase the number of projectors that project the image.
The video in visualization 1 takes an average of 0.15 s to create a single projected image and can display only approximately 6.6 frames per second. In addition, as the resolution increases, the time required to create a projected image increases, so in the future, using input images with 256 × 256 pixels will take 16 times longer than using input images with 64 × 64 pixels. When considering a display at 30 frames per second, the projection image generation would need to be approximately five times faster when using a 64pixel input image. Similarly, when input images with 256 × 256 pixels are used, the acceleration is approximately 80 times.
To realize an interactive directional volumetric display that updates the displayed image according to the observer's observation position, directional displays to 0 • and 30 • , 0 • and 60 • , and 0 • and 90 • were simulated using 384 threads. The results are shown in Figure 12. Figures 12 (a) However, when the directional volumetric display is observed from a position other than the front and side directions, the overlapping of threads and the expansion and shrinkage of the display size in the horizontal direction cause a loss of the displayed image. This is because the proposed method only optimizes the placement of threads in the 0 • and 90 • directions owing to reducing the number of threads assigned to each pixel. Therefore, it is necessary to develop a model based on this method to place the threads so that the quality of the displayed image is maintained uniformly, regardless of the viewing direction.

VI. CONCLUSION AND FUTURE WORK
This study proposes and validates a new design algorithm to improve the resolution of a directional volumetric display using threads and a projector. The conventional method requires threads of the square of the horizontal resolution to build a directional volumetric display, whereas the proposed method represents the directional volumetric display with threads of at least the horizontal resolution. By representing each pixel of the input image with 5-6 threads of horizontal resolution, it was confirmed that the image quality of the displayed image was almost the same as that of the conventional method.
In the future, with the goal of practical application in the entertainment field, the aim is to fabricate a directional volumetric display capable of displaying (256 × 256)-pixel images with approximately 1,500 threads. As described in Subsection V-D, to fabricate the directional volumetric display capable of displaying images of 256 × 256 pixels, it is necessary to reduce the number of pixels projected on each thread or increase the number of projectors that project images. However, in the former case, the resolution cannot be higher than the projector's performance (1,920 × 1,080 pixels). In addition, although there are projectors that can display images with higher resolution than 1,920 × 1,080 pixels, they are expensive. Therefore, a directional volumetric display that displays images with 256 × 256 pixels will be realized by projecting images using multiple projectors.
In this study, only the CPU was used to generate projected images. Matsumoto et al. [21] achieved an acceleration of up to 27.45 times faster than using only the CPU using NVIDIA GeForce GTX 1050 GPU in generating projected images when using a 64-pixel input image. Therefore, using a highperformance GPU, such as NVIDIA GeForce RTX 3090, to generate projected images, it may be possible to display (256 × 256)-pixel images at 30 frames per second. Further, a new interactive system will be developed using a directional volumetric display to present multimedia.