Optimized Binary Patterns by Gradient Descent for Ghost Imaging

Ghost imaging reconstructs images using a single-element photodetector; it performs imaging by illuminating an object with binary modulation patterns. This technique has various advantages, including a wide wavelength, noise robustness, and high measurement sensitivity. However, one challenge is the low image quality in the undersampling. The examination of modulation patterns is intended to solve this issue. In ghost imaging, randomly generated or basis patterns have been studied as modulation patterns; however, in undersampling, random patterns exhibit noise robustness but low image quality, whereas basis patterns exhibit high image quality but are sensitive to noise and low resolution. Thus, ghost imaging requires patterns that simultaneously achieve high image quality, resolution, and noise robustness. This study proposes a method of pattern optimization using gradient descent and a binarization method to further improve image quality. Numerical simulation and experimental results show that the proposed approach offers high image quality, high resolution, and robustness to noise.


I. INTRODUCTION
Conventional imaging records an image using a sensor composed of several light detectors, such as charge-coupled devices. However, ghost imaging [1], [2] is a unique technique that uses only a single-element photodetector. Figure 1 shows a schematic of ghost imaging, first, random binary patterns on a digital micromirror device (DMD), which is high speed binary display, modulate a target object. Then, the modulated light from the target object is collected by a lens, and its intensity is recorded with a single-element photodetector. This intensity is obtained by sequentially modulating the patterns; subsequently, we reconstruct the target object's image by performing a correlation calculation between the measured data and the known modulation patterns. With the features obtained using a single-element photodetector, ghost imaging offers several advantages, such as the ability to operate in a low-light environment, as well as high-speed and broad-wavelength imaging. Hence, ghost imaging can be applied to various fields, such as remote sensing [3], object tracking [4], high-speed cytometry [5], and terahertz imaging [6].
The associate editor coordinating the review of this manuscript and approving it for publication was Md. Abdur Razzaque . However, there are several challenges to ghost imaging, such as low image quality due to the random patterns generally used as modulation patterns. One of the studies to solve this problem is the development of modulation patterns. Particularly, modulation patterns of orthogonal basis [7]- [10] (also known as single-pixel imaging) have been proposed, and Hadamard patterns have been used for hyperspectral imaging [11], phase imaging [12], [13], color holography [14], and high-definition imaging [15]. Although these techniques improved the image quality of ghost and single-pixel imaging, these patterns have limited resolution in undersampling because they obey spatial frequency transforms. Such patterns are sensitive to noise for the same reason. Thus, ghost imaging requires modulation patterns that simultaneously achieve high image quality, robustness against noise, and high resolution.
In this study, we propose a two-step optimization of modulation patterns to improve image quality by offering high resolution and robustness against environmental noise. We apply a sophisticated optimization technique on the basis of deep learning to better optimize modulation patterns. In recent years, deep learning has achieved great successes in image noise reduction [16] and super-resolution [17]. The most common examples of applying deep neural networks to ghost imaging in previous studies are both refining the image reconstructed by correlation calculation with deep neural network [18], [19], and directly reconstructing the image with networks without using correlation calculations [20]. However, these methods have some disadvantages, such as additional and high computational load and decreased noise robustness which is a feature of ghost imaging.
In this study, we performed the two-step pattern optimization using Keras framework with Tensorflow backend as an optimization solver, not as a deep learning framework. We implemented correlation calculation into the framework to obtain optimized patterns. Thus, similar to conventional ghost imaging, the proposed method can obtain reconstruction images with better image quality to use correlation calculations in the reconstruction process without using deep learning network and noise reduction methods [21]. Optimization is performed in two steps: first, optimized grayscale patterns are acquired; then, the optimized grayscale patterns are binarized with as little negative effect as possible. Our concept follows that of a previous study [22], whereby the learning of modulation patterns is incorporated into a deep neural network by considering them as network parameters. Unlike spatial frequency-based patterns [7], [8], the proposed modulation patterns improve image quality while retaining robustness against noise and high resolution, even for undersampling. We can therefore achieve ghost imaging with an improved image quality while retaining high resolution and noise robustness using the proposed method.
The remainder of this paper is organized as follows. First, we describe the proposed method; then, we present the results of a simulation and an optical experiment. In simulation, we compare noiseless and noisy cases. Furthermore, we compare the proposed binarization method with simple binarization. Finally, we present the conclusions.

II. PROPOSED METHOD
The starting point of the proposed method is differential ghost imaging (DGI) [23], which is expressed as where I i (x, y) is the modulation pattern with gradation, the subscript i = {1, 2, . . . , N } indicates the index of the modulation patterns, N is the number of patterns, represents the ensemble average for all the measurements, and O(x, y) is the reconstructed image. S i and R i represent the i-th total light intensities of the object and the patterns, as expressed by where X and Y represent the width and height of an image and T (x, y) represents the target object. The above calculations are differentiable, hence, to optimize the modulation patterns via gradient descent, we obtain gradients using the error between the reconstructed image and the original target object. Afterward, we update the modulation patterns according to the gradients. By repeating these updates, we can obtain the optimized patterns. The optimized patterns have nonbinary values; thus, our proposed method only performs well using spatial light modulators such as liquid crystal displays, which can display grayscale patterns with low refresh rates. Although the use of binary spatial light modulators such as DMDs can project binary patterns at high speed, simple binarization of the optimized grayscale patterns may degrade the quality of the reconstructed images. Hence, we need pattern optimization based on our binarization assumption. We propose a two-step optimization scheme with scaling factors. Let us begin by explaining the scaling factor.

A. SCALING FACTOR
The scaling factor is used to improve the accuracy of binary neural networks [24]. One binary illumination pattern has a scaling factors α i , which is a scalar value. By approximating y) is the desired binary pattern), we can express S i and R i as If optimal scaling factors α i can be obtained, it will reduce the binarization error of the patterns. The scaling factors are decided to minimize the differences between the binary patterns with α i and the grayscale patterns; thus, we optimize α i as where the grayscale and binary patterns are represented as I i and B i , which are R X ×Y vectors, with each vector element denoted by otherwise, because an actual optical system cannot represent negative values. In this study, we initialize I i,k ranging from 0 to 1. Since the optimal scaling factors α i in Eq. (6) can be simply found by Finally, the proposed reconstruction calculation with a scaling factor α i is expressed as

B. SUMMARY OF THE PROPOSED METHOD
Algorithm 1 shows the overall flow of the proposed method combining two-step optimization with the scaling factor. In the first step, the optimized grayscale patterns I i (x, y) with floating point precision are obtained via stochastic gradient descent (SGD) with the state-of-the-art optimizer, Adam [25]. Adam (adaptive moment estimation) is one of the most commonly used optimization methods in neural networks. It is one of the derivatives of gradient descents, which allows for more efficient optimization. The feature of Adam is that it can apply an adaptive learning rate to each parameter by using the previous gradient. We used the default parameters of the Adam optimizer. The gradients are obtained by performing the DGI calculation of Eq. (8). The normalize function in Algorithm 1 normalizes the reconstruction results ranging from 0 to 1, which is expressed as where O max (x, y) and O min (x, y) are the maximum and minimum values of O(x, y), respectively. SGD proceeds with optimization using mini-batches that divide entire datasets into subsets. In the second step, we obtain binary patterns optimized from grayscale patterns using SGD with Adam, as well as a binarization method with scaling factors α i to reduce binarization's negative effects. One set of modulation patterns and scaling factors is obtained per one training dataset.

III. SIMULATION AND EXPERIMENT
We conduct simulations and optimal experiments to verify the effectiveness of the proposed method. Using Python 3.5.  B new ← Optimizer(B, ∂E ∂B ) // Adam [25] is used as optimizer 15: 16: B ← Binarization(I) 17:

18: end for
A. EVALUATION OF NOISELESS ENVIRONMENT Table 1 and Fig. 2 show the reconstructed images using each modulation pattern. Table 1 presents the evaluated image qualities for the reconstructions. Figure 2 shows the reconstructed images when the number of measurements is 1,024. Random modulation patterns are generated with random numbers; for the basis pattern, we used a state-of-the-art pattern called Origami, which was proposed in [8]. These patterns are binary. The insets in Fig. 2 show the enlarged parts of the reconstructed images. It can be seen that the proposed method has smooth tones and high resolution. Table 1 shows the average peak-signal-to-noise ratios (PSNRs) for the 4,000 test data. The percentage indicates the ratio of the number of modulation patterns to the size (128 × 128[pix]) of the original images. From Table 1, the proposed method resulted in an improved image quality compared with that via a random pattern. The image quality  obtained via the proposed method was confirmed to be lower than that via the basis patterns without noise. From Fig. 2, we can see that the resolution of the proposed method is higher than that of the basis pattern. Since noise affects measurements in a real environment, our next step is to investigate the noise tolerances of these methods. Figures 3 and 4 compare between the basis and proposed patterns in a noisy environment. We added white Gaussian noise to modulation patterns because we assume time-varying noise such as smoke, rain, and fog as a noise environment. The addition of noise intensity is expressed via an SNR, which is defined as the ratio of the mean squared total values of the modulation pattern and added noise, as expressed by 10 log 10 (S pattern /N noise ), where S pattern and N noise are the mean squared total values. The noise increases as the SNR decreases; a SNR of 10 dB is 1/10 of the intensity of the modulation pattern, and a SNR of 0 dB represents the same noise as the intensity of the modulation pattern. The number of patterns is 1,024. Figure 3 shows the reconstructed image and its quality when white Gaussian noise is applied in 2 dB steps from 10 to −10 dB. Figure 4 shows the image quality evaluation that the noise intensity to the image quality of reconstruction in the basis and proposed patterns. The evaluation presents the average PSNR of 4,000 test data when white Gaussian noise is applied from 10 to −10 dB. Figure 3 shows the deterioration of the image quality for the basis patterns. However, the proposed patterns maintain image quality, even with the increase in noise intensity.

B. EVALUATION OF NOISY ENVIRONMENT
From Fig. 4, we can see that there is already no significant difference in image quality of approximately 10 dB, and that the image qualities of the basis and proposed patterns reverse at 8.64 dB. On the basis of these results, we find that the proposed method has higher noise robustness than the basis patterns.
Because the basis pattern is a method of projecting a spatial frequency pattern, reconstructed images with sufficient resolution cannot be obtained with few measurements. In addition, when noise is applied to the spatial frequency-based basis pattern, the noise is strongly reflected in the reconstructed images. On the other hand, the proposed method is based on random patterns, so it has high noise immunity and high resolution.

C. EVALUATION OF PROPOSED BINARIZATION
To confirm the effectiveness of the proposed binarization, we compared it with a simple binarization scheme using a threshold value. In the simple thresholding method, the grayscale patterns were binarized according to B i,k = 1 if I i,k ≥ 0.5 or B i,k = 0 if otherwise, where I i,k is obtained from the first optimization step in Algorithm 1.     Table 2 shows the results. The evaluation is the average PSNR for the 4,000 test data in Table. 2. These results show that the proposed binarization is superior to the simple thresholding binarization in the subjective and quantitative evaluations of all measurement times.

D. OPTICAL EXPERIMENT
Finally, we conducted an experiment using an actual optical system. Figures 6 and 7 show the optical system and reconstructed images with and without noise, respectively. The insets in Fig. 7 show the enlarged parts of the reconstructed images. As sources of noise, we used smoke from incense sticks placed between a target object and the DMD. As a result of noise measurement, the noise intensity was ranging from −5 to −10 dB approximately. The PSNRs in Fig. 7 were measured by comparing a reconstructed image in a noisy environment with one in a noiseless environment. The number of modulation patterns is 1,024. In a noiseless environment, the proposed patterns improve the contrast of the reconstructed images compared with that of the random patterns. Although the proposed patterns exhibit less contrast with reconstructed images than do the basis patterns,  the resolution of the reconstructed image is higher than that using basis patterns. In a noisy environment, the proposed patterns improve the quality of the reconstructed images compared with that of the basis patterns while retaining robustness against noise and high resolution, even for undersampling.

IV. CONCLUSION
In conclusion, we have proposed binary modulation patterns with scaling factors optimized using SGD for ghost imaging. We found that the proposed method has higher image quality than that obtained using random patterns. We also showed that the proposed method has a higher resolution and noise robustness than those obtained using basis patterns. From these results, we could conclude that the proposed method could improve image quality while retaining a high resolution and noise robustness. In this study, we used a universal dataset to optimize binary patterns. However, in practice, we will obtain better results if we use an application-specific dataset. In future work, it will not be easy to increase the number of pixels in the current Keras and Tensorflow-based framework because of the lack of memory in a graphics processing unit (GPU). Hence, we intend to increase the number of pixels via full-scratch GPU programming. IKUO