Loading [MathJax]/extensions/MathMenu.js
Compensating Image Overlay 16QAM Using Dual Cameras for Rolling Shutter-Based Screen Camera Communication | IEICE Journals & Magazine | IEEE Xplore

Compensating Image Overlay 16QAM Using Dual Cameras for Rolling Shutter-Based Screen Camera Communication


Abstract:

Image overlay uplink screen camera communication consisting of a small display with LED backlight modulation and an indoor rolling shutter camera has been studied. A prev...Show More

Abstract:

Image overlay uplink screen camera communication consisting of a small display with LED backlight modulation and an indoor rolling shutter camera has been studied. A previous study demonstrated that image overlay QPSK was achieved by compensating pixel values of the overlaid image using dual cameras. However, even after the compensation, variation in the pixel values remained, making it difficult to realize further M-ary modulation. In this study, a compensation method taking both the overlaid image and backlight modulation into account is proposed. Constellation, error vector magnitude, and bit error rate for image overlay 16QAM are experimentally evaluated using dual cameras.
Published in: IEICE Communications Express ( Volume: 14, Issue: 6, June 2025)
Page(s): 242 - 245
Date of Publication: 11 April 2025
Electronic ISSN: 2187-0136

Funding Agency:


SECTION 1.

Introduction

High-speed screen camera communications (SCCs) imperceptible to the human eye that use a display with backlight LEDs for transmitting and a rolling shutter (RS) camera for receiving have been studied [1]–​[3]. Reference [1] evaluates RS-based SCC (RS-SCC) using inverted 4 pulse position modulation (I-4PPM), which provides constant average screen luminance. However, in I-PPM, symbol length cannot be shortened due to the limited exposure time of RS camera. It is difficult to increase bitrate by using M-ary modulation. In addition, Reference [3] evaluates RS-SCC using on-off keying (OOK). Since OOK does not provide constant average screen luminance, neural network processing is used to reduce the impact of the overlay images.

On the other hand, Reference [2] evaluates RS-SCC using phase shift keying (PSK), which also provides constant average screen luminance. To compensate pixel values of the overlaid image, the backlight modulation and the overlaid image are separated by using dual cameras with different exposure times. However, M-ary PSK is limited to quadrature PSK (QPSK) and the details of the compensation method for the overlaid image using dual cameras are not revealed in the reference.

In a previous study, 16 quadrature amplitude modulation (16QAM) was achieved using a single camera in optical camera communication without the overlaid image [4]. To reveal the details of the compensating method, we evaluated the constellation, and error vector magnitude (EVM) for the image overlay QPSK and demonstrated the effectiveness of the overlaid image compensation using dual cameras [5]. However, even after the compensation, there are still amplitude and phase variations due to the overlaid images. It was difficult to achieve further M-ary PSK or QAM.

In this study, we improve the overlaid image compensation using dual cameras. To achieve image overlay 16QAM, gamma correction, which takes both the overlaid image and backlight modulation into account, is newly introduced for the compensation. The compensation based on gamma correction that takes both the overlaid image and backlight modulation into account is compared with that takes only the overlaid image into account and compared with a single camera alone. The amplitude and phase characteristics, constellation, EVM, and bit error rate (BER) for the image overlay 16QAM are experimentally evaluated.

SECTION 2.

A Model for Overlaid Image Compensation Using Dual Cameras

Figure 1 shows a configuration of RS-OCC with dual cameras for overlaid image compensation. The transmitter is backlight LEDs for a small liquid crystal display (LCD). The sender can transmit data using backlight LEDs while viewing LCD screen. The receiver is dual RS cameras with different exposure times for overlaid image compensation.

The dual cameras are the same other than the exposure times.

Let the pixel value of the overlaid image on the LCD screen be p(k,l). The horizontal and the vertical pixel numbers of a full high definition (HD) RS camera are k=0,1919, and l=0, 1079, respectively. On the other hand, let the sampling interval of RS camera be T_{px}, let the symbol length of the modulated signal be T_{s}, let the short exposure time of RS camera be T_{short}, and let the long exposure time of RS camera be T_{long}. The modulated signal, S(t) is expressed as \begin{equation*} S(t)=b_0+a_i \cos \frac{2 \pi}{T_s} t-a_q \sin \frac{2 \pi}{T_s} t,\tag{1}\end{equation*}View SourceRight-click on figure for MathML and additional features. where the bias is b_{0} and the symbol is (a_{i},a_{q}). Since the received pixel value of RS camera, P(k, l) is sampled every T_{px}, P(k, l) is expressed as \begin{equation*} P(k,l)= \int_{0}^{T}p(k,l)S(t-lT_{px})dt,\tag{2}\end{equation*}View SourceRight-click on figure for MathML and additional features. where the exposure time of RS camera is T and the sampled pixel number is l.

Fig. 1 - A configuration of uplink RS-OCC with dual cameras for overlaid image compensation.
Fig. 1

A configuration of uplink RS-OCC with dual cameras for overlaid image compensation.

When the exposure time of RS camera is short, the received pixel value of RS camera with short exposure time, P_{short}(k,l) is expressed as \begin{align*} & P_{{short}}(k, l)=\left[\int_{0}^{T_{{short}}} p (k, l) \left\{b_0+a_i \cos \frac{2 \pi}{T_s}\left(t-l T_{p x}\right)\right.\right. \\ & \left.\left.-a_q \sin \frac{2 \pi}{T_s}\left(t-l T_{p x}\right)\right\} d t\right]^{\gamma_{{short}}(p, S)} \\ & \cong\left[p (k, l) \left[T_{{short}}b_0+\int_0^{T_{{short}}}\left\{a_i \cos \frac{2 \pi}{T_s}\left(t-l T_{p x}\right)\right.\right.\right. \\ & \left.\left.\left.-a_q \sin \frac{2 \pi}{T_s}\left(t-l T_{p x}\right)\right\} d t\right]\right]^{\gamma_{{short}}(p, S)} \quad\left(T_{{short}} < T_s\right),\tag{3}\end{align*}View SourceRight-click on figure for MathML and additional features. where \gamma_{short}(p,S) is gamma coefficient when the exposure time of RS camera is short. The coefficient, \gamma_{short}(p,S) indicates that the overlaid image compensation needs to take both the pixel values of the overlaid image, p(k,l) and the backlight modulation, S(t) into account.

Since the exposure time, T_{short} is shorter than the symbol length, T_{s}, RS camera with short exposure time receives both the pixel values of the overlaid image, p(k,l) and the backlight modulated signal, S(t).

On the other hand, when the exposure time of RS camera is long, the received pixel value of RS camera with long exposure time, P_{long}(k,l) is expressed as \begin{align*} & P_{{long}}(k, l)=\left[\int_{0}^{T_{{long}}}p(k,l)\left\{b_0+a_i \cos \frac{2 \pi}{T_s}\left(t-l T_{p x}\right)\right.\right. \\ & \left.\left.-a_q \sin \frac{2 \pi}{T_s}\left(t-l T_{p x}\right)\right\} d t\right]^{\gamma_{{long}}(p)}\quad\left(T_{{long}}\gg T_s\right),\tag{4}\end{align*}View SourceRight-click on figure for MathML and additional features. where \gamma_{long}(p) is gamma coefficient when the exposure time of RS camera is long. Since the exposure time, T_{long} is much longer than the symbol length, T_{s}, it is assumed that T_{long}\approx nT_{s}, where n is an integer. By substituting T_{long}\approx nT_{s} into Eq. (4), S(t)\approx b_{0} and Eq. (4) is approximated by \begin{equation*} P_{{long}}(k, l) \cong\left\{p(k, l) T_{{long}} b_0\right\}^{\gamma_{{long}}(p)} \quad\left(T_{{long}} \approx n T_s\right).\tag{5}\end{equation*}View SourceRight-click on figure for MathML and additional features.

The backlight modulated signal, S(t) becomes a constant value b_{0} and only the pixel value of the overlaid image, p(k,l) is received.

From Eq. (3) and (5), the ratio of the received pixel value between RS cameras with short and long exposure times, P_{short}(k,l)/P_{long}(k,l) is given by \begin{align*}& \frac{P_{{short}}(k, l) / p(k, l)^{\gamma_{{short}}(p, S)}}{P_{{long}}(k, l) / p(k, l)^{\gamma_{{long}}(p)}}=\left[T_{{short}} b_0+\int_0^{T_{{short}}}\left\{a_i\right.\right.\\ & \left.\left.\cos \frac{2 \pi}{T_s}\left(t-l T_{p x}\right)-a_q \sin \frac{2 \pi}{T_s}\left(t-l T_{p x}\right)\right\} d t\right]^{\gamma_{s h o r t}(p, S)} \\ & /\left(T_{{long}} b_0\right)^{\gamma_{long}(p)}.\tag{6}\end{align*}View SourceRight-click on figure for MathML and additional features.

From Eq. (6), if gamma coefficients of RS camera with long and short exposure times, \gamma_{long}(p) and \gamma_{short}(p,S) are known, the pixel value of the overlaid image, p(k,l) can be removed. It is important that gamma correction of RS camera with short exposure time needs to take both the pixel values of the overlaid image, p(k,l) and the backlight modulation signal, S(t) into account. In the experiments, a single camera alone and gamma coefficient of RS camera with short exposure time, \gamma_{short}(p) that takes only the overlaid image into account are also evaluated for comparison. In this case, \gamma_{{short}}(p, S) is replaced by \gamma_{{short}}(p) in Eq. (6).

In addition, RS camera receives the same data in the horizontal k direction (see Fig. 1). The ensemble average value, P_{en}(l) from k=0 to 1919 in the l direction of the image is expressed as \begin{equation*} P_{e n}(l)=\frac{1}{1920} {\sum}_{k=0}^{1919} P(k, l). \tag{7}\end{equation*}View SourceRight-click on figure for MathML and additional features.

The ensemble average is also available for the overlaid image compensation by Eq. (6).

SECTION 3.

Procedure for Overlaid Image Compensation Using Dual Cameras

Gamma coefficients, \gamma_{long}(p),\ \gamma_{short}(p, S), and \gamma_{short}(p) were measured in advance. Figure 2(a) shows measured gamma coefficient, \gamma_{long}(p) where grayscale pixel values of a white image on the LCD screen were changed and received with RS camera with the exposure time, T_{long} at a distance of 1.9 meters. In \gamma_{short}(p,S), the pixel value of the white image is changed in accordance with the normalized backlight modulation signal, S(t) as shown in Fig. 2(c). On the other hand, in \gamma_{short}(p), the modulation signal, S(t) was set to only the bias, b_{0} as shown in Fig. 2(b).

Figure 3 shows the procedure for the overlaid image compensation using dual cameras. First, using \gamma_{long}(p), gamma correction for the long-exposure RS camera is performed to obtain the pixel value of the overlaid image, p(k,l). Next, using \gamma_{short}(p,S), backlight modulation signal, S(t) can be obtained from the pixel value of the overlaid image, p(k,l) and the pixel value of the short-exposure RS camera, P_{short}(k,l). The increase in computational complexity due to the compensation is small because the gamma coefficients measured in advance are used as look-up tables. The number of multiplications with look-up tables required for gamma correction for the long- and short-exposure RS camera is 1080, respectively. Finally, symbol is determined based on the correlation between the demodulated and the transmitted symbol, (a_{i},a_{q}).

Fig. 2 - Measured gamma coefficients.
Fig. 2

Measured gamma coefficients.

Fig. 3 - Procedure for overlaid image compensation using dual cameras.
Fig. 3

Procedure for overlaid image compensation using dual cameras.

SECTION 4.

16QAM Experiments

4.1 Experimental Parameters

Table I shows specifications of LCD transmitter and RS camera receiver. The dual cameras are the same RS cameras with exposure times, T_{short} and T_{long} estimated to be around 594.4 μs and 16.7 ms, respectively. The backlight LED modulation is 16QAM with a bitrate of 1682 bit/s.

Figure 4 shows 3 typical standard images, “ship”, “stained glass”, “butterflies” [6] and their ensemble average values, P_{en}(l) used for the overlaid imges. The image, “ship” has high pixel values that change slowly. On the other hand, the images, “stained glass” and “butterflies” have low pixel values that change fast.

4.2 16QAM Waveform

Gamma correction taking both p(k,l) and S(t) into account was compared with that taking only p(k,l) into account and compared with a single camera alone to verify the improvement of 16QAM waveforms by the overlaid image compensation. Figure 5 shows typical examples of 16QAM waveforms of the overlaid images, “ship”, “stained glass”, and “butterflies,” where a single camera alone, dual cameras compensation taking only p(k,l) into account, and compensation taking both p(k,l) and S(t) into account are compared. In addition to each waveform, an ideal transmit waveform is shown by the green line for comparison.

Compared with the single camera alone, the dual cameras compensation taking only p(k,l) into account improves 16QAM phase but not the amplitude. On the other hand, the compensation taking both p(k,l) and S(t) into account improves both 16QAM phase and amplitude. Especially for “stained glass” and “butterflies” which have low pixel values that change fast, the amplitude can be improved by the compensation taking both p(k,l) and S(t) into account.

Fig. 4 - Three typical standard images and their ensemble average values.
Fig. 4

Three typical standard images and their ensemble average values.

Fig. 5 - Typical examples of 16QAM waveforms.
Fig. 5

Typical examples of 16QAM waveforms.

Table I Specifications of the transmitter and the receiver.
Table I- Specifications of the transmitter and the receiver.

4.3 16QAM Constellation

Figure 6 shows 16QAM constellations with the standard images, “ship”, “stained glass”, and “butterflies” overlaid, where the solid lines indicate the range of each symbol. Even the compensation taking only p(k,l) into account can improve the constellation more than a single camera alone. However, amplitude variation still remains in the images, “stained glass” and “butterflies” as shown in Fig. 6(b) and (c). On the other hand, the compensation taking both p(k,l) and S(t) into account improves the constellation further because the variation in amplitude is suppressed.

Fig. 6 - 16QAM constellations.
Fig. 6

16QAM constellations.

Fig. 7 - 16QAM EVMs and BERs.
Fig. 7

16QAM EVMs and BERs.

4.4 16QAM EVM and BER

Figure 7(a) shows 16QAM EVMs with “ship”, “stained glass”, and “butterflies” overlaid. The compensation taking both p(k,l) and S(t) into account is effective in reducing the EVM for the images, “stained glass” and “butterflies,” which have low pixel values and rapid changes in pixel values. Furthermore, Fig. 7(b) shows 16QAM BERs. Only the compensation taking both p(k,l) and S(t) into account was able to reduce BERs even for the images with low pixel values and fast changes, “stained glass” and “butterflies.” By using the proposed compensation method, BERs for images, “stained glass” and “butterflies” were reduced to 1.7\times 10^{-3} and 3.6\times 10^{-3}, respectively. The EVMs and BERs results show that the compensation taking both the overlaid image and backlight modulation into account is effective for the image overlay 16QAM.

SECTION 5.

Conclusion

The overlaid image compensation was investigated for 16QAM RS-SCC using dual cameras. The compensation method takes both the overlaid image and the backlight modulation into account, which was compared with that taking only the overlaid image into account and compared with a single camera alone. The waveform, constellation, EVM, and BER were measured for the image overlay 16QAM. The proposed method was able to improve both the phase and amplitude of 16QAM. It was particularly effective for images with low pixel values and fast changes.

The bitrate was increased to 1682 bit/s and BER was reduced to less than 4\times 10^{-3}. The symbol length of 16QAM was 2378 μs. In contrast, the symbol length of I-2PPM and I-4PPM was 1190 and 2380 μs, respectively, achieving the same 840 bit/s. Although the symbol length of 16QAM was longer than that of I-PPM, the bitrate was able to be twice as much as I-PPM using M-ary modulation.

However, to realize 64QAM, further compensation accuracy is required. The gamma coefficients need to be compensated for each color, such as RGB, rather than for grayscale image. Furthermore, each pixel value in the image needs to be adaptively compensated instead of the ensemble average.

ACKNOWLEDGMENTS

This work was supported by JSPS KAKENHI Grant Number JP24K07484.

References

References is not available for this document.