Unpaired Screen-Shot Image Demoiréing With Cyclic Moiré Learning

We propose an end-to-end unpaired learning approach to screen-shot image demoiréing based on cyclic moiré learning. The proposed cyclic moiré learning algorithm consists of the moiréing network and the demoiréing network. The moiréing network generates moiré images to construct a pseudo-paired set of moiré and clean images. Then, the demoiréing network is trained in a supervised manner using the generated pseudo-paired dataset to remove moiré artifacts. In the moiréing network, the moiré generation is separately learned as global pixel intensity degradation and moiré pattern generation for more realistic moiré artifact generation. Furthermore, the moiréing network and the demoiréing network are integrated together to be trained in an end-to-end manner. Experimental results on different datasets demonstrate that the proposed algorithm significantly outperforms state-of-the-art unsupervised demoiréing algorithms as well as image restoration algorithms.


I. INTRODUCTION
Despite recent significant advances in digital imaging technologies, undesired artifacts still appear in captured images and degrade the image quality, depending on the capturing environment. For example, when we take pictures of screens or scenes with high-frequency repetitive patterns, undesired colorful artifacts, called moiré artifacts, may appear in captured images. These moiré artifacts have various and disruptive colorful patterns, such as stripes, curves, and ripples in the captured image, which degrade the image quality. Moiré artifacts can be categorized into two types. The first screen-shot moiré artifacts are caused by frequency aliasing between the camera's color filter array and the screen's subpixel layout. In a screen-shot moiré image, intensity (or brightness) degradation appears alongside moiré patterns. The second type is texture moiré artifacts, which are generated by the interference between the camera's color filter The associate editor coordinating the review of this manuscript and approving it for publication was Qingli Li . array and high-frequency repetitive patterns, such as textures on clothes. The texture moiré artifacts are mixed with the original textures of the scene without intensity degradation. Figure 1 shows examples of screen-shot moiré artifacts (top row) and texture moiré artifacts (bottom row). Because of their different characteristics, different approaches have been developed to remove screen-shot moirés [1], [2] and texture moirés [3]- [7].
Extensive research has been conducted to remove moiré artifacts particularly in screen-captured images. Early works on image demoiréing exploited the prior information of moiré artifacts based on moiré generation models. For example, Pekkucuksen and Altunbasak [10] and Menon and Calvagno [11] used multiscale color gradients of multiple directions during demosaicing. Yang et al. [12] removed moiré artifacts by dichotomizing a moiré image into background and moiré layers based on the assumption that moiré artifacts can be represented as a sparse matrix in the frequency domain. Yang et al. [13] further improved decompositionbased demoiréing by exploiting the low-rank and sparse FIGURE 1. Examples of different moiré images. The images contain (top row) screen-shot moiré artifacts [8] and (bottom row) texture moiré artifacts [9].
constraints on the texture and moiré images, respectively. However, since these model-based algorithms rely on specific priors of the moiré artifacts, they may fail to effectively remove real-world moiré patterns, which are diverse and spread over a wide range of regions in both the spatial and frequency domains.
Recently, learning-based approaches using convolutional neural networks (CNNs) have been developed and have achieved significant performance improvements in image demoiréing. For example, in [2], [14]- [16], multiscale CNNs were employed for image demoiréing. In [17], multiple priors, including edge information and appearance attributes, were exploited to consider various types of moiré patterns. In addition to the spatial features, the frequency domain information has been exploited. For example, Zheng et al. [1] developed learnable multi-scale bandpass filters to deal with the diversity of moiré artifacts in the frequency domain. Further, in [6], [18], [19], different characteristics of moiré artifacts in both spatial and frequency domains were exploited. However, the aforementioned approaches are based on supervised learning, which requires large amounts of clean and moiré image pairs for training. To address this limitation, unsupervised learning-based algorithms that use generative adversarial networks (GANs) have recently been developed [20]- [22].
In this work, we develop an end-to-end screen-shot image demoiréing algorithm that uses an unpaired set of clean and moiré images based on cyclic moiré learning. The proposed algorithm is composed of two networks: the moiréing network, which degrades clean images by adding moiré artifacts, and the demoiréing network, which removes moiré artifacts in moiré images. The moiréing network generates fake moiré images by learning the intensity degradation and the distribution of moiré patterns in the real moiré images, constructing pseudo-pairs of clean and moiré images. The demoiréing network is trained with the generated paired dataset in a supervised manner. In addition, we propose a two-stage training scheme for each network to improve training efficiency.
The main contributions of this paper are summarized as follows: • We develop an end-to-end screen-shot image demoiréing algorithm that uses an unpaired set of clean and moiré images based on cyclic moiré learning. The proposed cyclic moiré learning constructs a pseudo paired dataset from a given unpaired set of clean and moiré images, which is subsequently used to train the demoiréing network in a supervised manner.
• We design two types of GANs for moiré generation and demoiréing, respectively, and propose a two-stage training scheme for each network to improve the training efficiency.
• We develop a two-stage moiré artifact generation model that separates moiré generation into intensity degradation and moiré pattern generation, for realistic moiré generation. To this end, we employ an image histogram and multiscale architecture for intensity degradation and moiré generation, respectively, which makes the generation of moiré images more efficient and reliable.
• We experimentally show that the proposed unpaired learning algorithm outperforms state-of-the-art unsupervised image demoiréing algorithms [21], [22] and image restoration algorithms [14], [23]- [27]. Note that this paper is an expanded version of our conference paper [22], in which preliminary results have been presented in part. In this paper, we improve the performance of both moiréing and demoiréing networks by employing an image histogram and multiscale architecture. To the best of our knowledge, this is the first attempt to employ the image histogram and multiscale generator architecture in image demoiréing. Furthermore, we present new experiments to verify the effectiveness and generalization ability of the proposed algorithm, including comparisons using an additional dataset, comparisons with more algorithms, and more comprehensive ablation studies.
The remainder of this paper is organized as follows: Section II reviews the related work. Section III describes the proposed unpaired demoiréing algorithm. Section IV discusses the experimental results. Finally, Section V concludes the paper.

II. RELATED WORK A. MODEL-BASED IMAGE DEMOIRÉING
Model-based algorithms, which exploit moiré generation models, have been developed based on particular prior information and assumptions on moiré artifacts. For example, Sasada et al. [28] and Sidorov and Kokaram [29] identified moiré patterns based on the assumption that moiré patterns have specific shapes, such as striped, dotted, or monotonous. Pekkucuksen and Altunbasak [10] and Menon and Calvagno [11] used multiscale color gradients of multiple directions during demosaicking. In [12], [13], [30], [31], decomposition-based demoiréing algorithms were developed based on the assumption that a moiré image can be decomposed into a background layer and a moiré layer. Giorgian-Marius et al. [32] removed moiré artifacts in the frequency domain by assuming that the moiré artifacts correspond to noise components in an image. However, the model-based algorithms typically demand high computational complexity. Furthermore, because model-based algorithms rely on certain priors of moiré artifacts, they are less effective at removing real-world moiré with diverse characteristics in shape and frequency.

B. LEARNING-BASED IMAGE DEMOIRÉING
Learning-based image demoiréing algorithms that use CNNs have recently been developed most actively. In [2], [14], [15], multiscale CNNs were developed to remove complex moiré patterns with different scales by learning the mapping between moiré and moiré-free images. Guo et al. [16] added multi-attention submodules to a multiscale network to focus more on text content and suppress moiré patterns in those regions. He et al. [17] developed a demoiréing algorithm consisting of multiple submodules, each of which learns differences between moiré and natural images in shape, color, and frequency, respectively. In addition, attempts have been made to remove moiré artifacts in the frequency domain. For example, in [33], [34], demoiréing was performed in the wavelet transform domain. Zheng et al. [1] addressed the diversity of moiré artifacts by developing learnable bandpass filters. In [6], [18], [19], [35], both the spatial domain and discrete cosine transform domain were used to exploit the complementary characteristics of moiré artifacts. Note that all the aforementioned CNN-based demoiréing algorithms require a large amount of aligned training pairs, and their performances rely heavily on the characteristics of the pairs.
To address the limitations of CNN-based approaches, unsupervised learning-based approaches using GANs have been developed. Liu et al. [20] developed a GAN-based demoiréing algorithm, in which a generator network is trained first using a synthesized paired dataset, and then generator and discriminator networks are jointly trained using a real-world moiré dataset. Yue et al. [21] converted unsupervised demoiréing into an image-to-image translation by developing two complementary discriminators to distinguish moiré patterns and the image features at different scales. While Yue et al. [21] generated demoiréd images using a GAN with cycle consistency, we employ a GAN to generate moiré images given clean images to construct a pseudo-paired dataset from an unpaired dataset, which is subsequently used to learn the demoiréing network in a supervised manner.

C. LEARNING-BASED IMAGE RESTORATION
The image restoration task is aimed at recovering a clean original image from its corrupted observation caused by the poor environmental conditions, physical limitations of the acquisition systems, or postprocessing. In particular, extensive research efforts have been made to remove undesired artifacts in captured images. Recently, learning-based approaches using CNNs have demonstrated excellent performance in various image restoration tasks, such as dehazing [36]- [38], deraining [39], [40], glare removal [41], and reflection removal [42]. However, these supervised learning-based algorithms require a large amount of paired datasets, which is difficult, or even impossible, to construct in real-world image restoration tasks.
To avoid the challenges associated with constructing large paired datasets for real-world image restoration tasks, unsupervised learning approaches are desirable for these tasks. GAN [43] is one of the most successful algorithms for unsupervised learning without a paired dataset. In particular, a variant of GAN, CycleGAN [23], which learns to translate an image from a certain domain to another without paired data, has been widely adopted for image restoration tasks, such as underwater image enhancement [24], denoising [25], dehazing [44], deraining [45], and contrast enhancement [46], [47]. Unsupervised or unpaired image demoiréing is essential due to the difficulty of constructing paired moiré and moiréfree images. Therefore, we develop an end-to-end unpaired learning-based image demoiréing approach based on cyclic moiré learning.

III. PROPOSED ALGORITHM
We develop an end-to-end demoiréing algorithm that uses an unpaired moiré and clean image dataset based on cyclic moiré learning. Let X and Y denote the sets of moiré and clean images, respectively, and X ∈ X and Y ∈ Y moiré and clean images, respectively. Then, given X and Y, the objective of unpaired image demoiréing is to learn a mapping from moiré images to clean images. Figure 2 illustrates the proposed algorithm, which consists of two types of GANs: moiréing network and demoiréing network. The moiréing network in Figure 2(a) learns the mapping G M : Y → X , which degrades clean images by adding moiré artifacts. On the other hand, the demoiréing network in Figure 2(b) learns the mapping G D : X → Y, which removes moiré artifacts in moiré images. In other words, given clean images Y, the moiréing network constructs the pseudo-moiré and clean image pairs {G M (Y), Y}. Then, the demoiréing network learns the mapping G D using the constructed pseudo pairs in a supervised manner. In addition, D M and D D in Figures 2(a) and (b) denote the discriminators, which discriminate between a real moiré image X and a generated fake moiré image G M (Y ) and between a demoiréd image G D (X ) and a clean image Y , respectively. We describe each network subsequently.

A. MOIRÉING NETWORK 1) CYCLIC MOIRÉ LEARNING
Because the proposed algorithm consists of two GANs with reverse mapping, it cannot be guaranteed that the generated fake moiré images are mapped to desired real moiré images. To address this issue and stabilize the training of the moiré network, we employ cyclic color consistency [22], [47], which enforces similarity of the reconstructed image to its origin. Specifically, the generator G M is learned to yield reconstructed moiré images G M (G D (X )) that are similar to the real moiré images X . This cyclic color consistency makes the moiré network generate more realistic moiré artifacts while preserving the original clean images' information.

2) GENERATOR G M
In the screen-shot moiré image generation model [8], [20], pixel intensities are decreased while moiré patterns are added. Specifically, given a clean image Y , a moiré image X is generated as where M denotes an image of moiré patterns, and an intensity degradation parameter 0 < α ≤ 1 controls the decrease in intensity. Therefore, we can separate the moiré generation problem into two subproblems: pixel intensity degradation and moiré pattern generation. Figure 3 shows the architecture of the moiré generator G M in Figure 2. Given a clean image Y , the intensity degradation module estimates the global intensity degradation parameter α, whereas the moiré generation module generates the moiré pattern M .

a: INTENSITY DEGRADATION MODULE
The intensity degradation module first constructs a 256-dimensional histogram vector, of which the kth element denotes the number of pixels with intensity k. The intensity histogram contains the global information of the intensity distribution in the input image. Then, the neural network f takes the histogram vector and estimates the intensity degradation parameter α. The neural network f consists of three fully connected (FC) layers. For the first and second FC layers, a batch normalization layer and the Leaky-ReLU activation are followed. A sigmoid function is used to normalize the output of the last FC layer to constrain 0 < α ≤ 1. Figure 4 illustrates that moiré patterns are diverse and spread over a wide range of regions in both the spatial and VOLUME 10, 2022  frequency domains. Furthermore, moiré patterns have different shapes and scales at different regions, even within an image. Therefore, it is essential to consider contexts in the images of various sizes to generate faithful moiré patterns.

b: MOIRÉ GENERATION MODULE
To this end, we develop the moiré generation module using a coarse-to-fine approach. Specifically, it consists of three branches, each of which is responsible for the coarse, middle, and fine scales, respectively. As shown in Figure 3, an clean input image is first downsampled recursively, twice, by a factor of 2. Then, at the ith scale, we obtain the intensity degraded image αY i . Since α is estimated using the intensity histogram of the input image and is thus scale-invariant, we estimate α only at the coarsest level and use it at all scales to obtain αY i for i ∈ {1, 2, 3}. Each branch contains a CNN g i to generate moiré patterns M i . At the coarsest level, CNN g 1 generates global moiré patterns with larger receptive fields. As the input size increases, the CNN g i tends to generate smaller patterns by reducing the receptive field. The network g i estimates moiré patterns M i from an intensity-degraded image αY i and the generated moiré patterns from the previous scale M i−1 . Because the coarsest-scale branch has no previous moiré pattern map, it uses random noise sampled from a uniform distribution in [0, 1]. c: ARCHITECTURE OF THE GENERATOR g i Figure 5 shows the architecture of the CNN g i in Figure 3, which is designed as a U-Net-like structure [26]. The generator consists of an encoder, a decoder, and skip connections. The encoder progressively downsamples the input features to increase the receptive fields, whereas the decoder upsamples these features to generate a moiré image. The main difference between U-Net [26] and the proposed g i is the use of a global average pooling layer (GAP), which extracts global features [46] of moiré patterns spread over the entire image.
The encoder takes an intensity degraded image αY i and a moiré pattern Up(M i−1 ) as inputs, which constitutes a tensor of size H × W × 6, where H and W are the height and width, respectively. The encoder consists of 11 convolutional blocks, each of which has convolution filters of a size of 3 × 3 with ReLU activation, and GAP layers. The spatial resolution of feature maps is halved after the third, fifth, seventh, ninth, and tenth convolutional blocks. The decoder  has six upsampling blocks, each of which contains a bilinear interpolation layer for upsampling and convolution filters followed by ReLU activation. Finally, skip connections are used to restore feature maps at each scale. Figure 6 shows the architecture of the discriminator D M , which learns to discriminate between a real moiré image X and a generated fake moiré image G M (Y ). As shown in Figures 4(e) and (f), moiré artifacts correspond to high-frequency components. To take advantage of this property and make the discriminator focus more on moiré patterns, we first extract its high-frequency components using a high-pass filter. Then, we use both a moiré image and its high-frequency components as inputs for the discriminator. Next, the discriminator predicts a score, which denotes whether the input is real or generated, through eight convolutional blocks and an FC layer. The convolutional blocks consist of convolution filters followed by Leaky-ReLU activation.

B. DEMOIRÉING NETWORK
Once the moiré network constructs the pseudo-pairs {Y, G M (Y)}, the demoiréing network is trained to remove moiré artifacts in a supervised manner using the learned generator G M , as shown in Figure 2(b). The overall structure of the demoiréing network is similar to that of the moiréing network. The main difference is that the cyclic color consistency is computed between Y and G D (G M (Y )), which we call the content loss.

1) GENERATOR G D
From the moiré generation model in (1), the demoiréing can be modeled as Therefore, similar to the moiréing network in Section III-A, we can separate the demoiréing problem into two subproblems: moiré artifact removal and global intensity restoration. Figure 7 shows the architecture of the generator G D in Figure 2. Given a moiré image αY + M , the moiré removal module removes the moiré pattern M , and the intensity restoration module estimates the global intensity restoration parameter α.

a: MOIRÉ REMOVAL MODULE
The moiré removal module, which is similar to the moiré generation module in the generator G M in Figure 3, consists of three branches for the coarse, middle, and fine scales, respectively, for coarse-to-fine generation. The ith branch includes the CNN h i , which is responsible for moiré artifact removal. We employ the architecture of g i in Figure 5 for h i .

b: INTENSITY RESTORATION MODULE
The intensity restoration module is similarly designed to the intensity degradation module in Figure 3. Specifically, it first constructs a 256-dimensional histogram vector. Then, the neural network i takes the histogram vector and estimates the intensity restoration parameter α. We employ the architecture of f in Figure 3 for i.

2) DISCRIMINATOR D D
The discriminator D D takes a clean image and its high-frequency components as input and then predicts a score to discriminate if the input is a real clean image Y or a demoiréd image G D (X ). We use the architecture of the discriminator D M in Figure 6 for D D .

C. LOSS FUNCTIONS
We train two networks: the moiréing network and the demoiréing network. More specifically, in the moiréing network, the generator G M and discriminator D M are trained, whereas in the demoiréing network, the generator G D and discriminator D D are trained. For stable training, we employ the least squares GANs [48] (LSGANs) and cyclic color consistency [47]. The loss functions to train the two networks are described subsequently.

a: MOIRÉING NETWORK
To train proposed moiréing network, we define the moiréing loss L M by where L G M and L cycle are the generator loss and the cyclic color loss, respectively. The hyperparameter ω M controls the relative impact between the two losses. We define the generator loss L G M and discriminator loss L D M , respectively, of LSGANs as In addition, we define the cyclic color loss L cycle as the 1 -norm between the reconstructed moiré image and the original moiré image, given by

b: DEMOIRÉING NETWORK
Similarly to the moiréing loss in (3), we define the demoiréing loss L D as the sum of the generator loss L G D and two losses, L content and L ASL , for cyclic consistency, given by where the hyperparameters ω D,c and ω D,A control the balance among the three losses. We define the generator and discriminator losses of LSGANs for G D and D D , respectively, as Note that L G D and L D D in (8) and (9) are identical to L G M and L D M in (4) and (5), respectively, except for the inputs. Next, we compute the content loss L content as the 1 -norm between Y and its cyclically generated version, given by Finally, unlike with the moiréing network, we employ an additional cyclic loss to train the demoiréing network, called  [23], CWR [24], UID-Net [25], ERRNet [27], U-Net [26], DMCNN [14], MR-GAN [21], and Park et al . 's algorithm [22] on the LCDMoire, TIP2018, and MRBI datasets. The boldface values denote the highest scores for each metric.
the advanced Sobel loss (ASL) [1], given by where S i (·) denotes the edge map obtained by the ith filter in the Sobel filtering among the horizontal, vertical, and two diagonal filters. Therefore, L ASL in (11) quantifies the cyclic color consistency between the edge maps of the reconstructed and original clean images. Since natural images include meaningful edge information, the ASL encourages the removal of moiré artifacts that have been detected as spurious edges. Note that, since the edge maps of moiré images contain a large amount of noise, corresponding to moiré artifacts, it is difficult to compare the edge maps of real and generated moiré images. Thus, we only use the ASL for training the demoiréing network.

A. DATASETS
We evaluate the performance of the proposed algorithm on the LCDMoire [8], TIP2018 [14], and MRBI [49] datasets. The LCDMoire dataset contains 10,100 synthetic moiré and clean image pairs, which are composed of 10,000 training pairs and 100 validation pairs. To construct the unpaired moiré dataset, we divided 10,000 training pairs evenly into two groups and then picked 5,000 moiré images from the first group and 5,000 clean images from the second group. We use the unpaired moiré dataset to train the proposed networks and the 100 validation pairs for the test. The TIP2018 dataset contains 135,000 captured real moiré and clean image pairs, which are composed of 125,000 training pairs and 10,000 testing pairs. In this work, we randomly chose 100 testing pairs for the test. The MRBI dataset also contains captured real moiré images, composed of 352 pairs for training and validation and 340 pairs for testing. Note that, although we only use a single training dataset, we evaluate the performance of the proposed algorithm on all the three datasets.

B. IMPLEMENTATION DETAILS
We train the proposed networks using the AdamW optimizer [50] with a learning rate of 10 −4 , β 1 = 0.5, and  (3) and ω D,c and ω D,A in (7) are fixed to 50, 50, and 25, respectively. Also, the kernel size for high-pass filtering in the discriminators is 3 × 3. For reproducibility, we provide the source codes and pretrained models on our project website. 1

C. QUANTITATIVE AND QUALITATIVE EVALUATION
We evaluate the performance of the proposed algorithm against those of two existing unpaired image demoiréing algorithms: MR-GAN [21] and Park et al.'s algorithm [22]. To the best of our knowledge, they are the only two existing attempts for unpaired image demoiréing. Since unpaired image demoiréing can be regarded as an image-to-image translation task [21], we compare the proposed algorithm with three image-to-image translation algorithms, Cycle-GAN [23], CWR [24], and UID-Net [25], as a benchmark for unpaired image-to-image translation. Finally, we compare the proposed algorithm with the supervised image demoireing algorithm (DMCNN) [14], image restoration algorithm (U-Net) [26], and reflection removal algorithm (ERRNet) [27]. For unpaired demoiréing, the networks DMCNN, U-Net, and ERRNet are used as generators of GANs. We retrained  [23], (d) CWR [24], (e) UID-Net [25], (f) ERRNet [27], (g) U-Net [26], (h) DMCNN [14], (i) MR-GAN [21], (j) Park et al . 's algorithm [22], and (k) the proposed algorithm. The second, fourth, sixth, and eighth rows show the magnified parts of the red squares in the first, third, fifth, and seventh rows, respectively.
all the conventional algorithms with the parameter settings recommended by the authors using the same dataset in Section IV-A.

1) COMPARISON ON THE LCDMoire DATASET
For quantitative assessment, we employ the PSNR, structural similarity index (SSIM) [51], and learned perceptual image patch similarity (LPIPS) [52] metrics. Table 1 shows the average PSNR, SSIM, and LPIPS scores over all images on the validation set in LCDMoire. Table 1 also compares the numbers of network parameters and the average execution times obtained by applying different demoiréing algorithms to the test images in LCDMoire. The numbers of network parameters for all the algorithms listed in Table 1 correspond to those of the entire network architectures, including generators and discriminators.
The image-to-image translation algorithms CycleGAN [23], CWR [24], and UID-Net [25] show the lowest scores because they do not consider the diverse global and local structures of moiré artifacts despite using cyclic consistency. The reflection removal algorithm ERRNet [27] yields higher scores than some image-to-image translation algorithms but lower than those of most demoiréing algorithms because it does not restore degraded pixel intensities. DMCNN and U-Net yield higher PSNR scores than CycleGAN, CWR, and UID-Net. MR-GAN [21] and Park et al.'s algorithm [22] provide higher scores than the other conventional algorithms with large margins because they exploit the properties of moiré artifacts. The proposed algorithm outperforms all conventional algorithms, yielding a higher PSNR score than MR-GAN and Park et al.'s algorithm with margins of 2.79 and 1.77 dB, respectively. The proposed algorithm also provides the best demoiréing performance in terms of SSIM. This indicates that the proposed cyclic moiré learning with the two-stage training scheme is effective for unpaired image demoiréing. . Qualitative comparison of demoiréing results for the testing set in the TIP2018 dataset [14]: (a) moiré image, (b) ground-truth, and outputs of (c) CycleGAN [23], (d) CWR [24], (e) UID-Net [25], (f) ERRNet [27], (g) U-Net [26], (h) DMCNN [14], (i) MR-GAN [21], (j) Park et al . 's algorithm [22], and (k) the proposed algorithm.  [49]: (a) moiré image, (b) ground-truth, and outputs of (c) CycleGAN [23], (d) CWR [24], (e) UID-Net [25], (f) ERRNet [27], (g) U-Net [26], (h) DMCNN [14], (i) MR-GAN [21], (j) Park et al . 's algorithm [22], and (k) the proposed algorithm.   Figures 8(c), (d), and (e), respectively, fail to remove complex and globally spread moiré patterns, as shown in the first and seventh rows. In addition, in the third and fifth rows, CycleGAN loses the original colors, whereas UID-Net generates severe visible artifacts in the third and seventh rows. The reflection removal algorithm ERRNet in Figure 8(f) removes moiré patterns more effectively than the image-to-image translation algorithms, but it does not restore degraded pixel intensities. U-Net and DMCNN in Figures 8(g) and (h), respectively, also fail to remove globally spread moiré patterns, since they consider only pixel-level or local information. Further, they fail to restore color intensity faithfully, as shown in the first and seventh rows. MR-GAN in Figure 8(i) effectively removes small local moiré patterns in the third and fifth rows, but, similarly to the image-to-image translation algorithms, it preserves global moiré patterns and loses the original colors, as shown in the first and seventh rows. Park et al.'s algorithm provides better results in terms of both pixel intensity restoration and moiré pattern removal. Nevertheless, in the magnified parts in the second and eighth rows in Figure 8(j), parts of the large moiré patterns appear because Park et al.'s algorithm employs single-scale generators and thus has difficulties in removing both local and global moiré patterns. On the contrary, the proposed algorithm in Figure 8(k) outperforms all conventional algorithms, effectively removing moiré patterns across different scales, while restoring the color information faithfully.

2) COMPARISON ON THE TIP2018 DATASET
Next, we evaluate the demoiréing performance on the TIP2018 dataset [14], which was not used for training. Table 1 also shows the average PSNR, SSIM, and LPIPS scores over test images in TIP2018. The scores of the proposed algorithm remain the highest, which confirms its superior generalization ability. More specifically, the proposed algorithm outperforms Park et al.'s algorithm, which provides the second best scores, by larger margins than that on the LCDMoire dataset. This indicates that the proposed algorithm provides better restored textures than conventional algorithms. Figure 9 shows the demoiréing results for the test images from the TIP2018 dataset. The image-to-image translation algorithms CycleGAN, CWR, and UID-Net in Figures 9(c), (d), and (e), respectively, fail to restore the intensity degradation and preserve moiré patterns in the input images. ERRNet in Figure 9(f) fails to remove moiré   Figure 9(i) restores the pixel intensity but fails to remove global moiré patterns in the first and fourth rows. In Figure 9(j), Park et al.'s algorithm also preserves widely distributed moiré patterns in the fourth row. In contrast, the proposed algorithm in Figure 9(k) provides demoiréing results without visible artifacts. The results in Figure 9 for a dataset that was not used for training and has different characteristics from the training dataset confirm that the proposed algorithm has superior generalization ability to conventional algorithms.

3) COMPARISON ON THE MRBI DATASET
Finally, we evaluate the demoiréing performance on the MRBI dataset [49], which was obtained by capturing the screens using cameras. Table 1 shows that the proposed algorithm also outperforms all conventional algorithms with large margins on the MRBI dataset.   Figure 10(j) effectively removes moiré patterns but preserves the degraded pixel intensities in the third and fifth rows. In contrast, the proposed algorithm in Figure 10(k) outperforms all conventional algorithms, effectively removing moiré patterns and restoring degraded pixel intensities.

D. ABLATION STUDIES
We conduct several ablation studies to analyze the contributions of the key components in the proposed algorithm: cyclic moiré learning, cyclic color consistency, and moiréing network architecture. All experiments are performed for the LCDMoire dataset [14].

1) CYCLIC MOIRÉ LEARNING
In the proposed cyclic moiré learning algorithm, the moiréing network generates moiré images to construct a paired set of moiré and clean images. Then, the demoiréing network is trained to remove moiré artifacts. To demonstrate the effectiveness of cyclic moiré learning, we compare it against the vanilla GAN. Specifically, the generator of the vanilla GAN is trained to generate clean images from moiré images by learning the distribution of the clean images using the cyclic consistency. Table 2 compares the quantitative results. The proposed cyclic moiré learning provides significantly higher scores than the vanilla GAN. This indicates that the proposed cyclic moiré learning algorithm is effective at learning demoiréing using an unpaired dataset.

2) CYCLIC COLOR CONSISTENCY
We analyze the effectiveness of the cyclic color consistency loss L cycle to train the moiréing network in Section III-C. To this end, we train the proposed networks using loss functions with and without L cycle . Table 3 compares the results quantitatively. Training without L cycle yields lower scores because training GANs becomes unstable, thus generating undesired moiré images. In contrast, training with L cycle provides higher scores with large margins, which indicates that the cyclic color consistency allows the moiréing network to generate more realistic moiré artifacts while preserving the original clean images' information.

3) MOIRÉING NETWORK
To analyze the effectiveness of the intensity degradation module in the moiréing network, we train the moiréing network with and without the intensity degradation module. Table 4 compares the average PSNR and SSIM scores of these two settings. In Table 4, g denotes the case where only the moiré generation module is used, whereas f + g indicates the case where both the intensity degradation module and the moiré generation module are used. By separating the moiréing process into two tasks, the proposed algorithm can perform more effectively by generating complex and diverse moiré patterns. Figure 11 shows the input clean images, the outputs of the intensity degradation module αY 3 , the generated moiré artifacts M 3 , and the outputs of the moiré generation module αY 3 + M 3 in Figure 3. As shown in Figures 11(b) and (c), the intensity degradation module f degrades the global image intensity while the moiré generator g 3 generates the moiré patterns. By separating the moiré generation into the two procedures, the proposed moiréing network can generate realistic moiré images in Figure 11(d). Figure 12 shows the generated moiré patterns and the corresponding moiré images over epochs during the training of the moiréing network. As shown in Figure 12, the generator gradually generates more realistic moiré patterns and decreases pixel intensity as the number of epochs increases.

4) COARSE-TO-FINE MOIRÉ GENERATION
We analyze the effectiveness of the coarse-to-fine approach in the generator G M in the moiréing network. Figure 13 shows the generated moiré images αY i + M i at each scale i ∈ {1, 2, 3}. As shown in Figures 13(b)-(d), the generator gradually generates local moiré patterns from globally spread patterns by learning moiré patterns at different scales.

V. CONCLUSION
We proposed an end-to-end image demoiréing algorithm that uses an unpaired training dataset based on cyclic moiré learning. The proposed algorithm is composed of two networks: the moiréing network and demoiréing network. The moiréing network constructs a pseudo-paired dataset from an unpaired dataset, whereas the demoiréing network is trained using the generated pseudo-paired dataset to remove moiré artifacts in a supervised manner. The moiré generation in the moiréing network was separately learned as global pixel intensity degradation and moiré pattern generation for more realistic moiré artifacts generation. Further, we reinforced the cyclic color consistency so that the moiréing network could generate moiré-looking artifacts while preserving information of the original clean image. Experimental results on different datasets demonstrated that the proposed algorithm outperforms state-of-the-art unsupervised demoiréing algorithms as well as image restoration algorithms.