Effective Chroma Subsampling and Luma Modification for RGB Full-Color Images Using the Multiple Linear Regression Technique

Differing from the traditional chroma subsampling on the YUV image converted from a RGB full-color image, in this paper, we propose a novel and effective chroma subsampling and luma modification (CSLM) method. For each <inline-formula> <tex-math notation="LaTeX">$2\times 2$ </tex-math></inline-formula> YUV block, first, a newly reconstructed <inline-formula> <tex-math notation="LaTeX">$2\times 2$ </tex-math></inline-formula> RGB full-color block-distortion model is proposed, and then we propose a multiple linear regression approach to tackle our CSLM method such that the reconstructed <inline-formula> <tex-math notation="LaTeX">$2\times 2$ </tex-math></inline-formula> RGB full-color block-distortion can be minimized, achieving significant quality improvement of the reconstructed RGB full-color image. Based on the Kodak and IMAX datasets, the comprehensive experimental results demonstrated that on the versatile video coding (VVC) platform VTM-8.0, our method achieves substantial quality and quality-bitrate tradeoff improvement of the reconstructed RGB full-color images relative to six traditional methods and the three state-of-the-art methods.


I. INTRODUCTION
As a target image for human visual perception, the RGB full-color image I RGB is the most important medium. In the traditional coding system, as shown in Fig. 1, prior to compression, I RGB is first converted to the YUV image I YUV by the following RGB-to-YUV conversion formula: The associate editor coordinating the review of this manuscript and approving it for publication was Zhaoqing Pan .
In this paper, our discussion focuses on 4:2:0, although it is applicable to 4:2:2. 4:2:0 has been used in Blu-ray discs (BDs) and digital versatile discs (DVDs) for recording movies, sports, and so on. After decompressing the encoded subsampled YUV image by the decoder, each decoded subsampled YUV image is upsampled at the client side, as shown in the lower part of Fig. 1. Furthermore, each upsampled YUV pixel is converted into a RGB full-color pixel by the following YUV-to-RGB conversion formula: As a result, the reconstructed RGB full-color image is produced.
Note that all discussion in this paper can be applied to other color spaces, such as the YC b C r color space, because the RGB-to-YC b C r and YC b C r -to-RGB color conversions are also linear as the color conversion between the RGB space and the YUV space appeared in Eqs. (1)- (2). The digital YUV data in Eqs. (1)- (2) are originally converted from analog signals, while the YC b C r data are originally digital. The YUV VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ color space is often used in analogue color TV broadcasting; the YC b C r color space is often used in digital videos and BT.601.

A. RELATED WORK
In this subsection, we introduce six traditional chroma subsampling methods and three state-of-the-art chroma subsampling methods [1], [6], [12]. All of them will be included in the comparative methods. The common weakness of the above traditional chroma subsampling methods is the failure to take the reconstructed RGB full-color block distortion into account, limiting the quality improvement of the reconstructed RGB full-color image.
2) Three state-of-the-art chroma subsampling methods: To overcome the weakness existing in the traditional methods, three state-of-the-art methods were developed.
Based on the new edge-directed interpolation (NEDI) [5], Zhang et al. [12] proposed an interpolation-dependent image downsampling (IDID) method for chroma subsampling, and their combination IDID-NEDI, in which IDID is used at the server side and NEDI is used at the client side, can tackle the chroma downsampling well.
Lin et al. [6] proposed a modified chroma 4:2:0(A) subsampling method, namely modified 4:2:0(A), by considering the truncation and carry operations influence; among the four considered variants, they select the best subsampled (U , V )-pair. At the client side, they improved the previous chroma upsampling process [10] by considering the distance between each estimated upsampled chroma value and its three neighboring (TN) pixels; their combination, modified 4:2:0(A)-TN, achieves good quality of the reconstructed RGB full-color image.
After performing a chroma subsampling, e.g. 4:2:0(A), under the COPY-based chroma upsampling process in which each estimated (U , V )-pair of each 2 × 2 UV block B UV just copies the subsampled (U , V )-pair of B UV , Chung et al. [1] proposed a pixel-based approach to adjust each luma pixel in the 2 × 2 luma block to improve the quality of the reconstructed RGB full-color image. The first weakness in [1] is that the COPY-based chroma upsampling process used at the server side is too simple to meet the chroma upsampling capability at the client side, such as the BILI upsampling method, thus limiting the quality improvement. The second weakness in [1] is the failure to consider chroma subsampling and luma adjustment simultaneously to achieve better quality improvement.

B. CONTRIBUTIONS
In this paper, we propose a novel chroma subsampling and luma modification (CSLM) method for I RGB . The three contributions of this paper are clarified in the following aspects.
In the first contribution of our CSLM method, for each 2×2 YUV block, a novel reconstructed 2×2 RGB full-color blockdistortion model is proposed. Considering the neighboring subsampled (U , V )-pairs of the current 2 × 2 UV block, we deploy the BILI interpolation into the block-distortion model to better meet the chroma upsampling capability at the client side.
In the second contribution of CSLM, the reconstructed 2 × 2 RGB full-color block-distortion model can be transformed to an overdetermined system by a multiple linear regression approach. Furthermore, the matrix pseudoinverse technique is applied to determine the subsampled (U , V )-pair and the four luma values such that the reconstructed blockdistortion can be minimized.
In the third contribution, based on the Kodak and IMAX datasets, the comprehensive experimental results demonstrated that on the versatile video coding (VVC) platform VTM-8.0 [8], our CSLM method achieves significant quality and quality-bitrate tradeoff improvement of the reconstructed RGB full-color images relative to the six traditional methods and three state-of-the-art methods in [1], [6], [12]. Here, the quality metrics used are CPSNR (color peak-signal-noiseratio), SSIM (structure similarity index) [9], and FSIMc (feature similarity index) [11]; the quality-bitrate tradeoff metric is illustrated by the RD-curves (rate-distortion curves) for different quantization parameter (QP) values. In addition, based on the video sequence ''Boat'' which can be accessed from [6], the quality-bitrate tradeoff merit of our method is reported.
The rest of this paper is organized as follows. In Section II, the reconstructed 2×2 RGB full-color block-distortion model is first presented, and then the corresponding overdetermined system is derived. In Section III, the matrix pseudoinverse technique is applied to determine the subsampled (U , V )-pair and the four modified luma values. In Section IV, the comprehensive experimental results are demonstrated to justify the significant quality and quality-bitrate tradeoff merits of our CSLM method. In Section V, some concluding remarks are addressed.

II. THE RECONSTRUCTED BLOCK-DISTORTION MODEL AND THE DERIVED OVERDETERMINED SYSTEM
In this section, we first present a mathematical model to estimate the reconstructed 2×2 RGB full-color block-distortion, and then we derive the overdetermined system by deploying the two chroma subsampled parameters and four luma modification parameters into the block-distortion model.
Before presenting the proposed reconstructed 2 × 2 RGB full-color block-distortion model at the server side, we first describe how to estimate the four (U , V )-pairs of the current 2 × 2 UV block B UV by referring to the eight neighboring subsampled (U , V )-pairs of B UV .
Because our CSLM method combines chroma subsampling and luma modification together, and is performed on I YUV in a row-major order, for the current 2 × 2 UV block B UV , the eight neighboring subsampled (U , V )-pairs of B UV consist of four already known subsampled (U , V )-pairs obtained by our CSLM method and four future subsampled (U , V )-pairs which can be obtained by performing any traditional chroma subsampling scheme, e.g. 4:2:0(A), on the four future reference 2 × 2 UV blocks. As depicted in Fig. 3, the eight reference subsampled (U , V )-pairs are denoted by For estimating (U 3 , V 3 ), the four reference subsampled , and (U s , V s ) are thus located at (0, 1), (1, 0), and (1, 1), respectively. According to the bilinear interpolation, 16 In the same estimation way as for (U 3 , V 3 ), the estimation of (U 1 , V 1 ), (U 2 , V 2 ), and (U 4 , V 4 ) can be followed. In general, we have Replacing the three parameters Y i , U i , and V i at the right ride of Eq. (2) with Y i , U i , and V i , 1≤ i ≤4, respectively, it yields the following three equalities: For 1≤ i ≤4, it yields the following twelve equalities: From Eq. (7), the reconstructed 2 × 2 RGB fullcolor block-distortion model is naturally denoted by . Because by Eq. (3)-(5), we know that U 1 , U 2 , U 3 , and U 4 are the functions with the parameter U s ; V 1 , V 2 , V 3 , and V 4 are the functions with the parameter V s , so the reconstructed 2 × 2 RGB full-color block-distortion is defined by Ideally, the solution of X (= (Y 1 , Y 2 , Y 3 , Y 4 , U s , V s )) aims to zeroize the reconstructed 2 × 2 RGB full-color blockdistortion in Eq. (8). At the right side of Eq. (7), for 1≤ i ≤4, we replace U i and V i with ( 9 16 U s + δ(U i )) and ( 9 16 V s + δ(V i )) (see Eq. (4)), respectively. Therefore, ideally, the solution of X aims to satisfy the following overdetermined system: In the above overdetermined system, there are six parameters, namely Y 1 , Y 2 , Y 3 , Y 4 , U s , and V s , to be solved. Because it is intractable to solve X such that all equalities in Eq. (9) are totally satisfied, in the next subsection, a matrix pseudoinverse technique is proposed to solve X such that the sum of the square errors between the left side and right side of Eq. (9) could be minimized, obtaining the best solution of X.

III. DETERMINING THE SUBSAMPLED CHROMA PAIR AND MODIFIED LUMA VALUES
In this section, we first transform the overdetermined system in Eq. (9) to a matrix form, and then we show that it can be solved by the matrix pseudoinverse technique, determining the solution of X for each 2 × 2 YUV block B YUV . Finally, the whole procedure to realize our CSLM method is provided.
which is often called the response vector [2]. Therefore, Eq. (10) is simplified to Eq. (11), where the matrix T is often called the design matrix [2].
Based on the geometry relation, (b − TX ) is perpendicular to the range of T , namely R(T ), which is spanned by the column vectors of T . Therefore, it yields T t (b − TX ) = 0, and then the normal equation T t b = T t TX holds. Because the design matrix T is full rank and the rank is 6, the pseudoinverse (T t T ) −1 T t exists [2]. Therefore, with our CSLM method, the solution of X for Eq. (11) can be obtained by where the design matrix T and the response vector b have been defined in Eqs. (10)- (11). In fact, the pseudoinverse (T t T ) −1 T t can be computed in advance as a fixed 6 × 12 matrix which is shown in Eq. (13), as shown at the bottom of the page, achieving the executiontime reduction effect.

B. THE WHOLE PROCEDURE TO REALIZE OUR CSLM METHOD
Consequently, using our CSLM method, for the current 2 × 2 YUV block B YUV , by Eqs. (11)- (12), the four modified luma values, Y 1 , Y 2 , Y 3 , and Y 4 , and the two subsampled chroma values, V s and U s , can be determined quickly such that the reconstructed 2 × 2 RGB full-color block-distortion could be minimized in the least square errors sense. The whole procedure to realize our CSLM method is listed below.

IV. EXPERIMENTAL RESULTS
Based on the Kodak dataset with 24 images [4] and the IMAX dataset with 18 images [3], all the considered experiments are carried out on the VTM-8.0 platform. To compare the quality performance among the considered chroma subsampling methods, the three quality metrics used are CPSNR, SSIM, and FSIMc. Besides the three quality merits of our CSLM method, the quality-bitrate tradeoff merit of our method is also demonstrated for different QP values. In addition, the luma mean-preserving effect of CSLM is reported. The execution time comparison of the considered methods Step 1: By Eq. (2.1), estimate the four reconstructed (U , V )pairs of B UV .
Step 4: By Eq. (3.4), calculate X = Sb to determine the four modified luma values and the subsampled (U , V )-pair, is also made. In addition, based on the video sequence ''Boat'', the quality-bitrate tradeoff merit of our method is also reported.
All the considered methods are implemented on a computer with an Intel Core i7-4790 CPU 3.6 GHz and 24 GB RAM. The operating system is the Microsoft Windows 10 64-bit operating system. The program development environment is Visual C++ 2017. SSIM [9] is used to measure the product of the luminance, contrast, and structure similarity preserving effect between the original image and the reconstructed image. For I RGB , the SSIM value is measured by the mean of the three SSIM values for the R, G, and B color planes. To measure the FSIMc metric value [11], we first utilize the contrast invariant feature ''phase congruency (PC)'' and the minor feature ''gradient magnitude'' to obtain the local quality map. Further, we utilize PC as a weighting function to calculate the quality score as the FSIMc metric value. Note that the available code for FSIMc can be accessed from [11]. To justify the CPSNR, SSIM, and FSIMc merits of our CSLM method, we set QP to zero and the related results are computed by passing the compression and decompression process.  For the reconstructed RGB full-color images, Table 1 tabulates the quality comparison in terms of CPSNR, SSIM, and FSIMc. Here, the three chroma upsampling processes at the client side, namely COPY, BILI, and BICU, are included. From Table 1, we observe that our CSLM method under the BILI chroma upsampling process has the highest CPSNR, SSIM, and FSIMc in boldface when compared with the eighteen combinations for the six considered traditional chroma subsampling methods and the three considered chroma upsampling processes.

B. QUALITY-BITRATE TRADEOFF MERIT, LUMA MEAN PRESERVATION EFFECT, AND EXECUTION TIME COMPARISON
In this subsection, we first present the quality-bitrate tradeoff merit of our CSLM method, and then present the luma mean preservation effect. Finally, the execution time comparison is reported.

1) QUALITY-BITRATE TRADEOFF MERIT
When setting QP = 0, 4,8,12,16,20,24,28,32,36,40,44,48, and 51, the quality-bitrate tradeoff of each considered VOLUME 8, 2020 method is depicted by the RD curve for the reconstructed RGB full-color images. The bitrate of one compressed dataset is defined by where B denotes the total number of bits required in compressing N test images in that dataset. On VVC platform, the RD curves corresponding to the Kodak dataset and the IMAX dataset are depicted in Fig. 4(a) and Fig. 4(b), respectively, in which the X-axis denotes the average bitrate required and the Y-axis denotes the average CPSNR value of the reconstructed RGB full-color images. Fig. 4 indicates that under the same bitrate, our CSLM method has the highest CPSNR among the nine considered methods. Based on the testing video sequence ''Boat'', Fig. 5 indicates that under the same bitrate, our CSLM method still has the highest CPSNR.

2) LUMA MEAN PRESERVATION EFFECT
The luma mean-loss of our CSLM method is measured by where N denotes the number of test images in the dataset; I Y k andĪ rec,Y k denote the luma mean values of original kth luma image and reconstructed kth analogue, respectively.
Although most of the comparative methods do not consider modifying the luma values in chroma subsampling, their luma mean-loss values are the same, empirically 0.0028 dB, due to the floating point-to-integer conversion error before compression; it indicates a nearly perfect luma mean-preservation effect. For the Kodak dataset, the average luma mean-loss value by our CSLM is 0.0157 dB and for the IMAX dataset, the average luma mean-loss by our CSLM is 0.0151 dB. On average, the luma mean-loss value is 0.0154 dB, indicating a good luma mean-preservation effect of our CSLM method. Table 1 tabulates the execution time (in seconds) comparison among the six traditional chroma subsampling methods and our CSLM method. For simplicity, the execution time of each traditional chroma subsampling method is listed once in Table 1, and from Table 1, although our method takes more time than the traditional methods, our method has clearly better quality. In Table 2, we observe that besides the quality merit, our CSLM method is also much faster than the two state-of-the-art methods, IDID [12], and 4:2:0(A)-LM [1]; our CSLM method has worse execution time performance but has better quality performance relative to modified 4:2:0(A)-TN [6].

V. CONCLUSION
We have presented the proposed CSLM chroma subsampling method for RGB full-color images. First, a newly reconstructed 2 × 2 RGB full-color block-distortion model is presented. Then, an overdetermined system is derived to deploy the two chroma subsampled parameters and four luma modification parameters into the distortion model. Furthermore, we show that the derived overdetermined system can be solved by the matrix pseudoinverse technique, determining the solution of the required chroma subsampled pair and four modified luma values for each 2 × 2 YUV block. Finally, a whole procedure is provided to realize our CSLM method. Based on the Kodak and IMAX datasets, the comprehensive experimental results have justified the quality and qualitybitrate tradeoff merits of our CSLM method relative to six traditional chroma subsampling methods and three state-ofthe-art methods. In addition, based on the video sequence ''Boat'', the quality-bitrate tradeoff merit of our method has been justified.
How to extend the delivered process to estimate the four chroma pairs of each 2 × 2 chroma block, as described in Subsection II.A, using the other nonlinear upsampling processes, such as the bicubic interpolation-based estimation process, is our first future work. In our second future work, we hope to combine CSLM and the discrete cosine transform based (DCT-based) downsampling approach, and then compare it with the current work [13] in which the downsampling process is only done in the DCT domain, while it does nothing on the chroma subsampling and luma modification prior to the compression. HONG-BIN YANG is currently pursuing the bachelor's degree in computer science and information engineering with the National Taiwan University of Science and Technology, Taiwan. He has participated in some programming contests such as ACM ICPC. Under the supervision of Prof. K. L. Chung, he is also working on a project in image processing. VOLUME 8, 2020