Multiresolution Analysis Based on Dual-Scale Regression for Pansharpening

Pansharpening technique is used to merge the original multispectral image with a high spatial resolution panchromatic image. Due to its robustness, the multiresolution analysis (MRA) is an important part of pansharpening. The scale regression model is effective for improving MRA. However, the existing MRA based on scale regression results into single-scale regression information, thus affecting the final pansharpening result. In order to address this problem, in this work, we propose a dual-scale regression-based MRA for pansharpening. First, we establish a scale regression-based model. Then, this model is improved using a high-pass modulation (HPM) injection scheme. Finally, the dual-scale information is added to the scale regression to construct the dual-scale regression for obtaining the final pansharpening result. We perform experiments by using five datasets. The results show that the proposed method obtains a better pansharpening result as compared to various state-of-the-art MRA methods. In addition, the quantitative and qualitative analysis of the results shows that the proposed method achieves appropriate spatial and spectral resolution fusion. Therefore, it has a great potential in pansharpening technique.


INTRODUCTION
hysical limitations and processing capabilities of satellite remote sensing equipment hinder a single sensor from collecting the remote sensing images with high spatial and spectral resolutions simultaneously [1]. The MS images have rich spectral information, but a continuous improvement in the spectral resolution of MS images affects their spatial resolution. Remote sensing image processing has developed many topics, such as super-resolution [2][3][4], feature extraction [5], cloud removal [6], and classification [7]. In addition, the pansharpening technique [8][9][10] has been proposed to improve the spatial resolution of an MS image using a high spatial resolution PAN image. Pansharpening has important applications in environmental monitoring, land and resource use, precision agriculture, urban planning, military reconnaissance and other key areas.
Currently, the common pansharpening methods can be categorized into three types [11,12], including component CS, MRA, and deep learning [13]. The CS methods include intensity-hue-saturation [14], principal component analysis [15], Gram-Schmidt [16], and adaptive GS [17]. Although the CS methods usually have a simple physical meaning and high computational efficiency, the pansharpening results differ from the ideal output. Recently, the deep learning-based techniques have been applied to pansharpening, including pansharpening neural network [18][19][20] and its complex variations in terms of network depth [21][22][23], topology [24,25], and fine-tuning [26][27]. Although the deep learning-based methods achieve better performance as compared to the traditional machine learning methods, such as CS and MRA-based methods, the deep learning usually requires a large amount of labelled training data to achieve the desired performance. In addition to the above three types, geostatistics-based technique is also an effective solution to the pansharpening problem, such as areato-point regression kriging [28] and the information loss-guided image fusion [29].
The MRA-based methods inject spatial information in the MS images. Since the MRA methods do not require the training data and are more effective in retaining the spectral characteristics of the original MS image than the CS, they have been relatively successful and popular for implementing pansharpening techniques [2]. The MRA-based methods include the additive wavelet luminance proportional (AWLP) [30], smoothing filter based on intensity modulation (SFIM) [31], and morphological filters (MF) [32]. In addition, the generalized Laplacian pyramid based on Gaussian filters matches the modulation transfer function of an MS sensor (MTF-GLP) [33]. It is notable that the MTF-GLP has been successfully applied to the MRA. The MTF-GLP with a context-based decision (MTF-GLP-CBD) [34], contextadaptive MTF-GLP-CBD (C-MTF-GLP-CBD) [35]， MTF-GLP with HPM (MTF-GLP-HPM) [36,37], MTF-GLP-HPM based on haze-corrected version (MTF-GLP-HPM-H) [38], and MTF-GLP-HPM based on post-processing (MTF-GLP-HPM-PP) [39] belong to MAR based on the MTF-GLP technique. The scale regression model is effective for improving the MRA and obtaining better results. The scale regression model [40] is a mathematical model that quantitatively describes the statistical relationships. This indicates the influence and significant relationship between the independent and dependent variables. Some of the MRA methods are based on scale regression methods, including the MTF-GLP based on full-scale regression (MTF-GLP-REG-FS) [41] and MTF-GLP-HPM based on multivariate linear regression (MTF-GLP-HPM-R) [42].
However, the existing MRA methods based on scale regression suffer from insufficient scale regression information, which affects the pansharpening result. In order to overcome this problem, in this work, we propose an MRA method based on dual-scale regression for pansharpening method, namely MTF-GLP-HPM-DS. The proposed method improves the accuracy of pansharpening and promotes wider application of pansharpening. We aim to solve the issue of scale regression information based on MRA pansharpening. In the proposed method, first, a scale regression based MRA model is designed. Then, this model is improved by using the HPM injection scheme. Finally, the dual-scale information, including the finescale and coarse-scale information, are added to the scale regression to obtain the final pansharpening result. The experimental results show the superiority of the proposed MTF-GLP-HPM-DS over various state-of-the-art MRA methods.
The contributions of this paper are summarized below.
1) The dual-scale regression model uses abundant scale regression information, which improves the pansharpening result.
2) In the dual-scale regression model, the fine-scale and coarse-scale information are linked by a parameter that can be adjusted to make the proposed method adaptive for different scenarios, unlike the existing MRA methods.
3) We show that the dual-scale regression model is feasible in terms of mathematical analysis and experiments.
The remaining of this paper is organized as follows. The proposed method is described in Section II. The experimental results and comparisons of the proposed method with state-of-the-art methods are presented in Section III. The discussion of the results is presented in Section IV. Finally, the conclusions are drawn in Section V.

A. Scale Regression-based MRA Model
Based on [4], the MRA for the bth spectral band is equivalent to expressions presented in (1)-(3).

( )
where, h denotes the MTF filter, g b denotes the injection coefficients for the bth spectral band of an MS image, and is defined as follows: where, cov(A, B) denotes the covariance of images A and B, and var( ) denotes the sample variance of image A. Based on [4], another pansharpening type CS is equivalent to the expressions presented in (4)-(6).

( )
The specific iterative operations are presented in Algorithm 1. First, the initial injection coefficients g b 0 are obtained for HR,0 LR bb MM = . Then, multiple iterations are performed to obtain the injection coefficients g b . Finally, the iterative process is stopped when the convergence is achieved.
-Using g b to fuse MS and PAN ( )

B. HPM Injection Scheme
The HPM injection scheme improves the MRA [34,35], so it is employed in this work to improve the performance of an MRA model based on scale regression. According to the HPM injection scheme, (1) can be rewritten as: Now, we explain the process of adding the scale regression to the HPM injection scheme. A digital MS or PAN image N s XR acquired by a sensor s is defined as a convolution of the sensor spatial response and the total energy collected by the sensor in its spectral band [43,44] as: where, the superscript XR either denotes HR or LR , s represents an additive constant, and k s represents a multiplicative constant. x and y denote the pixel coordinates.
The two constants are introduced to obtain the complete digital range of the A/D converter; however, s is usually negligible. Further, S s denotes the spatial response of sensor s, and W s represents the integral over the frequency of the atsensor radiance E s (α, y, λ) weighted by the relative spectral response R s ( ) [45], and is obtained as: Based on (8) and (9), LR b M , P LR , and P HR are now expressed as follows: Considering the HPM injection scheme, (11) -(13) are substituted into (8), and M b HR is obtained as follows: ( ) (15) This can be converted to the target expression, i.e., (14), by appropriately setting the multiplication coefficients of are different from (15). Therefore, these variables are expressed using an additional tilde to differentiate between the actual and estimated variables.
Therefore, the modified equation is given as: Now, the desired result is obtained by deriving the following equalities. Since the spatial response of the HR-MS sensor should be the same as that of the existing PAN camera, the first equality is expressed as: Furthermore, since the LR-PAN image P LR is constructed by the MTF-GLP, which utilizes a filter to match the spatial response of the LR-MS sensor, the second equality is defined as follows: Based on [42], the third equality is defined as: Thus, considering (11) and (12), we obtain (20).
Based on (20) and (21), a linear affine function is proposed to solve M b XR , and is mathematically expressed as: Thus, the problem is transformed into the problem of finding coefficients m and n to obtain M b XR . In this work, the spectral matching based on scale regression between the HR-PAN image and HR-MS image is used to compute the coefficients m and n [42] where, E(X) represents the mean of image X. Therefore, we rewrite (16) as follows: According to the definition of the injection coefficients g b based on the scale regression in (7), (25) can be written as follows: As presented in (26), the scale regression is successfully added to HPM injection scheme.

C. Dual-Scale Regression Model
According to (7), the injection coefficients g b based on the scale regression only consider the covariance of P HR at a fine scale and M b HR . Therefore, the scale regression information in g b is single, which affects the performance of the proposed MRA based on the scale regression model. In order to enrich the scale information for scale regression and improve the final pansharpening results, the dual-scale information is introduced in the scale regression to construct the dual-scale regression. In other words, the proposed dual-scale regression not only considers the fine-scale information of the covariance between P HR and M b HR , but also the coarse-scale information of the covariance between P LR and M b HR . Thus, we obtain: where, μ is adjusted to make the proposed method more flexible in different scenarios. In this section, the final injection coefficients are introduced to realize the dual-scale regression iteratively. In addition, a closed-form solution of the iterative process is adopted, so that the iterative method achieves a considerable performance. Now, we show that the dual-scale regression model is iterative and convergent. Appendix A demonstrates that the progress of the mathematical induction in detail. First, the initial injection coefficients g b 0 are obtained for HR,0 LR bb MM = and is defined as: Thus, the expression of g b n−1 is obtained as: Finally, in order to show that the dual-scale regression model is iterative and convergent, mathematical induction is used to obtain g b n derived from g b n−1 in (30) as: Similarly, the iterative process used in the regression requires a fixed point when n approaches infinity. Therefore, g b ∞ is defined as follows: Particularly, due to the fact that, We have, As presented in (33), g b ∞ has a fixed value. Based on (28) -(33), the proposed dual-scale regression model is iterative and convergent. When g b reaches this fixed value, it is assumed that the convergence is achieved, and the iterative operations are stopped.
The process of the proposed MTF-GLP-HPM-DS is presented in Algorithm 2. The injection coefficients g b are obtained after each iteration and substituted into (26) to obtain the pansharpening result. When the iterative process is stopped, the final pansharpening result is obtained. Fig. 1 shows the flowchart of MTF-GLP-HPM-DS pansharpening method.

Algorithm 2 The proposed MTF-GLP-HPM-DS
Input: an LR-MS image and an HR-PAN image 1) Interpolate the LR-MS image to the size of the HR-PAN image; 2) Obtain P LR using the HR-PAN image by using the MTF-GLP; 3) Calculate the gain coefficients g b by using (27); for i = 0, … ,N − 1 do -Calculate the injection coefficients g b as:

A. Experimental Design
There are two types of quality assessments used to evaluate pansharpening, namely, reduced resolution and full resolution. Please note that the pansharpening results should satisfy two metrics, including consistency and synthesis [2]. The consistency shows that the pansharpening result once degraded at the original MS resolution, should be spectrally similar to the original MS image as much as possible. The synthesis requires the pansharpening result to be similar to the image obtained by the MS sensor. Meanwhile, the reduced resolution evaluation is to reduce the resolution on the basis of the original MS image, so that the original MS image can be used as the reference image of the pansharpening result as they have the same size.
Reduced resolution assessment: We evaluate the synthesis property of Wald's protocol [46,47], and the following quality/distortion quantitative indices are used for performance evaluation.
The relative dimensionless global error in synthesis (ERGAS) [48] denotes a normalized dissimilarity index. It represents a global indicator of the multi-band distortion of the pansharpening results. Generally, a low ERGAS value indicates a high similarity between the pansharpening result and the reference MS image.
The spectral angle mapper (SAM) [49] represents an absolute value of the spectral angle between two-pixel vectors. When the pansharpening result is spectrally identical to the reference image, SAM is assumed to be ideal, i.e., zero.
Another (scalar) index of the pansharpening result with 2 n bands is called Q-index, denoted as Q2 n [50]. Q2 n ranges between [0, 1], and its ideal value is 1.
Full resolution assessment: The quality no-reference (QNR) protocol is used to perform the quality evaluation for the original resolution of the data [51] as follows: where, Q(A, B ) denotes the Q-index between A and B; D denotes the spectral distortion, which is calculated using the original LR-MS image and the pansharpening result; D S represents the spatial distortion, which combines the Q-index between the original LR-MS image and the LR-PAN image and the Q-index between the pansharpening result and the HR-PAN image. The parameter p is typically set to 1. and represent the weight coefficients. It is notable that higher the QNR index, lower is the D index. A lower D S index indicates a better quality of pansharpening result. Therefore, the maximum theoretical value of the QNR index is 1, when both D and D S are equal to zero.
The performance of the proposed MTF-GLP-HPM-DS is compared with the state-of-the-art methods for MRA including:

B. Reduced Resolution
As presented in Table I, we use three datasets obtained by different sensors for evaluating the reduced resolution [53]. In the reduced resolution experiments, the reduced resolution input MS image and the reduced resolution input PAN image are obtained by performing low-pass filtering at a spatial resolution ratio of R = 4.
Toulouse dataset: This dataset comprises images of buildings in an urban area of Toulouse (France), which are acquired by an aerial CNES platform. As shown in Fig. 2(a), the original MS images have a size of 256 × 256 pixels, and have four spectral bands, and a spatial resolution of 0.6m. We use these as the reference images. The original PAN images are presented in Fig.  2(b). The size of each PAN image is 1024 ×1024 pixels with a spatial resolution of 0.15m. As shown in Figs. 2(c) and 2(d), the reduced resolution MS image with a size of 64 × 64 pixels and reduced resolution PAN image with a size of 256 × 256 pixels are obtained by down-sampling the original MS and PAN images by using a low pass filter with a spatial resolution ratio of R = 4. Therefore, the derived pansharpening result has a size of 256 × 256 pixels and can be compared with the reference image. In this dataset, μ = 0.05. WorldView-3 sensor dataset: This dataset comprises imagery of green vegetation in the rural area. The original MS images have a size of 256 × 256 pixels, and have eight spectral bands, and a spatial resolution of 1.24m. These are used as reference images. The original PAN images have a size of 1024 × 1024 pixels and a spatial resolution of 0.31m. The spatial resolution ratio of R = 4 is used to produce the reduced resolution MS and PAN images. The instances of the WorldView-3 sensor dataset are shown in Fig. 3. The parameter μ for this dataset is 0.95. IKONOS sensor dataset: This dataset comprises the images of water in suburban area. The original MS images have a size of 256 × 256 pixels, and have four spectral bands, and a spectral resolution of 4m. These images are used as the reference images. The original PAN image has a size of 1024 × 1024 pixels and has a spatial resolution of 1m. Similarly, the reduced resolution MS and PAN images are obtained by using the spatial resolution ratio of R = 4. The instances of the IKONOS sensor dataset are presented in Fig. 4. The parameter μ for this dataset is 0.5.         To show the differences of pansharpened images better in reduced resolution experiments, the SAM error maps of the sub-areas which are framed in Fig. 2(a), Fig. 3(a) and Fig. 4(a) are shown in Fig. 6, Fig. 8 and Fig. 10. The pansharpening results of the Toulouse dataset at reduced resolution are shown in Fig. 5 and the SAM error maps are shown in Fig. 6. It is evident that MTF-GLP-HPM-DS in Fig. 5(o) is most similar to the reference image presented in Fig. 2(a). There is the least error in Fig. 6(o). Three evaluation indices at reduced resolution are presented in Table II, where it can be seen that Q2 n is equal to Q4 due to four spectral bands in the Toulouse sensor dataset. The three evaluation indices demonstrate that the proposed MTF-GLP-HPM-DS has a better performance in terms of Wald's protocol as compared to the other methods. Although the SAM of MTF-GLP-HPM-DS is not better than the best result, the values of Q4 of the proposed method are closest to 1 as compared to the other methods, while its ERGAS value is the lowest.
The WorldView-3 dataset is used to illustrate the performance of the proposed method on MS images with eight spectral bands. In this test, Q2 n is equal to Q8. The visual performance and quality assessment of fifteen pansharpening results for the WorldView-3 sensor dataset at a reduced resolution are shown in Fig. 7, the SAM error maps in Fig. 8 and Table III. The results show that the proposed MTF-GLP-HPM-DS achieves the best performance in terms of both the visual and index comparison.    The visual comparison and quality assessment of fifteen pansharpening results obtained using the IKONOS sensor dataset at a reduced resolution are shown in Fig. 9, the SAM error maps in Fig. 10 and Table IV. The results in Table IV show that the proposed MTF-GLP-HPM-DS method obtains the highest value in for Q4 and lowest value for SAM among all the compared methods. Although the value of ERGAS for the MTF-GLP-HPM-DS method is not the lowest, it is very close to the best-obtained value of the MF method. This was because a lower μ value brought a better Q-index and SAM value but a worse ERGAS, so μ of 0.5 was used to balance the performance. In general, the proposed method achieves the best performance among all the compared methods.

C. Full Resolution
We evaluate two datasets under a full resolution [54,55] as shown in Table I.
QuickBird sensor dataset: This dataset is acquired under the water in a suburban area. In the full-resolution assessment, the original MS images have a size of 256 × 256 pixels, have four spectral bands, and a spectral resolution of 2.44m, as shown in Fig. 11(a). The original PAN images have a size of 1024 × 1024 pixels and a spatial resolution of 0.61m, as shown in Fig. 11(b). These images are directly used as the input data. Therefore, the derived pansharpening result has a size of 1024 × 1024 pixels. Here, the parameter μ is 0.05. WorldView-2 sensor dataset: This dataset comprises the images of buildings in an urban area. The original MS image with a size of 256 × 256 pixels, eight spectral bands spectral, and a spectral resolution of 2m is presented in Fig. 12(a). The original PAN image with a size of 1024 × 1024 pixels and a spatial resolution of 0.5m is displayed in Fig. 12(b). For this dataset, μ = 0.05.        Because there are no reference images in full resolution experiments, the sub-areas which are framed in Fig. 11(a) and Fig. 12(a) are magnified in Fig. 14 and Fig. 16. The visual results and evaluation indicators of the QuickBird dataset obtained by using the fifteen pansharpening methods at a full resolution are presented in Fig. 13, the sub-areas in Fig. 14 and Table V. The experimental results show that the proposed MTF-GLP-HPM-DS achieves the best performance among all the compared methods. The MTF-GLP-HPM-DS method has the smallest D and D S , and the QNR is very close to 1.
The pansharpening results and evaluation indicators for the WorldView-2 dataset [56] are shown in Fig. 15, the sub-areas in Fig. 16 and Table VI. Similar to the results obtained by using the QuickBird sensor dataset at a full resolution, the proposed MTF-GLP-HPM-DS obtains the lowest values for D and D S and the highest QNR value among all the compared methods.
These results confirm that the proposed method has a good capability to process the urban-scenario images.Based on the overall results, the proposed MTF-GLP-HPM-DS is always superior to the other state-of-the-art methods in terms of both the reduced resolution assessment and the full-resolution assessment.

A. Spatial Resolution Ratio R
The different values of R indicate that the reducedresolution MS with a different resolution is obtained. Therefore, this experiment evaluates the performance of the proposed MTF-GLP-HPM-DS method for another spatial resolution ratio, i.e., R = 2, for MS on the Toulouse and IKONOS datasets. In this experiment, the reduced resolution PAN image is again obtained by down-sampling the original PAN image using a low pass filter with a spatial resolution ratio of R = 4 . The pansharpening result still has the same size as the original reference image [57]. The three evaluation indicators of the fifteen methods at a reduced resolution are given in Tables VII  and VIII. Based on the results presented in Tables VII and VIII, at R = 2, the proposed method shows a good performance on both datasets among all the compared methods. Therefore, the proposed method is suitable for MS with a different resolution.

B. Parameter μ
The parameter μ is introduced to balance the influence of fine M b HR , fine P HR and coarse P LR on (22). Considering the aforementioned experimental results for the reduced resolution and full-resolution assessments, the parameter μ is vital for the MTF-GLP-HPM-DS method and makes the proposed method more flexible towards different sensors and imagery as compared to the other MRA methods. In this experiment, the impact of parameter μ is studied for different sensors and images scenarios. The three datasets for reduced resolution experiments include Toulouse dataset comprising buildings in an urban area, WorldView-3 sensor dataset comprising green vegetation in rural areas, and IKONOS sensor dataset comprising water in a suburban area. The two datasets used for full resolution experiments include QuickBird sensor dataset comprising water in a suburban area and WorldView-2 sensor dataset comprising buildings in an urban area. Therefore, we vary the parameter μ in the range of [0.05, 0.95] with a step size of 0.1 for three datasets used in reduced resolution experiments and two datasets used in full resolution experiments.
In the reduced resolution experiments, the ERGAS, Q-index, and SAM of the proposed method on the three datasets for R = 4 are shown in Figs. [17][18][19]. As shown in Figs. 17 (a), 18 (a) and 19(a), the ERGAS value for Toulouse dataset increases with μ , while it decreases for WorldView-3 and IKONOS sensor datasets. As shown in Figs. 17 (b), 18 (b) and 19(b), the Q-index value for Toulouse dataset and IKONOS sensor dataset decreases with μ, while it increases for WorldView-3 sensor dataset. As shown in Figs. 17 (c), 18 (c) and 19(c), the SAM value for Toulouse dataset increases with μ while it decreases for WorldView-3 sensor dataset and IKONOS sensor dataset. Thus, as the value of μ increases, the values of ERGAS and Qindex for Toulouse dataset became worse, but the value of SAM is improved. As compared to the WorldView-3 sensor data set, the changes are opposite. The change for IKONOS sensor is not obvious as compared with the other two datasets. Considering that the proposed method achieves the best performance among all the methods, μ should be selected appropriately. which made several indicators of the proposed method were better than those of the other fourteen methods.
In the full resolution experiments, the values of D , D S , and QNR for different μ on the two datasets are shown in Figs. 20-21. As shown in Figs. 20-21, the value of μ should be as low as possible. This is because the better Q-index directly results in better D , D S , and QNR according to (35). Therefore, as the value of μ increases, the value of Q-index worsens. Thus, μ is set to 0.05 for the two datasets at full resolution to achieve good performance in terms of several indicators.

C. ERGAS for different R
For IKONOS dataset evaluation performed in previous experiment, we conclude that a lower μ leads to a better Q-index and SAM, but worsens the ERGAS. In order to further illustrate this effect, we test the ERGAS maps for IKONOS sensor dataset at R = 4 and R = 8. As shown in Fig. 22, whether R = 4 and R = 8, the ERGAS decreases with an increase in μ. Thus, for two different R, the μ should be higher in order to obtain better ERGAS. For IKONOS dataset evaluation performed in previous experiment, the parameter μ is set to 0.5 by balancing the other indices in order to obtain the best pansharpening results.

D. Processing Time
The processing time is another important evaluation index of the proposed method's performance. The processing time of the fourteen pansharpening methods for the datasets at reduced and full resolutions are presented in Tables Ⅸ and X, respectively. As shown in Tables Ⅸ and X, although the execution time of  the proposed method is not

E. Limitations
In this work, we build a dual-scale estimation of the HPM injection scheme model for regression based on MRA pansharpening method. It is worth noting that the manual adjustment of the parameter μ is accomplished based on many experiments. This means that any change in parameter μ requires us to modify the values and then again find the most suitable μ, which requires us to perform plenty of experiments. In addition, the accuracy of the parameters obtained by manual adjustment is not too high. Moreover, the processing time of the proposed method can be further improved by using a simplified model.

V. CONCLUSION
In this work, we propose an MRA method for pansharpening based on dual-scale regression, which achieves a better pansharpening result as compared to various state-of-the-art MRA pansharpening methods. In the proposed method, an MRA model based on the scale regression is established. Then, this model is improved by adding the scale regression to the HPM injection scheme. The fine-scale information and coarse-scale information are integrated by the weight parameter and added to the scale regression to construct dual-scale regression, generating the final pansharpening result. The proposed method has two main advantages, including more scale information (fine-scale and coarse-scale information) for the scale regression and better adaptability towards different scenarios obtained by adjusting the weight parameter. The experimental results of the proposed method obtained using the two fourband datasets (the QuickBird and IKONOS datasets) and two eight-band datasets (the WorldView-2 and WorldView-3 datasets) demonstrate a good performance achieved within an acceptable time. The experimental results show that the performance of the proposed MTF-GLP-HPM-DS method is better for both the reduced and full-resolution assessments as compared to the other MRA methods.
In this work, an appropriate weight parameter μ is obtained experimentally. However, manual adjustment of parameter μ can be further improved, and the self-adaptability of this parameter can be considered in the future.

APPENDIX A
We show that the dual-scale regression model is iterative and convergent step by step. First, the initial injection coefficients Thus, the expression of g b n−1 is obtained as: Finally, in order to show that the dual-scale regression model is iterative and convergent, mathematical induction is used to obtain g b n derived from g b n−1 in (30) Similarly, the iterative process used in the regression requires a fixed point when n approaches infinity. Therefore, g b ∞ is defined as follows: