Deep-Learning-Based Retrieval of an Orange Band Sensitive to Cyanobacteria for Landsat-8/9 and Sentinel-2

The lack of an orange band (∼620 nm) in the imagery captured by Landsat-8/9 and Sentinel-2 restricts the detection and quantification of harmful cyanobacterial blooms in inland waters. A recent study suggested the retrieval of orange remote sensing reflectance, Rrs (620), by assuming green, red, and panchromatic (Pan) bands of Landsat-8 as predictors through a linear model. However, this method is not applicable to Sentinel-2 imagery lacking a Pan band. Moreover, the Pan-based method does not account for the nonlinear relationships among the Rrs data at different wavelengths. We propose a deep-learning model called Deep OrAnge Band LEarning Network (DOABLE-Net) that leverages a large training set of Rrs data from radiative transfer simulations and in situ measurements. The proposed DOABLE-Net is structured as five fully connected layers and implemented either with or without the Pan band as an input feature, which only the latter applies to Sentinel-2. DOABLE-Net provided more accurate and robust retrievals than the Pan-based method on a wide range of independent validation datasets. The performance of DOABLE-Net on Landsat-8/9 data was minimally impacted by including the Pan band. The results from Sentinel-2 data analysis also confirmed that the DOABLE-Net provides promising results without using a Pan band.


I. INTRODUCTION
R EMOTE sensing of water quality in inland and nearshore coastal waters has undergone significant progress over the past decade with the launch of Landsat-8/9 and Sentinel-2 satellites [1], [2]. Operational land imager (OLI) and multispectral imager (MSI) sensors aboard Landsat-8/9 and Sentinel-2, respectively, are not originally designed for aquatic applications. However, their spatial resolution (OLI 30 m, MSI 10−60 m) and dynamic range (12−14 bit) have been favorable in various studies conducted in optically complex inland and coastal waters [3], [4], [5]. OLI and MSI capture five and seven bands, respectively, within the visible and near-infrared portion of the spectrum (< 800 nm), which are useful for water quality studies. Although OLI and MSI opened up new opportunities, their spectral band designations are not optimal for retrieving Manuscript  some water constituents. In particular, mapping and quantifying cyanobacteria have appeared challenging due to the lack of an orange band in these sensors [6]. Cyanobacterial harmful algal blooms (cyano-HABs) can develop dense biomasses producing toxins that severely endanger the aquatic habitat and associated ecosystem services [7], [8]. Given the substantial adverse effects of cyano-HABs, the accurate and timely monitoring of cyanobacteria biomass by means of satellite remote sensing is a high priority. Most ocean color sensors, such as ocean and land color instrument onboard Sentinel-3, capture an orange band centered at ∼620 nm that facilitates the detection of phycocyanin, which is the characteristic pigment of cyanobacteria [9], [10]. However, the spatial resolution of these sensors (hundreds of meters) is too coarse for studying inland waters. On the other hand, the acquisition of images from spaceborne hyperspectral missions, such as PRISMA [11], [12], is not regular despite the sufficient spatial resolution (30 m). Therefore, Landsat-8/9 and Sentinel-2 remain the key satellites for studying inland and coastal water quality, offering suitable spatial and temporal resolutions (5-8 days).
Phycocyanin is a photosynthetic pigment with a unique spectral response of an absorption peak at ∼ 620 nm and a reflectance peak at ∼650 nm that distinguishes cyanobacteria from other phytoplankton communities [13], [14]. However, OLI and MSI lack such an orange band that undermines their utility for mapping cyanobacteria. A recent study proposed an approach to retrieve the remote sensing reflectance at 620 nm, R rs (620), for Landsat-8 imagery based on a linear relation incorporating the panchromatic (Pan), green, and red bands [9]. Note that R rs is the ratio of water-leaving radiance to downwelling irradiance just above the surface and has units of inverse steradians (sr −1 ) [15]. The Pan-based orange band retrieval technique provided promising results in mapping phycocyanin concentration in Lake Erie [16] and Eagle Creek Reservoir [17] and has been implemented in the ACOLITE processor [18]. The main hypothesis of this method is to exploit the spectral information available in the panchromatic band, which is a wide band overlapping with multispectral channels. However, this method, called the Pan-based method hereafter, is built upon a simple linear model that employs Pan, green, and red bands as predictors for the orange band retrieval. Thus, it does not account for the complex nonlinear relations among R rs data at different wavelengths, which are typical in optically complex waters [19]. Moreover, the Pan-based method is not applicable to the imagery from sensors, such as MSI, which does not capture a panchromatic band.
This study aims to develop an advanced deep-learning-based model called Deep OrAnge Band Learning Network (DOABLE-Net) for retrieving a virtual orange band from multispectral bands of Landsat-8/9 or Sentinel-2. Our proposed method is grounded in the high level of spectral interdependence of R rs data [20], [21], [22] and the ability of deep models to learn complex nonlinear features from a large set of training data. In a recent study, a deep-learning model is developed for retrieving near-blue ultraviolet R rs bands from the visible ones obtained by ocean color sensors and provided encouraging results [19]. We build upon this new line of research to advance the stateof-the-art in orange band retrieval by pursuing the following objectives.
1) Develop DOABLE-Net for Landsat-8/9 and Sentinel-2 with high generalization capability by leveraging a large training dataset of radiative transfer simulations and in situ spectral measurements. 2) Examine the impact of involving the Landsat-8/9 Pan band on the performance of the proposed DOABLE-Net. 3) Evaluate the accuracy of orange band retrieval in a wide range of bio-optical conditions and perform comparisons with the Pan-based method [9]. The rest of this article is organized as follows. The training and validation datasets are presented in Section II. Section III introduces the proposed DOABLE-Net and briefly describes the existing Pan-based method and its calibration for retrieving the orange band from Landsat-8/9 data. Section IV illustrates the validation results of DOABLE-Net compared with the Panbased method. Finally, Section V concludes this article.

II. DATASETS
One of the key aspects of developing supervised deep-learning models is creating a large and representative training dataset. In aquatic applications, the in situ data are quite limited in space and time, which hinders building a comprehensive training set. Complementary to in situ data, deep-learning models can largely benefit from radiative transfer simulations [23], [24] providing a massive dataset spanning diverse bio-optical conditions. The radiative transfer models account for the absorption and backscattering properties of pure water and its constituents. The simulation allows the creation of a large set of R rs spectra spanning a wide range of inherent optical properties and concentrations of constituents [23].
We use a large set of hyperspectral R rs for training DOABLE-Net consisting of radiative transfer simulations (100 000 samples) up to 800 nm representing a wide range of bioptical conditions, which are taken from the article presented in [19], and shipborne reflectance measurements (5805 samples) up to 950 nm in the optically complex Baltic Sea conducted by Tilstone et al. [25].
We employ four independent datasets (4311 samples in total) representing very diverse bio-optical conditions to examine the prediction performance of DOABLE-Net. The validation datasets and the sources are summarized in Table I. The data   TABLE I  IN SITU HYPERSPECTRAL DATASETS USED FOR THE VALIDATION OF  DOABLE-NET AND PAN-BASED METHODS originally used for training and validation of the Pan-based method, acquired at Belgian and Dutch lakes, are also among our validation datasets. The abbreviations of the datasets (see Table I) are taken from the data sources.
Although the four in situ R rs datasets provide an independent and comprehensive means of evaluating the performance of orange band retrieval methods, we also apply DOABLE-Net to Landsat-9 (OLI-2) and Sentinel-2 (MSI) imagery from San Francisco Bay (see Fig. 1). The images from the two sensors are acquired on the same day (December 10, 2021). The ACOLITE processor [18], [30] is used to apply dark spectrum fitting atmospheric correction. As shown in Fig. 1, the bay and surrounding coastal area show a wide range of color gradients representing various water types from clear blue to turbid waters with sediment plumes. We perform a consistency analysis between the results of the DOABLE-Net and Pan-based methods, retrieving the R rs (620) for the Landsat-9 image. Moreover, the consistency of R rs (620) retrieval for Sentinel-2 is assessed relative to Landsat-9.

III. METHODS
The generic workflow of this study is summarized in Fig. 2. The main steps include the following.
1) Preparation of training data by spectral resampling of the hyperspectral R rs data: This step provides the input multispectral R rs data of the DOABLE-Net. The workflow is shown for the OLI data that includes the Pan band allowing for training two different individual networks with and without including this band as an input feature.
In the case of MSI, the training does not involve a Pan band. The R rs (620), created from spectral resampling of the hyperspectral R rs , serves as the response variable of the network. 2) Training the DOABLE-Net with an architecture described in Section III-A. 3) Prediction of the orange band using the trained model and accuracy assessment compared with the reference data.
Once the DOABLE-Net is trained, it is applied to the validation OLI or MSI datasets to predict the orange band. The validation set is not seen through the training and allows for assessing the model's generalization capability.   The rest of the section describes the details of each step. We first elaborate the proposed DOABLE-Net method. Then, the Pan-based method is presented in its original and calibrated versions. Finally, the accuracy metrics are provided for assessing results and consistency analysis among the methods.

A. Proposed DOABLE-Net
Neural networks (NNs) have been proven to be powerful in learning complex and nonlinear relations between input features and the target parameter [31]. Moreover, deep models handle the feature extraction automatically and can leverage low-and high-level features [32], [33]. Given the nonlinear relationships of R rs at different wavelengths [19], we also build upon a deep NN architecture to enable predicting the orange band from multispectral R rs data.
An initial step for the development of supervised deep models is the preparation of the training dataset. The hyperspectral training data (see Section II) are resampled with the Landsat-8/9 and Sentinel-2 spectral response functions for the bands up to 800 nm. Every multispectral band is calculated by a weighted averaging of the hyperspectral R rs data. The weights are taken from the spectral response function of either OLI or MSI sensors. Similarly, an orange band is obtained from the hyperspectral data equivalent to the one considered in the Pan-based method [9]. The relative spectral responses of OLI and MSI sensors and the orange band centered at 620 nm are shown in Fig. 3 [34].
The proposed DOABLE-Net comprises multiple fully connected hidden layers. We adapted this architecture from the article presented in [19] that provided promising results in a similar task, i.e., retrieving near-blue ultraviolet R rs from visible bands. Although the types of layers are taken from the article presented in [19], the number of layers and hidden units are optimized through a tuning process. Similar architectures have shown outstanding results also in other aquatic applications (e.g., bathymetry and water quality retrieval), dealing with comparable regression tasks using data without spatial context information [35], [36], [37], [38].
The first layer is supplied with the multispectral R rs bands of either Landsat-8/9 or Sentinel-2 created from the hyperspectral training data. Thus, the input layer involves four R rs bands of Landsat-8/9 [b1-b4, Fig. 3(a)] or seven bands of Sentinel-2 [b1-b7, Fig. 3(b)]. We have implemented another version of DOABLE-Net for Landsat-8/9 that includes the Pan band as an extra input feature (five features in total). The latter version is termed DOABLE-Net-Pan, hereafter, which is applicable only to Landsat-8/9 data. The last fully connected layer is connected to the output layer that produces the R rs (620) as the response parameter of the network. As typical fully connected architectures, every layer multiplies the input by a weight matrix followed by adding a bias vector. The working mechanism of such NNs is well-documented in the literature [39].
The hyperparameters of DOABLE-Net, including the number of layers, the number of neurons in each layer, and the type of activation function, are tuned through a Bayesian optimization [40], [41]. The hyperparameter tuning investigates a search space for every parameter to find the optimal value that minimizes the cross-validation error. The tuned architecture of the proposed DOABLE-Net for Landsat-8/9 consists of five fully connected layers with 250, 174, 96, 85, and 22 neurons at each layer, successively.
A similar architecture is defined for Sentinel-2 with layer sizes of 239, 205, 105, 34, and 15 neurons. The tuning is performed in a search space with layer sizes of up to 500 neurons that are large enough for this task [19]. The rectified linear unit is considered as an activation function that effectively tackles gradient  [42]. A schematic representation of the DOABLE-Net architecture is shown in Fig. 4.

B. Original and Calibrated Pan-Based Method for Retrieving Orange Band for Landsat-8/9
The Pan-based method is the only study in the literature that addresses the orange band retrieval for Landsat-8/9 [9]. This approach relies on a simple linear regression model considering the green (R Green rs ), red (R Red rs ), and Pan (R Pan rs ) bands as predictors of the orange band R rs (620) formulated in (1). The linear model is originally trained using 428 in situ hyperspectral R rs spectra resampled to Landsat-8 bands and the orange band. The training set represents a wide range of bio-optical conditions in Belgian and Dutch lakes [9], which is used as a validation set in this study (see Table I The coefficients of the Pan-based method are assumed to be generic and implemented in ACOLITE processor [34], although one may update the coefficients using other training datasets. In this study, we compare the performance of the Pan-based method with the proposed DOABLE-Net considering both original and calibrated coefficients. The calibration of the Pan-based model is based on the same dataset used for training DOABLE-Net.
Hereafter, we refer to this version of the model as Pan-based-Cal. The calibrated coefficients are provided as follows: R Cal rs (620) = 3.0807× 10 −4 + 3.703× R Pan rs − 1.6714× R Green Although the Pan-based method is originally developed for Landsat-8, it can be directly applied to Landsat-9 data due to the identical spectral designation of OLI and OLI-2 [43]. However, this method does not apply to Sentinel-2 data that lacks the Pan band. Note that the Pan-based method aims to enhance the spectral resolution and does not involve any pan sharpening to enhance the spatial resolution. Thus, in the image analysis, the Pan band is downsampled to the spatial resolution of multispectral bands (i.e., 30 m).

C. Accuracy Assessment Metrics
We use a set of metrics to quantify the accuracy of orange band retrieval, including coefficient of determination (R 2 ), rootmean-square error (RMSE), normalized RMSE (NRMSE), and bias E i and O i stand for the estimated and observed/known values of the R rs (620), respectively. The NRMSE indicates the percentage of the RMSE relative to the range of the retrieved parameter (orange band). The bias is computed in a log-transformed space that accounts for the proportionality of the errors [44]. Bias > 1 implies an overestimated R rs (620), whereas bias < 1 denotes an underestimation error. For instance, a 15% overestimation in R rs (620) translates into a bias of 1.15. So, the closer the bias values to 1, the lower the systematic errors. The accuracy metrics are computed for the four validation datasets (see Table I) by comparing the orange band retrievals with the reference in situ observations. In the case of image analysis (San Francisco Bay), the consistency of the maps derived from different methods is assessed by quantifying the pixel-by-pixel agreement in terms of R 2 and normalized root-mean-square difference (NRMSD).

IV. RESULTS AND DISCUSSION
The validation scatterplots of retrieved versus measured R rs (620) are shown in Fig. 5 for the four validation R rs datasets (see Table I). The accuracy statistics based on different metrics are reported in Table II. The proposed DOABLE-Net for Landsat-8/9, without incorporating the Pan band, provides better or comparable retrievals of R rs (620) relative to the Pan-based method. For instance, DOABLE-Net improved the NRMSE of orange band retrieval by ∼3% for the NORCOHAB dataset. Inclusion of the Pan band through DOABLE-Net-Pan improves the results but not significantly. In the case of the Belgian-Dutch lakes dataset, the DOABLE-Net provides very close R rs (620) retrievals compared with the Pan-based method, even without using the Pan band. This is a promising result for the DOABLE-Net as this dataset was originally used for developing the Pan-based method, whereas it is totally independent for DOABLE-Net (unseen through training). The results also indicate that the calibration of the Pan-based method slightly drops the accuracy of the R rs (620) retrieval. For instance, the NRMSE for SeaSWIR dataset increases from 3.8% to 6.5% when using the calibrated version of the Pan-based method instead of the original one (see   Table II). This can be attributed to the nonlinear and complex interrelationships among the R rs data at different wavelengths that are more pronounced in the large training set from diverse biooptical conditions. So, deep models, such as the proposed DOABLE-Net, are preferred to provide robust retrievals. The orange band retrievals from the Sentinel-2 data are comparable with those of Landsat-8/9 data (see Fig. 5 and Table II). This finding indicates that the proposed DOABLE-Net can reasonably retrieve the R rs (620) for Sentinel-2 data in the lack of a Pan band.
Examples of randomly selected Landsat-8/9 spectra from different datasets, including the retrieved virtual orange band, are shown in Fig. 6 compared with the reference spectra from the hyperspectral in situ data. The original Landsat-8/9 spectra, without the orange band, are also illustrated to better understand the impact of the retrieved R rs (620) in capturing the spectral shape of water-leaving radiance. As evident, the proposed DOABLE-Net, even without incorporating the Pan band, provides stronger or comparable agreements with the reference spectra relative to the Pan-based method. As evident, the retrieved R rs (620) based on DOABLE-Net enables better capturing of the spectral shape around 620 nm. Fig. 7 shows the R rs (620) maps retrieved from the Landsat-9 and Sentinel-2 imagery acquired over San Francisco Bay. There are strong agreements among different cases derived from DOABLE-Net and Pan-based methods (R 2 > 0.97, RMSD < 0.0018 sr −1 , NRMSD < 6%, and 0.98 <bias< 1.03 for any pair of maps). The orange band retrieved based on the Pan-based-Cal for the Landsat-9 image is slightly noisy in the coastal area implying again that the calibration of the linear regression model based on large training data (2) is not helpful. This experiment also conveys that the inclusion of the Pan band has minimal impacts on retrieving the R rs (620) from the Landsat-9 image over diverse water types. Similarly, the orange band retrieval from Sentinel-2 in the lack of a Pan band is comparable with that of Landsat-9.

V. CONCLUSION
Multispectral satellite sensors, such as OLI and MSI, are yet the main sources of imagery for studying inland waters from space. However, their limited spectral bands hinder the retrieval of specific bio-optical attributes. The lack of an orange band is a key limitation in these sensors' spectral designation, making the detection and quantification of cyano-HABs challenging. This issue has motivated the idea of retrieving a virtual R rs (620) with some potentials previously demonstrated based on the Pan-based method [9]. However, this method is not applicable in the lack of a Pan band, which is the case for Sentinel-2. Furthermore, the Pan-based method is based on a simple linear model, neglecting the complexity and nonlinearity of interrelationships among the R rs data at different portions of the spectrum. We proposed DOABLE-Net, which is based on a deep regression model that allows for automatically capturing the informative features from multispectral data to predict the R rs (620) either with or without using a Pan band. Although the results indicate the minimal impact of the Pan band on the performance of the proposed DOABLE-Net, there is no reason to exclude this band when it is available, such as for Landsat-8/9 data. However, these results encourage retrieving the orange band for Sentinel-2 without a Pan band. The calibration of the Pan-based model using the same training set of DOABLE-Net degraded the performance of R rs (620) retrieval. This is because our training set is much larger (∼ 250 times) and more diverse than the dataset used in computing the Pan-based method's original coefficients, amplifying the nonlinearity and complexity of the orange band retrieval. Thus, the proposed DOABLE-Net provides a more robust means of retrieving R rs (620) by automatically extracting nonlinear features.
We focused on developing a methodology for R rs (620) retrieval that is evaluated with diverse datasets. Future studies need to assess the impact of the proposed DOABLE-Net in retrieving phycocyanin concentration and mapping cyano-HABs. However, the benefits of R rs (620) retrievals from the Pan-based method have previously been demonstrated in some studies [16], [17]. Thus, such benefits can also be expected with more promises for DOABLE-Net, given its robustness and less dependency on the Pan band. The DOABLE-Net can also potentially be extended for any other multispectral sensor lacking the orange band that requires more investigation. Although we employed a five-layer fully connected network, other deep architectures can also be assessed through orange band retrieval.