Photovoltaic Cell Defect Detection Based on Weakly Supervised Learning With Module-Level Annotations

Recently, convolutional neural networks (CNNs) have proven successful in automating the detection of defective photovoltaic (PV) cells within PV modules. Existing studies have built a CNN based on fully supervised learning, which requires a training dataset consisting of PV cell images annotated according to whether the individual cells are defective. However, manually annotating all the PV cells is labor-intensive and time-consuming, leading to substantial annotation costs. In this study, we propose a weakly supervised learning method to build a CNN for cell-level defect detection in a cost-efficient manner. Our method uses a training dataset solely with module-level annotations indicating whether each PV module contains defective cells, thereby substantially reducing the required annotation costs. The CNN is trained in a weakly supervised manner such that all cells in a normal module are classified as normal and at least one cell in a defective module is classified as defective. The CNN can then be used to detect cell-level defects in new PV modules. The effectiveness of the proposed method is validated through experiments using real-world data provided by a PV module manufacturer.


I. INTRODUCTION
Photovoltaic (PV) cells, also known as solar cells, are semiconductor devices that convert light energy into electricity through photovoltaic effects.In the PV module manufacturing, a PV module consists of several PV cells that are wired and sealed in a protective laminate.During the manufacturing process, PV modules are occasionally damaged for various reasons, such as thermo-mechanical stresses and assembly failures [1], [2].Therefore, manufacturers inspect the fabricated PV modules for quality assurance.
Electroluminescence (EL) imaging is a standard inspection method for detecting defects in the cells in PV modules [3].

Various defects in the cells, such as micro-cracks and wiring
The associate editor coordinating the review of this manuscript and approving it for publication was Mostafa M. Fouda .problems, can be identified by inspecting the EL image of a PV module, an example of which is shown in Figure 1.After the EL inspection, the engineers replace the defective cells with new cells.
Typically, experienced engineers visually inspect EL images.However, the visual inspection of numerous EL images is laborious and time-consuming.Engineers working for long periods of time can become visually fatigued, which increases the risk of inspection errors.Therefore, the automation of EL inspection has been in demand [1].
Research attempts have been made to apply machine learning to automate the inspection of defective cells in PV modules.Existing studies have built a convolutional neural network (CNN) that uses a cell image as input to predict whether it is defective [1], [2], [4], [5], [6].In the context of fully supervised learning, a CNN is trained using a cell-level In this study, we propose a weakly supervised learning method to build a CNN with reduced annotation costs for PV cell defect detection.Unlike the existing studies, the proposed method uses a training dataset annotated at the module level as shown in Figure 2b, where each module is annotated to indicate whether it contains at least one defective cell.Since annotating a module can be performed more efficiently by engineers than annotating each individual cell within the module, acquiring N module-level annotations incurs substantially lower costs than acquiring N × K cell-level annotations.To train the CNN using the module-level annotated training dataset, we introduce a weakly supervised learning objective such that all cells in a normal module are classified as normal and at least one cell in a defective module is classified as defective.The effectiveness of the proposed method is verified by experiments using a real-world dataset provided by a PV module manufacturer.
The remainder of this paper is organized as follows.We review related work in section II.The proposed method is presented in section III.section IV describes the experimental settings and results.Finally, section V presents the conclusions.

II. RELATED WORK A. AUTOMATION OF PHOTOVOLTAIC CELL DEFECT DETECTION
To automate the detection of defective PV cells, most previous studies have attempted to build a classification model based on supervised learning using a cell-level annotated training dataset.Demant et al. [7] manually extracted handcrafted features from cell images and built a support vector machine based on the handcrafted features as a classification model to detect defective cells with microcracks.With advances in deep learning for computer vision tasks, recent studies have adopted CNNs as classification models [1], [5], [6].A CNN processes a cell image without requiring manual feature extraction to predict the defect score.Deitsch et al. [2] and Ahmad et al. [4] showed that CNNs were superior to models based on manual feature extraction in detecting defective cells.Otamendi et al. [8] and Mayr et al. [9] built CNNs for the pixel-level defect detection in defective cell images.
Some studies have implemented an unsupervised learning approach using a training dataset consisting of normal cell images only in the absence of defective cell images.This approach is useful when defective cell images are unavailable during the training phase.Tsai et al. [10] and Tsai et al. [11] applied the Fourier transformation and independent component analysis, respectively, to learn the normal class structure of a training dataset.A query cell image was identified as an anomaly when the reconstruction error was high.Shou et al. [12] sequentially trained a generative adversarial network and an autoencoder network using a training dataset.They determined the anomaly score by calculating the residuals between the query image and the image produced by the generator.
The supervised learning approach generally performs better than the unsupervised learning approach, because defective cell images are explicitly used to train the classification model.However, a major challenge is the high cost of acquiring cell-level annotations that indicate whether a cell is defective.The objective of this study is to alleviate the annotation cost required to build a classification model for cell-level defect detection.We propose a method based on a weakly supervised learning approach that utilizes module-level annotations instead of cell-level annotations.

B. WEAKLY SUPERVISED LEARNING
Weakly supervised learning is an umbrella term that encompasses various attempts to build a model by learning from a training dataset under weak supervision [13].It is practically useful when it is difficult to acquire strong supervision information for the training dataset, owing to the high annotation cost.Weak supervision can be categorized as incomplete, imprecise, and inaccurate supervision.
Incomplete supervision refers to a situation in which only some training instances are labeled, whereas others are not.Active learning [14] is used to selectively annotate unlabeled instances to improve the performance of the model.Semisupervised learning [15] uses both labeled and unlabeled instances in a partially annotated training dataset to train the model.
Imprecise supervision refers to a situation where the training dataset contains only coarse-grained labels.The multiple-instance learning approach [16], [17] uses a set of labeled bags, each containing many instances, as a training dataset to build a model for instance-level prediction.Each bag is annotated with a single label and the labels of individual instances are not provided.
Inaccurate supervision refers to a situation where the labels of the training instances are noisy.Noisy labels in the training dataset negatively affect the training of the model, thereby degrading its performance [18].The preprocessing approach [19] identifies mislabeled training instances before training the model.The re-labeling approach [20], [21] iteratively re-annotates labeled instances whose labels are expected to be incorrect by the model.The robust learning approach [22] uses a model architecture, loss function, or regularization that is robust to noisy labels.
This study aims to use a module-level annotated training dataset to build a CNN for cell-level prediction, which corresponds to imprecise supervision.In the training dataset, each module consists of multiple cells and is annotated as defective if at least one cell is defective and as normal otherwise.We adapt the multiple-instance learning approach by considering a module and a cell as a bag and an instance, respectively.

III. METHODOLOGY A. OVERVIEW
We formulate the PV cell defect detection as a binary classification problem, where a classification model f takes a PV cell image x as the input to predict whether the cell is defective as follows: where ŷ ∈ [0, 1] is the predicted defect score.A high defect score signifies an increased likelihood that a given cell is defective.As the classification model f , the objective is to build a CNN by learning from a module-level annotated training dataset , where X i = (x i,1 , . . ., x i,K ) is the i-th module containing K cells and Y i ∈ {0, 1} is the corresponding class label.Each cell x i,k ∈ R m×n in the module is represented as a grayscale image of shape m × n.The cell-level class label y i,k is not provided.The module-level class label Y i takes values of 0 and 1 to indicate that the module belongs to the normal and defective classes, respectively.
Although the module-level annotated training dataset D module is provided in the training phase, the trained CNN f is required to make cell-level predictions during the inference phase, as illustrated in Figure 2b.The supervised learning approach shown in Figure 2a, which has been used in existing studies, is not applicable in this situation because it requires cell-level annotations in the training dataset.
In this study, we introduce a weakly supervised loss function to bridge the gap in granularity between the training and inference phases.The loss function leverages information from the module-level annotations in the training dataset D module .The model f is trained in a weakly supervised manner so that all cells in a normal module are predicted to be normal, and at least one cell in a defective module is predicted to be defective.The details of the loss function are described in the following subsection.The notations used in this paper are listed in Table 1.

B. TRAINING WITH MODULE-LEVEL ANNOTATIONS
A PV module is considered normal if all associated cells are normal, whereas it is considered defective if at least one cell is defective.Accordingly, all the cells in a normal module should be classified as normal, whereas at least one cell in a defective module should be classified as defective.Owing to the absence of explicit cell-level annotations, the weakly supervised loss function is designed to identify the cell in a defective module that differs the most from normal cells.Using the loss function, the CNN f is trained to classify all cells in a normal module as normal and the identified cell in a defective module as defective.
For a training instance (X i , Y i ), the CNN f predicts the defect scores of the individual cells x i,1 , . . ., x i,K as ŷi,1 , . . ., ŷi,K .The K defect scores are compared with the module-level label Y i .The loss function is computed as follows: where γ is a hyperparameter that balances these two terms.The computational path of the loss function is illustrated in Figure 3.If the module is normal (Y i = 0), then the first term is used to decrease the defect scores of all cells in the module.If the module is defective (Y i = 1), then the second term is used to increase the maximum defect score among the cells in the module.As training progresses, the CNN f is encouraged to exclusively output higher defect scores for defective cells within defective modules, while suppressing the defect scores for normal cells in both normal and defective modules.
For each training iteration with a mini-batch S ⊂ D module , the parameters of the CNN f are updated by minimizing the following objective function J : where |S| is the number of modules in the mini-batch.
In terms of computational cost during the training phase, the proposed method is comparable to existing methods that rely on fully supervised learning with cell-level annotations.In each training iteration, the existing methods require K forward and backward passes for each module.The proposed method requires the same for each normal module.However, it requires K forward passes and only a single backward pass for each defective module.

C. INFERENCE FOR CELL-LEVEL AND MODULE-LEVEL DEFECT DETECTION
For inference, the trained CNN f is used to detect defective PV cells.For cell-level defect detection, the inference process of the proposed method is identical to that of existing methods that rely on fully supervised learning with cell-level annotations.Given a query cell image x * ,k , the defect score   ŷ * ,k can be obtained as follows: A cell is classified as defective if the defect score ŷ * ,k is greater than a threshold θ, i.e., ŷ * ,k > θ.
For module-level detection, the defect score of the query module X * = (x * ,1 , . . ., x * ,K ) is calculated as maximum defect score among the cells in the module: (5)

IV. EXPERIMENTS A. DATA ACQUISITION AND PREPROCESSING
The effectiveness of the proposed method was investigated using a real-world dataset collected from Shinsung E&G, a PV module manufacturer in South Korea, by EL imaging of the manufactured PV modules.This dataset consists of two parts with different levels of annotations by domain experts, the first part with module-level annotations D module and the second part with cell-level annotations D cell .The first part D module contained 1,418 module EL images with module-level annotations.The resolution of the images ranged from 5243 × 2676 to 5392 × 2702.Each image consisted of 144 cell images (i.e., K = 144) in 6 rows and 24 columns.Of the 1,418 images, 604 (42.6%) and 814 (57.4%) images were annotated as normal and defective, respectively.We split the EL image of each raw module into 144 individual cell images using border lines.As the borders between the cells were relatively dark, the positions of the border lines were automatically detected by finding the peaks of the horizontal and vertical means of the pixel values in the module image [6].
The second part D cell contained 4,861 cell EL images that were separately examined by domain experts.Each image was annotated at the cell level.Among the 4,861 images, 4,464 (91.8%) and 397 (8.2%) images were annotated as normal and defective, respectively, indicating that the class distribution was imbalanced.The types of defects included crack, contamination, dead zone, dark area, dark spot, and solder darkness.The resolution of the images ranged from 84×201 to 105×213. Figure 4 shows examples of EL images for normal and defective cells.
In the experiments, the first part of the dataset was used as the training dataset to train the classification model.The second part was used as the test dataset to evaluate the performance of the model.
Each cell image was preprocessed as follows.We resized the image to 50 × 100 and increased the contrast by a factor of two to emphasize the defective region.The pixel values were standardized using the mean and standard deviation of ImageNet [23].

B. EXPERIMENTAL SETTINGS
The performance of the proposed method was evaluated using different settings for the classification model.It is important to note that the existing methods are not applicable to the problem situation addressed in this study, where a cell-level defect detection model needs to be trained using a module-level annotated training dataset.We used two common CNN architectures with different numbers of layers: VGG11, VGG13, VGG16, and VGG19 for the VGGNet family [24] and ResNet18, ResNet34, ResNet50, and ResNet101 for the ResNet family [25].We initialized each model with ImageNet-pretrained parameters [23].For the VGGNet models, we removed all fully-connected layers and the flatten operation, and then added a new global average pooling and a new fully-connected layer with a single output unit.For the ResNet models, the existing fully-connected layer was replaced by a new fully-connected layer with a single output unit.
In the training phase, we trained the classification model with the module-level annotated training dataset, among which 90% of the dataset was used to update the parameters and the remaining 10% was used to monitor the validation loss.The hyperparameter γ in the objective function J was set to 0.01.Data augmentation was applied by randomly transforming the cell images at each training iteration based on vertical/horizontal flip (p = 0.5), rotation ([−1, 1 • ]), vertical/horizontal shift ([−1, 1%]), brightness adjustment ([−10, 10%]), and contrast adjustment ([0, 50%]).To update the parameters, we used the Adam optimizer with an initial learning rate of 10 −5 and a weight decay factor of 10 −8 .The mini-batch size was set to eight modules, with 144 cells each (i.e, 144 × 8 = 1, 152 cells).The learning rate was halved if the validation loss did not decrease over 5 consecutive epochs.The training was stopped if the validation loss did not decrease for 20 consecutive epochs or if the number of epochs reached 1,000.
In the inference phase, we evaluated the performance of the model by using the cell-level annotated test dataset.We applied test-time augmentation with the same transformation operations as in the training phase.For a query image, inference was performed 30 times with different random image transformations.The individual predicted defect scores were then averaged to obtain the final defect score.
We calculated four performance measures on the test dataset: accuracy, balanced accuracy, F1 score, and area under the receiver operating characteristic curve (AUROC).The first three measures are dependent on the threshold θ; thus, we individually optimized the threshold θ for each measure.The AUROC is a threshold-independent measure.
The proposed method was implemented using PyTorch in Python.The experiments were conducted with 10 independent replications using different random seeds.We report the mean and standard deviation of each result over 10 replications.

C. RESULTS AND DISCUSSION
Table 2 presents an overall comparison of the classification performance of the classification models trained using the proposed method.Most of the models achieved a considerably high classification performance with an AUROC of over 0.97, especially when the CNN architectures used were relatively small.This indicates that the proposed method successfully trained the models in a weakly supervised manner using only module-level annotations.Among the compared models, the smallest model, VGG11, yielded the highest AUROC value.On the other hand, when the architecture was larger (e.g., ResNet101), we observed a tendency towards instability in training and a deterioration in classification performance, thereby resulting in a lower average and a higher standard deviation of the performance measures over the 10 replications.
We further analyzed the classification results for VGG11, which yielded the highest AUROC value.Figure 5 presents the ROC curve of VGG11, where the true positive rate against the false positive rate is plotted at different threshold settings.Figure 6 illustrates the trade-off between false positive rate (FPR) and false negative rate (FNR) at various thresholds.Increasing the threshold θ leads to a lower FPR but a higher FNR, whereas decreasing θ leads to a lower FNR but a higher FPR.This suggests that the threshold θ can be optimized by taking into account the different costs associated with different types of misclassification errors.For example, we can decrease the threshold θ if false negatives (i.e., classifying defective cells as normal) should be avoided.The threshold θ can also be optimized based on a specific performance criterion.Figure 7 shows the confusion matrices of VGG11 for various threshold values, each optimized based on different performance measures.The results showed that the optimal thresholds differed depending on the specified performance measure.The use of accuracy and balanced accuracy resulted in the highest and lowest thresholds, respectively.
Figure 8 presents the heatmap visualization of the cell-level predictions of the PV module examples for VGG11.One was a normal module and the other three were defective modules containing different types of defective cells.The cells predicted as normal and defective were shaded green and red, respectively.When a module contained more than one defective cell, the proposed method successfully detected these cells by assigning high defect scores, as shown in Figure 8c and Figure 8d.In these examples, all cells in the normal module had low defect scores, whereas the cells with high defect scores in the defective modules were identified as defective by domain experts.

V. CONCLUSION
In this study, we proposed a weakly supervised learning method to build a CNN to detect PV cell defects without cell-level annotations.By introducing a weakly supervised loss function, the CNN was trained using a module-level annotated training dataset such that the most defective cell in a defective module was predicted as defective and all cells in a normal module were predicted as normal.The CNN successfully detected defective cell images in a real-world dataset.
While previous studies relied on supervised learning with costly cell-level annotations, this study pioneers the application of weakly supervised learning with module-level annotations to build a CNN for cell-level prediction.Although a CNN trained with cell-level annotations can provide better classification performance, the proposed method is useful in practical scenarios where reducing the annotation cost for constructing the training dataset is crucial.We expect that the proposed method will lead to cost-efficient applications in the automatic PV cell defect detection.In the future, we plan to extend the proposed method to multi-class or multi-label classifications, where each cell should be classified according to its specific defect categories.

FIGURE 1 .
FIGURE 1. Example of EL image for PV module.

FIGURE 2 .
FIGURE 2. Schematic of training and inference for PV cell defect detection.

FIGURE 4 .
FIGURE 4. Examples of EL images for normal and defective cells.

FIGURE 6 .
FIGURE 6. False positive rate and false negative rate at different thresholds (VGG11).