The Impact of Combining Datasets on the Robustness of Deep Learning Architectures: A Cross-Dataset Analysis

Convolutional Neural Networks have become a widely used technology. Typically, the performance of CNN models is measured using an accuracy score obtained on a single dataset. This often results in systems that perform significantly worse in real-world applications. The robustness of the model to unseen environments is still underresearched. Specifically, a research gap remains on how training data affects the robustness of deep learning systems. This research article investigates the impact of combining data in training datasets on the robustness and performance of deep learning models through a cross-dataset analysis. We employ a transfer learning approach to train deep learning models based on four popular architectures and two different datasets, as well as a combination of both datasets. Our results demonstrate that combining two datasets can improve robustness, but the specific effects on performance can vary between architectures, leading to a slight decrease in accuracy in most observed cases, or even an accuracy gain. Furthermore, we find that training on more complex datasets tends to outperform training on simpler datasets in cross-evaluation settings, indicating that models trained on more complex training datasets are more robust. However, we also observe that a simpler architecture fails to generalize when trained on the combined training data, indicating the need for caution and extensive evaluation when combining datasets during the development cycle of deep learning systems.