Comparative Analysis of Unsupervised and Supervised Autoencoders for Nuclei Classification in Clear Cell Renal Cell Carcinoma Images

This study explores the application of supervised and unsupervised autoencoders (AEs) to automate nuclei classification in clear cell renal cell carcinoma (ccRCC) images, a critical diagnostic task that traditionally relies on subjective visual grading by pathologists. We evaluate various AE architectures, including standard AEs, contractive AEs (CAEs), and discriminative AEs (DAEs), as well as a classifier-based discriminative AE (CDAE), optimized using the hyperparameter tuning tool Optuna. Bhattacharyya distance is selected from several metrics to assess class separability within the latent space for different architectures, revealing challenges in distinguishing adjacent grades using fully unsupervised models. CDAE, integrating a supervised classifier branch, demonstrated superior performance in both latent space separation and classification accuracy. Given that CDAE-CNN achieved notable improvements in classification metrics, affirming the importance of supervised learning for enhancing class-specific feature extraction in ccRCC nuclei, F1 score was incorporated into the hyperparameter tuning process to achieve the best-fitting model. The results demonstrate distinct improvements in identifying more aggressive ccRCC grades by leveraging the classification capability of AE through latent space clustering followed by supervised fine-grained classification so that our model outperfotms state of the art, CHR-Network, in all evaluated metrics. Our findings suggest that not only does the inclusion of a classifier branch in AEs provide a promising approach for improving grading automation in ccRCC pathology compared to unsupervised clustering or supervised learning alone, but the use of neural architecture search combined with contrastive learning techniques to structure embeddings in the latent space of the AE architecture also leads to enhanced performance, particularly in identifying more aggressive grades, which are more influential in tumor grading. This advancement has the potential to improve the accuracy of the final diagnosis.