Active Prompting of Vision Language Models for Human-in-the-loop Classification and Explanation of Microscopy Images

Current AI-based methods for the classification of cellular features in microscopy images require time- and labor-intensive processes for training models. Specific limitations include the need for a large amount of image data and major time commitments from domain experts for accurate ground truthing. We present a solution for overcoming these limitations using a state-of-the-art vision language model. Our approach uses GPT-4, the Vision Language Model (VLM) from OpenAI, for the analysis and classification of Iba-1 immuno-stained microglia cells in tissue sections through the mouse hippocampus. We used GPT-4 to classify a dataset of low-power (20x) images of Iba-1 immuno-stained microglia cells from tissue sections treated with saline or a potent neurotoxin (tri-methyl-tin, TMT). Rather than training with images from each class, the GPT-4 input consists of minimal ground-truth prompts for visual question answering. We introduce a novel human-in-the-loop approach to automate the selection of example image-text pairs as input prompts and generate explanatory text as the basis for separating images into distinct classes. We assess test accuracy and efficiency compared to the baseline results using a convolutional neural net applied to the same dataset. Compared to the baseline, the equivalence in accuracy (91%) and substantial (86%) improvement in throughput efficiency with considerably lower needs for input data or domain experts highlight the effectiveness of our new method for automatic image classification. Unlike traditional methods focused on labeling images with one-to-two-word tags, our pipeline generates understandable ground truth by incorporating explanatory text for each image.