A weakly supervised image classification framework is presented in this paper. Given reference images marked by clinicians as relevant or irrelevant, we learn to automatically detect relevant patterns, i.e. patterns that only appear in relevant images. After training, relevant patterns are sought in unseen images in order to classify each image as relevant or irrelevant. No manual segmentations are required. Because manual segmentation of medical images is extremely time-consuming, existing classification algorithms are usually trained on limited reference datasets. With the proposed framework, much larger medical datasets are now available for training. The proposed approach has been successfully applied to diabetic retinopathy detection in the Messidor dataset (Az =0.855). Moreover, we observed, in a new dataset of 473 manually segmented images, that all eight types of diabetic retinopathy lesions are detected.