A method is presented that uses grouping to improve local classification of image primitives. The grouping process is based upon a spin-glass system, where the image primitives are treated as possessing a spin. The system is subject to an energy functional consisting of a local and a bilocal part, allowing interaction between the image primitives. Instead of defining the state of lowest energy as the grouping result, the mean state of the system is taken. In this way, instabilities caused by multiple minima in the energy are being avoided. The means of the spins are taken as the a posteriori probabilities for the grouping result. In the paper, it is shown how the energy functional can be learned from example data. The energy functional is defined in such a way that, in case of no interactions between the elements, the means of the spins equal the a priori local probabilities. The grouping process enables the fusion of the a priori local and bilocal probabilities into the a posteriori probabilities. The method is illustrated both on grouping of line elements in synthetic images and on vessel detection in retinal fundus images.