The identification of anatomical landmarks in the brain is an important task in registration and morphometry. The manual identification and labelling of these landmarks is very time consuming and prone to observer errors, especially when large datasets must be analysed. In this paper we present an approach that describes landmarks based on their intrinsic geometry, rather than their intensity patterns. As the proposed approach moves away from the traditional way to describe landmarks (based on intensities), we show that using this kind of descriptors are well suited for the landmark localisation problem in MR brain images since the intensity information in these images is not quantitative (and intensity normalization is not straight forward). Our results show that for localizing 20 anatomical landmarks in brain MR images, the proposed descriptor performs better in 75% of cases when compared with a Haar feature based classifier and 100% of cases when compared to non-rigid registration.