A novel ellipse detector based upon edge following is proposed in this paper. The detector models edge connectivity by line segments and exploits these line segments to construct a set of elliptical-arcs. Disconnected elliptical-arcs which describe the same ellipse are identified and grouped together by incrementally finding optimal pairings of elliptical-arcs. We extract hypothetical ellipses of an image by fitting an ellipse to the elliptical-arcs of each group. Finally, a feedback loop is developed to sieve out low confidence hypothetical ellipses and to regenerate a better set of hypothetical ellipses. In this aspect, the proposed algorithm performs self-correction and homes in on “difficult” ellipses. Detailed evaluation on synthetic images shows that the algorithm outperforms existing methods substantially in terms of recall and precision scores under the scenarios of image cluttering, salt-and-pepper noise and partial occlusion. Additionally, we apply the detector on a set of challenging real-world images. Successful detection of ellipses present in these images is demonstrated. We are not aware of any other work that can detect ellipses from such difficult images. Therefore, this work presents a significant contribution towards ellipse detection.