Auto-rectification of user photos

The image auto rectification project at Google aims to create a pleasanter version of user photos by correcting the small, involuntary camera rotations (roll / pitch/ yaw) that often occur in non-professional photographs. Our system takes the image closer to the fronto-parallel view by performing an affine rectification on the image that restores parallelism of lines that are parallel in the fronto-parallel image view. This partially corrects perspective distortions, but falls short of full metric rectification which also restores angles between lines. On the other hand the 2D homography for our rectification can be computed from only two (as opposed to three) estimated vanishing points, allowing us to fire upon many more images. A new RANSAC based approach to vanishing point estimation has been developed. The main strength of our vanishing point detector is that it is line-less, thereby avoiding the hard, binary (line/no-line) upstream decisions that cause traditional algorithm to ignore much supporting evidence and/or admit noisy evidence for vanishing points. A robust RANSAC based technique for detecting horizon lines in an image is also proposed for analyzing correctness of the estimated rectification. We post-multiply our affine rectification homography with a 2D rotation which aligns the closer vanishing point with the image Y axis.