We describe techniques for matching two images or an image and a map. This operation is basic for machine vision and is needed for the tasks of object recognition, change detection, map up-dating, passive navigation, and other tasks. Our system uses line-based descriptions, and matching is accomplished by a relaxation operation which computes most similar geometrical structures. A more efficient variation, called the ``kernel'' method, is also described. We give results on complex aerial images which contain many image differences, caused by varying sun position, different seasons, and imaging environments, and also structural changes caused by man-made alterations such as new construction.