Comparative evaluation of visualization and experimental results using image comparison metrics

Comparative evaluation of visualization and experimental results is a critical step in computational steering. In this paper, we present a study of image comparison metrics for quantifying the magnitude of difference between visualization of a computer simulation and a photographic image captured from an experiment. We examined eleven metrics, including three spatial domain, four spatial-frequency domain and four HVS (human-vision system) metrics. Among these metrics, a spatial-frequency domain metric called 2nd-order Fourier comparison was proposed specifically for this work. Our study consisted of two stages: base cases and field trials. The former is a general study on a controlled comparison space using purposely selected data, and the latter involves imagery results from computational fluid dynamics and a rheological experiment. This study has introduced a methodological framework for analyzing image-level methods used in comparative visualization. For the eleven metrics considered, it has offered a set of informative indicators as to the strengths and weaknesses of each metric. In particular, we have identified three image comparison metrics that are effective in separating "similar" and "different" image groups. Our 2nd-order Fourier comparison metric has compared favorably with others in two of the three tests, and has shown its potential to be used for steering computer simulation quantitatively.