Objective quality assessment is important and widely used in image processing. Recently, the metric named structural similarity is proposed, which is based on the assumption that human visual perception is highly adapted for extracting structural information. This metric has a better performance than PSNR in many cases but fails in case evaluating the badly blurred images. This limitation is inconsistent with the characteristics of human visual system (HVS) and leads to our inspiration of applying HVS characters to images structural similarity. Our method, HVS-based structural similarity(HSSIM), employs the HVS characters both in frequency domain and spatial domain. It can be concluded in our experiment that HSSIM performs better than PSNR and SSIM, especially for badly blurred images.