Fragility is one of the most important properties of authentication-oriented image hashing. However, to date, there has been little theoretical analysis on the fragility of image hashing. In this paper, we propose a measure called expected discriminability for the fragility of image hashing and study this fragility theoretically based on the proposed measure. According to our analysis, when Gray code is applied into the discrete-binary conversion stage of image hashing, the value of the expected discriminability, which is dominated by the quantization stage of image hashing, is no more than 1/2. We further evaluate the expected discriminability of the image-hashing scheme that uses adaptive quantization, which is the most popular quantization scheme in the field of image hashing. Our evaluation reveals that if deterministic adaptive quantization is applied, then the expected discriminability of the image-hashing scheme can reach the maximum value (i.e., 1/2). Finally, some experiments are conducted to validate our theoretical analysis and to compare the performance of several quantization schemes for image hashing.