Impulse noise detection is a critical issue when removing impulse noise and impulse/Gaussian mixed noise. In this paper, we propose a new detection mechanism for universal noise and a universal noise-filtering framework based on the nonlocal means (NL-means). The operation is carried out in two stages, i.e., detection followed by filtering. For detection, first, we propose the robust outlyingness ratio (ROR) for measuring how impulselike each pixel is, and then all the pixels are divided into four clusters according to the ROR values. Second, different decision rules are used to detect the impulse noise based on the absolute deviation to the median in each cluster. In order to make the detection results more accurate and more robust, the from-coarse-to-fine strategy and the iterative framework are used. In addition, the detection procedure consists of two stages, i.e., the coarse and fine detection stages. For filtering, the NL-means are extended to the impulse noise by introducing a reference image. Then, a universal denoising framework is proposed by combining the new detection mechanism with the NL-means (ROR-NLM). Finally, extensive simulation results show that the proposed noise detector is superior to most existing detectors, and the ROR-NLM produces excellent results and outperforms most existing filters for different noise models. Unlike most of the other impulse noise filters, the proposed ROR-NLM also achieves high peak signal-to-noise ratio and great image quality by efficiently removing impulse/Gaussian mixed noise.