Scalable Image Compression Based on Diffusion Models at Ultra-Low Bitrates

Image codecs are typically optimized to trade-off between bitrate and distortion metrics. At low bitrates, they often result in compression artefacts. To solve this problem, we leverage the ability of diffusion models to produce high-quality images and design a scalable lossy image compression framework for ultra-low bitrates, as shown in Fig. 1. On the encoding side, we extract multiple conditional information from the image and perform scalable encoding on them. When the bandwidth is limited, only the text is encoded. We use Prompt Inversion (PI) to extract text and apply lossless compression using Lempel-Ziv (LZ) coding from the zlib library. As the bandwidth gradually increases, sketch and Spatial color palette can be encoded to supplement the conditional information. We use the edge prediction model PiDiNet to extract the sketch of the image. We further perform scalable palette compression to provide color information at different bitrates. We downsample the image by 64x using bicubic interpolation and upsample it to 1/32, 1/16, 1/4, and 1/2 of the original size with nearest-neighbor to obtain the spatial color palette at different bitrates. We use standard learned nonlinear transform codes (NTC) to compress the sketch and spatial color palette. On the decoding side, We use the T2I-Adapter [1] as our decoder. The text is processed through the CLIP text encoder to generate conditional text embeddings, while the sketch and spatial color palette are fed through adapters to obtain features at different scales. The decoding is performed in a scalable manner, where the transmitted information is processed as described above and input into the diffusion model to guide the image generation. Compared to JPEG and the latest diffusion-based methods (PIC and PICS [2]), our method achieves better perceptual quality, as shown in Table 1. It's important to note that t denotes using text as the sole condition, ts indicates using both text and sketch, and tsc1 to tsc4 refer to using text, sketch, and color palette at different bitrates as conditions. Experimental results show that as the number of conditional information increases and the accuracy of color information improves, the perceptual quality of the images improves.