Collaborative Human-machine Perception for Cross-modal Image Compression

Recent advances in deep learning-based image compression approaches have demonstrated significant improvements in terms of rate-distortion performance. However, most existing image compression methods only optimize pixel-level reconstruction at certain bitrate from the perspective of human perception, ignoring the effect of semantics contained in other modalities on image compression. This limitation results in suboptimal compression performance of existing codecs and makes them unsuitable for cross-modal reasoning. To address the issue, a textguided cross-modal image compression method with collaborative human-machine perception is proposed in this work. Specifically, a cross-modal encoding-alignment module is proposed to align textual representations with visual representations from the intermediate layers of neighborhood attention Transformer to improve the compressibility of cross-modal features. Meanwhile, learnable text prompts are designed into the encoder to expand the distinction between the semantics of target text and non-target pixels, which facilitates the compressed cross-modal representations to achieve cross-modal reasoning and pixel-level reconstruction for human. Experiments validate that the proposed crossmodal image compression method outperforms traditional coding standards and recent deep image coding methods in both pixellevel reconstruction and the performance of machine analysis at similar bitrates.