Private Sampling of Latent Diffusion Models for Encrypted Prompt

Generative artificial intelligence has made great progress in enabling clients to create a variety of realistic visual content (such as images, videos and audios), where diffusion model as an emerging generative model can obtain higher quality images than generative adversarial networks (GANs). For resource-constrained devices, high-definition images can be generated by outsourcing the AI model to the server, but the client’s local prompts and generated images may contain private information, increasing the risk of privacy disclosure. In order to alleviate these concerns, we adopt the idea of split learning and homomorphic encryption technology to ensure the privacy of data and prevent the attacker from stealing. Specifically, adopt homomorphic encryption to ensure the confidentiality of the text embedding and prevent the server from obtaining the prompt and text embedding. Secondly, during the process of denoising, a secure cross-attention mechanism is designed for ciphertext based on the embedding matrix to ensure the normal follow-up steps and prevent denoising model parameters from leaking to clients. In addition, since the decoder is deployed locally, the image generated by the denoising model is latent spatial features rather than pixel spatial features, so the generated image is not visible to the server. Finally, through the theoretical analysis and quantitative index of experiments, it is proved that the image generated under ciphertext prompt is almost the same as the image generated under plaintext prompt, which indicates the security and effectiveness of the proposed protocol.