CA-CoCon: An improved self-supervised method for controlled text generation using cross-activation attention mechanism

The widespread application of pre-trained large models in the field of natural language generation has facilitated the development of models capable of generating fluent textual content. While conventional text generation models often struggle to effectively control text content at the sentence level, self-supervised learning models like CoCon offer finer-grained control over generated text at the word and phrase levels. Although the text generated by CoCon tends to adhere closely to the given topic, it still exhibits deviations from human communication norms, lacking in textual diversity. Addressing these issues, we propose a novel approach based on the multi-granularity controllable text generation model, CoCon, termed as Cross-Activation (CA)-CoCon. This approach assists the model in capturing nonlinear features between input text and target content more effectively, thus facilitating the seamless integration of target content into the generated text. The proposed CA-CoCon model outperforms the untreated CoCon model across multiple thematic and emotional attribute categories, yielding text outputs that align more closely with human communication norms and exhibit enhanced textual diversity.