Contextualized Word Embeddings via Generative Adversarial Learning of Syntagmatic and Paradigmatic Structure

In this work, we propose a general regularized function-learning framework of contextualized word embeddings, named Generative Adversarial Learning of Syntagmatic and Paradigmatic structure of Context (GAL-SPC), aiming to build word embeddings via the trainable encoder function based on the distributional structure. What makes a difference is the context of a given word is viewed as a connected graph so that the context-level syntagmatic structure and the document-level statistical pattern can be formalized as a vocabulary co-occurrence matrix. Meanwhile, to better integrate the paradigmatic structure, GAL-SPC exploits posteriori approach based on edit distance and prior approach based on knowledge base to pick paradigmatic contexts with substitutable words and regularizes the training of autoencoder via generative adversarial learning process to better model the syntagmatic and paradigmatic structure of contexts. Finally, we trained GAL-SPC with a public wikipedia corpus, and the intrinsic evaluations with word analogy and similarity tasks and the extrinsic evaluations with document clustering and classification tasks demonstrates that GAL-SPC outperforms other comparative methods.