SMATCH-M-LLM: Semantic Similarity in Metamodel Matching With Large Language Models

Metamodel matching plays a crucial role in defining transformation rules in model-driven engineering by identifying correspondences between different metamodels, forming the foundation for effective transformations. Current techniques face significant challenges due to syntactical and structural heterogeneity. To address this, matching techniques often employ semantic similarity to identify correspondences. Traditional semantic matchers, however, rely on ontology matching tools or lexical databases, which often struggle when metamodels use different terminologies or hierarchical structures. Inspired by the contextual understanding capabilities of Large Language Models (LLMs), this paper explores the capability of GPT-4 potentials as a semantic matcher and alternative to existing methods for metamodel matching. However, metamodels can be large, which can overwhelm LLMs if provided in a single prompt, leading to reduced accuracy. Therefore, we propose prompting LLMs with fragments of the source and target metamodels, identifying correspondences through an iterative process. The fragments to be provided in the prompt are identified based on an initial mapping derived from their elements’ definitions. Through experiments with 10 metamodels, our results show that our LLMbased approach improves the accuracy of metamodel matching, achieving an average F-measure of $\approx 91 \%$, outperforming both the baseline and hybrid approaches, which have a maximum average F-measure of $\approx \mathbf{2 9 \%}$ and $\approx \mathbf{7 4 \%}$, respectively. Moreover, our approach surpasses single-prompt LLM-based matching, which has an average $\mathbf{F}$-measure of $\mathbf{8 0 \%}$, by approximately $\mathbf{1 1 \%}$.