Transformers as CFG Learners: A Formal Framework for Structured Data

Transformers excel at sequence modeling but lack theoretical grounding in learning context-free grammars (CFGs). We formalize this relationship, proving that transformers with d ≥ 2|V | layers and h ≥ 3 heads can provably recognize CFG-derived patterns. Experiments demonstrate 96.7% accuracy in JSON parsing and Dyck-k recognition, although ambiguity handling and energy efficiency remain challenges. Our framework provides tools for interpretable, grammar-aware structured data processing, with open-source implementations for reproducibility.