Exploring Large Language Models for Requirements on String Values

Behavior-driven development (BDD) enables collaboration among different stakeholders by employing a natural-language representation of system requirements and of test scenarios. These scenarios often involve constraints over string values, e.g. for the validity of email addresses, which are challenging to test comprehensively. Traditional methods like SMT solvers (e.g. Z3, Ostrich) handle constraints efficiently but produce unrealistic strings and require formal specifications that are often unavailable and expensive to compute. This paper explores the potential of large language models (LLMs) in generating realistic, constraint-satisfying strings for BDD. We propose an evaluation framework to assess LLMs’ ability to (1) generate consistent string values and (2) detect constraint inconsistencies. In our experiments, three LLMs are compared to state-of-the-art solvers using constraints from a software engineering course project. Results show that while solvers dominate in precision and recall, LLMs derive realistic strings more suitable for a requirements engineering context. With these trade-offs, we believe that, when formal constraints are available, a combined LLM-solver approach could offer a more effective solution.