Characterizing UX Evaluation in Software Modeling Tools: A Literature Review

Model-Based Software Engineering (MBSE) has a high potential to play a critical role in the whole process of software engineering, bringing many benefits to all stakeholders, yet it is not used by most software developers today, due to both lack of tool capabilities and poor user experience (UX) of tools. This study aims to understand the evaluation types and methods applied by researchers when studying UX in modeling tools (modeling experience or MX) and the types of issues uncovered in these studies. We conducted a literature review using a snowballing approach to gather all studies of this topic. A total of 41 research papers were reviewed. Data extraction was performed based on research questions and a categorization of discussed issues was presented. Several gaps and future opportunities were identified and discussed, which include 1) utilizing interview method in research design; 2) distributing testing tasks based on user profiles; 3) involving UX experts in analysis; 4) scalability testing using large models; 5) assessing MX in areas other than just usability and utility; 6) considering collaborative modeling as an important factor contributing to MX; 7) considering both language issues and tool issues in UX evaluation of software modeling tools; 8) improving the taxonomy of MX challenges; 9) triangulating using multiple methods; and 10) developing and validating MX tool design heuristics.