Assessing the Usefulness of a Visual Programming IDE for Large-Scale Automation Software

Industrial control applications are usually designed by domain experts instead of software engineers. These experts frequently use visual programming languages based on standards such as IEC 61131-3 and IEC 61499. The standards apply model-based engineering concepts to abstract from hardware and low-level communication. Developing industrial control software is challenging due the fact that such systems are usually one-of-a-kind systems that have to be maintained for many years. These challenges, together with the growing complexity of control software, require very usable model-based development environments for visual programming languages. However, so far only little empirical research exists on the practical usefulness of such environments, i.e., their usability and utility. In this paper, we discuss common control software maintenance tasks and tool capabilities based on existing research and show the realization of these capabilities in 4diac IDE. We first performed a walkthrough of the demonstrated capabilities using the cognitive dimensions of notations framework from the field of human-computer interaction. We then improved the tool and conducted a user study involving ten industrial automation engineers, who used 4diac IDE in a realistic control software maintenance scenario. Our findings demonstrate how the usefulness of IDEs can be successfully investigated using a multi-phase approach that includes a walkthrough and a user study. We discuss lessons learned and derive general implications with respect to large-scale applications for developers of IDEs that we deem applicable in the context of (visual) model-based engineering tools.