Evidence-Based Control Engineering Education: Evaluating the LCSD Simulation Tool

The advance in control engineering education needs well-designed studies that validate what methods and tools work best. This paper addresses the lack of empirical evidence supporting innovations in control engineering education by proposing a methodology that works at different abstraction levels. Hence, innovations’ impact on students’ performance can be statistically analyzed either globally or locally by examining competencies or fine-grained indicators, respectively. The article reports the application of the methodology for evaluating an interactive simulation tool, named LCSD, on 101 students at the Pontifical Catholic University of Valparaiso in Chile. According to the experimental results, LCSD is an effective free alternative to enhance the student’s skills on control system analysis for our automatic control course. Also, some improvements have been identified for future LCSD versions.


I. INTRODUCTION
Learning the fundamentals of automatic control requires acquiring a solid base on maths and physics to understand the theory thoroughly. Also, students need to learn how to interpret a variety of inter-related diagrams, whose trade-offs are crucial for the analysis and design of control systems [1]. For example, time-response plots, pole-zero map, root locus, and frequency domain diagrams (e.g., Bode, Nyquist, and Nichols). Unfortunately, students frequently struggle to interpret the control information these diagrams depict and to link it with theory [2] correctly.
Although several Interactive Learning Tools (ILT) have been proposed to boost students' understanding on these topics [2]- [15], their real effectiveness has not been validated in most cases. This problem has been pointed out by several authors [16]- [22], who claim that decision making on pedagogical interventions to enhance the students' academic performance should be supported by empirical evidence.
The associate editor coordinating the review of this manuscript and approving it for publication was Atif Iqbal . This paper seeks to overcome this problem by proposing a methodology for evaluating educational innovations in control engineering. The basic idea is to structure the assessment hierarchically, from competencies and learning outcomes to low-level measurement indicators. This way, the teaching innovation effectiveness can be examined at various abstraction levels.
The paper reports the application of the proposed methodology for evaluating an interactive simulation tool called Linear Control System Design (LCSD) [2], [3], which has been specifically designed for teaching the fundamentals of control engineering. LCSD is free 1 and is distributed as lightweight portable binaries for Microsoft Windows and Mac OS (not requiring the installation of additional software). We do not use LCSD as a replacement for actual laboratories but to prepare our students for the second semester, where they work with real hardware. Working with both simulations and actual labs also helps students to recognize that mathematical models are simplifications that do not always mimic hardware's behavior adequately.
LCSD evaluation was conducted on two groups of 44 and 57 students taking an automatic control course at the Pontifical Catholic University of Valparaiso, in Chile. The former group used LCSD, and the latter Matlab. The results (i) validate LCSD as a suitable tool for teaching control concepts, especially the topics related to results' interpretation and validation, and (ii) identify some aspects that need to be improved in future LCSD versions.
The remainder of this paper is organized as follows. Section II shows the shortage of empirical validations in control engineering education by analyzing a sample of 19 articles published in 2019. Section III introduces our methodology and describes the context where it was applied. Section IV reports our evaluation results. Finally, Section V provides some concluding remarks.

II. RELATED WORK
Initial research on the use of computer simulations for assisting control engineering education dates from the early seventies [23]- [25]. Since then, the literature on this topic has grown incessantly. To get an idea about up to what point the pedagogical value of this approach has been validated, let us review the research published in 2019.
In general, collecting the whole population of articles that fall into the scope of a literature review is unrealistic [26]. Accordingly, instead of performing an exhaustive analysis, let us examine a paper sample that represents the population. To gather such sample, we queried Elsevier Scopus that, together with Clarivate Analytics-Web of Science (WoS), is the highest-quality bibliographic databases for research literature [27], [28].
The query in Figure 1 was run. Lines 1-4 set the scope of the search, where TITLE-ABS-KEY(X ) means ''seek for X in the article's title, abstract, and keywords''. Line 5 limits the results to articles published in 2019. Line 6 specifies that we are interested only in papers published in conference proceedings and journals. Finally, Line 7 restricts the subject area to engineering. Table 1 lists the documents obtained with the query, indicating whether they report any empirical evaluation and, in the affirmative case, how the evaluation was carried out. Figure 2 summarizes the results. 68.42% of the 19 papers in the sample do not report any validation at all, 26.32% briefly summarize students' answers to questionnaires, and just one paper provides some descriptive statistics about students' marks. It is worth noting that no paper provides any statistical inference test.

SUMMARY
According to the article sample, there is a lack of empirical evidence supporting the educational value of control engineering simulations. This paper's contributions are (i) a framework that combines qualitative and quantitative statistical analyses to perform the needed experimental validations systematically, and (ii) a report obtained with the mentioned framework that accredits the effectiveness of the LCSD simulation tool for teaching automatic control in a university introductory course.

III. MATERIALS AND METHODS
This paper proposes a methodology to assess the effectiveness of innovations in control engineering education. Data are organized hierarchically in a competency-based model  [48]- [50] to enable instructors to analyze information at different abstraction levels. Whereas the highest levels provide a panoramic view of the results, fine-grained analyses help to identify the innovation effects accurately. We will motivate and illustrate the use of the methodology by assessing the educational value of LCSD in a control engineering course.

A. A BRIEF INTRODUCTION TO LCSD
LCSD is an interactive simulation tool created by researchers at the Universidad Nacional de Educación a Distancia of Spain [2]. LCSD assists students in learning the analysis and design of linear Single-Input Single-Output (SISO) control systems using the loop shaping methodology. Through its interactive interface, students can change the value of various parameters and immediately visualize the effects on the control diagrams. Figure 3 presents the user interface of the LCSD tool, in which 6 sections can be seen: 1) The block diagram section allows selecting the control structure (open or closed loop) and the type of input filter (F), controller (C) and process (P) to analyze. Students can either select and customize a transfer function among a set of predefined templates available for each control diagram block (F, C, and P), or define their own functions from scratch. Besides, it can enable a disturbance (d) at the input to the process and white noise (n) to the control system's output. VOLUME 8, 2020 2) The parameter selection section supports the selection of the parameters related to the input filter, compensator, process, and input signals (r, d, n).

3) The performance Specifications section allows con-
figuring the performance specifications in the time domain, such as overshoot and settling time, and frequency domain, like phase and gain margins. 4) The map of poles and zeros section shows the process poles and zeros. In this section, the user can modify the process roots. 5) The graphics section visualizes different analysis plots: Root locus, Bode, Nichols, Nyquist. 6) In the temporal response section, the user observes the different temporal signals of the block diagram: input, error, actuation and output, depending on which one is selected. Once the process and the control type are set, the student can modify various system parameters in the LCSD interface, receiving immediate visual feedback of these changes on the system. For example, the user can change the controller parameters with the keyboard, through numeric fields established for this, or dragging a slider with the mouse. Furthermore, he/she can make changes in the dynamic behavior of the system, sliding with the mouse the poles of the closed control loop, which correspond to the green x observed at the root locus in Figure 3. With this change, the student observes a new controller gain and new performance parameters in the ''Frequency Specifications'' and ''Time Specifications'' areas.
LCSD has different color indicators that show whether the controller design meets the specifications, both in the frequency and time domain. For instance, in Zone 3 of Figure 3, there are small circles in green or red, depending on whether the performance specification is met or not, respectively. Also, in the graphics area, tinted yellow areas will appear, representing spaces outside the specifications defined for the system performance.

B. COURSE OF AUTOMATIC CONTROL
We have evaluated LCSD on an automatic control course of the master degree on Electrical Engineering at the Pontifical Catholic University of Valparaiso (PUCV), in Chile. The master follows the competency-based pedagogical paradigm [51], encompassing a total of twenty competencies. In the automatic control course, students develop two of them: (C1) solving open and complex problems of electrical engineering, and (C2) simulating electrical systems to represent its behavior, optimize its parameters, and improve its operating conditions. Competencies are subsequently decomposed into learning outcomes, and the latter into assessment indicators. Students' marks are obtained in a bottom-up fashion, from indicators to competencies, computing each element's mark as the weighted average of its descendants' marks.
We proposed and applied this competency assessment model in [50] on the same course, but with a different purpose. Table 2 summarizes the course contents on which our  evaluation was focused. In previous course editions, students showed low performance on these contents, specifically when required to analyze and discuss their results in the applied assessment tools. In this context, instructors decided to incorporate LCSD to improve this situation, thus taking advantage of the continuous improvement scheme also proposed in [50].
To complement our initial proposal, a new evaluative element was included to validate LCSD. The following lowerlevel indicators were defined for measuring the learning outcome under analysis: TR-I and TR-C (Transient Response of a control system), SS-I and SS-C (Steady State), RL-I and RL-C (Root Locus), BD-I and BD-C (Bode diagram), and ND-I (Nyquist diagram); where I and C correspond to interpretation and calculus, respectively. This allowed checking accurately if LCSD helps students to interpret and validate the results obtained from the analysis of a control system. Particularly, we targeted the following research questions: • RQ1: Educational value of LCSD. Is LCSD useful to enhance analysis and interpretation skills of control systems?. In previous course editions, the Control System Toolbox (CST) of Matlab was used. Given that LCSD is free and provides a higher level of interactivity, we were interested in measuring the extent to which LCSD is comparable to what has been done previously, or even more appropriate for our educational purposes.
• RQ2: LCSD melioration. What LCSD improvements do students demand?. LCSD authors have expressed their commitment to support and improve the tool. Our validation could help developers by detecting bugs and new features that LCSD should incorporate.
To answer the questions above, two groups of students were compared in the first semester of 2019-2020. The content, instructors, and laboratory practices were the same for both groups. The only difference was the tool students used to carry out the practices: the control group utilized Matlab CST and the treatment group used LCSD.
To give an idea of the kind of laboratory practices our students undertook, the following section presents a summarized example of one of them by using the LCSD tool. A varied  set of examples illustrating the LCSD use for control system analysis and design can be found in [2].

C. THE DC MOTOR LABORATORY PRACTICE
DC motors are mechanisms typically used for teaching basic control concepts. The following subsections describe some of the learning activities students face to analyze a DC motor's closed-loop behavior by using the LCSD tool.

1) MODELING
In this practice, students work on the armature-controlled DC motor that Figure 4 sketches, with inertia J and viscosity B.
The motor electromechanical model is described by Equations 1-4 (a complete discussion can be found in [52] and [53]).
Since the model is described by a set of linear differential equations, these can be transformed into the Laplace domain, thus simplifying its analytical treatment. Equation 5 shows the transfer function obtained for the motor position θ when  an input voltage V is applied.
Each student has to determine the transfer function for a particular DC motor parameter. For example, Equation 6 would be the transfer function that corresponds to: In this case, and as expected for a DC motor, the transfer function has a pole at the origin of the Z-P diagram, being its closed-loop behavior especially interesting for pedagogical purposes.
2) TIME RESPONSE Based on the process transfer function G p (s), students analyze the control diagram in Figure 5, whose closed-loop transfer function is specified with Equation 7 . Students use LCSD to analyze the stability of the closedloop system, varying K from 0 to ∞, and observing the corresponding changes in the Time Specifications area and Root Locus and Time Response diagrams. Figure 3 shows the results obtained for K = 3, where an overdamped response is obtained (as expected for two dominant closed-loop real poles) without overshoot and a settling time of 4.06 seconds.
Likewise, LCSD adapts the diagrams dynamically as the student changes the gain K interactively, either modifying the numeric field or slider (hand-click 1 in Figure 6a) or dragging a closed-loop pole on a Root Locus branch (hand-click 2 in Figure 6a). Figure 6a also shows the diagrams obtained after the student has dragged the real closed-loop pole closer to the imaginary axis in Figure 3 until a new placement on the Root Locus branch for K = 25. In this case, an underdamped response is observed (as expected for two dominant closedloop complex poles) with an overshoot P o = 51.26% and a settling time of 4.5 seconds. If the student follows dragging the closed-loop pole on the Root Locus branch, the critical stability point is reached for K = 57 (see Figure 6b) and going   beyond, the imaginary axis is crossed getting the instability of the system (see Figure 7). Since LCSD displays all diagrams jointly in runtime, students easily relate changes on the Root Locus, Time Response, and Performance Specifications to gain changes, thus improving their insight on control system analysis in both time-response and stability concepts.

3) FREQUENCY RESPONSE
To study control systems in the frequency domain, students use the LCSD tool for stability analysis, primarily by looking at the phase and gain margins (ϕ m and g m ) using Bode or Nyquist diagrams as K varies. Figure 8 shows the plots that LCSD generates for K = 10 for the same transfer function G p (s) of the DC motor. Firstly, Root Locus plot is replaced by the Bode plot keeping the Time Response graph below so that students can observe the time and frequency response relationship.
Subsequently, students explore the Freq. Specification area to get the specific values of phase and gain margins. Likewise, two vertical lines in the Bode plot represent such values as well. Additionally, LCSD allows setting the desired stability requirements, displaying different colors depending on whether the system meets them, which is used for controller design purposes. On the other hand, they can see the relationship between the numerical value of the margins and the time specifications on runtime as they change the gain K interactively through the corresponding slider.
The student is also required to obtain the value of the gain crossover frequency ω gc and the phase crossover frequency ω φc . This is done by moving the mouse over the points where the gain crosses the 0 dB, and the phase crosses the -180 • , respectively, such as shown in Figure 8. After that, the student is required to obtain the phase and gain margins analytically in order to check their theoretical results with those obtained through the simulation. Figure 9 explains how LCSD helps to recognize the time and frequency response dependency. For example, for K = 30 the phase margin is ϕ m = 18.8, showing a time response with high oscillation (P o = 60%). When the student decreases the gain, the phase margin increases, and the time response presents fewer oscillations (K = 12, ϕ m = 45.6, and P o = 23% in Figure 9b).
Finally, by using the Nyquist diagram, the student can also observe the system stability visually. Figure 10 shows how the phase and gain margins are deployed in LCSD for K = 12. The student can change the gain and observe the effects instantaneously, helping him/her to relate different aspects such as margins, frontier stability point (−1 + 0j), and time response.

IV. RESULTS AND DISCUSSION
This section reports the results of our evaluation. The treatment and control groups were composed of 44 and 57 students that used Matlab CST and LCSD, respectively.

A. QUALITATIVE ANALYSIS
To gather students' opinions regarding the usability and usefulness of LCSD and Matlab CST, we asked them to fill a straightforward two-item questionnaire in the last laboratory session. Responding to the questionnaire was voluntary. The participation ratio was 42/44 and 52/57 students for the treatment and control groups, respectively.
Also, to encourage them to express honest and personal comments about LCSD, we proposed two more open-ended questions. Following Oppenheim's recommendations [54], open-ended questions were formulated as sentence-completion items to center students' responses. For example: Please complete the following sentence in your own words: LCSD strengths to learn the analysis of control systems are . . . Figure 11 summarizes the obtained frequencies for two questions that inquire students about the usefulness and usability of LCSD and Matlab CST. Responses are rated on a five-point Likert scale.    Table 3 summarizes the responses of the treatment group students to the open-ended questions. Note that the total number of participants in this group was 42 students. The right-hand column indicates the percentage of students that agree with the response in the left-hand column. The table only summarizes opinions supported by at least two students.

SUMMARY
According to the closed-questions, LCSD is as appropriate as Matlab CST for our introductory automatic control course, but less usable. The open-ended questions concerning LCSD reinforce this. Students appraise LCSD interactivity, diagrams, and simulation capabilities; however, they think LCSD needs to be polished in various aspects that affect its learnability and usability.

B. QUANTITATIVE ANALYSIS
All students from both the control and treatment groups undertook the same final exam. This section compares the exam marks of those groups. Table 4 and the box-plot in Figure 12 summarize the total scores, i.e., considering all exam questions. It is worth noting that, in the Chilean education system, students are graded from 1 to 7. The red dashed line in Figure 12 shows the threshold to pass the exam.

1) GLOBAL EXAM
The treatment group scores look slightly above the control group ones. To verify if the difference is statistically significant, a onetailed independent t-test was performed (see Table 5). First, a Levene's test was used to assess the groups' equality of variances. As the Levene's p-value ≥ 0.05, the equality hypothesis holds, and thus the t-test does not requires Welch's adjustment. The t-test p-value is 0.032; so, the group difference is statistically significant. Finally, VOLUME 8, 2020 the last column reports the effect size. According to the rule of thumb given in [55], a Cohen's d around 0.2 indicates a small effect.

2) LOW-LEVEL INDICATORS
We were particularly concerned about whether LCSD helps students to fulfill the learning outcome in Table 2. Accordingly, this section focuses on the fine-grained indicators TR (Transient Response of a control system), SS (Steady State), RL (Root Locus), BD (Bode diagram), and ND (Nyquist diagram). Table 7 and Figure 13 summarize the exam scores for these specific indicators. The notation INDICATOR-I and -C distinguishes between the scores that students obtained interpreting and calculating the indicator, respectively. For instance, TR-I stands for the students' scores on the interpretation of TR, and TR-C stands for the students' scores on TR calculation. It is worth noting that the course includes the interpretation of Nyquist diagrams, but not their calculation. Table 6 summarizes the results of the null hypothesis significance tests for all the exam indicators. To counteract the Type I error inflation due to the multiple comparisons, the Bonferroni corrections  The explanation regarding how to perform the calculations is theoretical, i.e., students learn them in a traditional classroom session. Consequently, there is no meaningful difference between the treatment and control groups concerning the calculation indicators (TR-C, SS-C, RL-C, and BD-C). This comparison verifies that both groups are similar in cognitive terms. In contrast, there are differences between LCSD and Matlab CST in all interpretation indicators, being statistically significant the ones in TR-I, SS-I, and BD-I.

SUMMARY
Concerning the global exam scores, there is a small but statistically significant difference between the control and treatment groups. Analyzing the exam scores at a more fine-grain level, there are statistically significant differences in the TR-I, SS-I, and BD-I indicators with medium, large, and small effect sizes, respectively.

V. CONCLUSION
The methodology presented in this paper guides researchers and practitioners to perform evaluations for collecting evidence about the strengths and shortcomings of innovations in control engineering education. This information is fundamental to advance toward better tools and methods.
On the other hand, we have identified LCSD's most useful features (e.g., high interactivity, multiple and simultaneous inter-related plots, etc.) and drawbacks (e.g., bugs, improvable documentation, interface rigidity, etc.), thus orienting LCSD future development. The use of low-level assessment indicators allows a fine-grained analysis regarding students' academic performance in the different sub-topics studied, thus observing their strengths and weaknesses. This information plays a crucial role in continuous academic improvement. However, we are aware that gathering the sub-scores in assessment tools requires a significant effort. Our future research will seek to take advantage of the automatic assessment paradigm to collect indicators in a fast and effective way, thus reducing the effort required in this process.
LORETO MARIN received the degree in electronic civil engineering from the Universidad Técnica Federico Santa María, Valparaíso, Chile, in 2006. She is currently pursuing the M.Sc. degree in electrical engineering with the Pontificia Universidad Católica de Valparaíso. Her current research interests include system identification, dynamic system simulation, industrial automation, automatic control, and control education.
HECTOR VARGAS received the degree in electrical engineering from the University of La Frontera, Temuco, Chile, in 2001, and the Ph.D. degree in computer science from UNED, Madrid, Spain, in 2010. Since 2010, he has been with the Electrical Engineering School, Pontificia Universidad Católica de Valparaíso, Chile. His current research interests include dynamic system simulation, automatic control, industrial automation, the IoT-based distributed systems, and engineering education.