Student Awareness of Models in First-Year Engineering Courses

Contribution: This study assesses more than 800 students’ awareness of engineering model types before and after taking two first-year engineering courses across two semesters and evaluates the effect of each course. Background: All engineers must be able to apply and create models to be effective problem solvers, critical thinkers, and innovative designers. To help them develop these skills, as a first step, it is essential to assess how to increase students’ awareness of engineering models. According to Bloom’s taxonomy, the lower remember and understand levels, which encompass awareness, are necessary for achieving the higher levels, such as apply, analyze, evaluate, and create. Research Questions: To what extent did student awareness of model types change after taking introductory engineering courses? To what extent did student awareness of model types differ by course or semester? Methodology: In this study, a survey was designed and administered at the beginning and end of the semester in two first-year engineering courses during two semesters in a mid-sized private school. The survey asked students questions about their definition of engineering modeling and different types of models. Findings: Overall, student awareness of model types increased from the beginning of the semester toward the end of the semester, across both semesters and courses. There were some differences between course sections, however, the students’ awareness of the models at the end of the academic year was similar for both groups.

engineering education [1]. Since modeling is essential to solving and designing engineering problems in the workforce, it is imperative engineering students are specifically taught about different types of models, as well as how to develop and apply them [2]. There are many benefits to explicitly teaching modeling, particularly in the first years of an engineering program [1], [3], [4]. Although there are some known pedagogical interventions (e.g., model-eliciting activities (MEAs) [4]), there is still a significant need to develop more meaningful ways of teaching modeling throughout the engineering curricula, especially for first-year engineering students [1], [2].
There has been an extensive amount of impactful research around modeling interventions, including development of pedagogical approaches and assessment tools, within the Computational adaptive expertise (CADEX) [3], [5] and models and modeling perspective (M&MP) [6] frameworks. For instance, Carberry, McKenna, Linsenmeier, and Cole [7] conducted research within the CADEX framework and found that explicit modeling interventions caused a significant shift in the modeling conceptions of senior engineering students. In addition, to gain a greater understanding of modeling conceptions, Carberry and McKenna [1] expanded their research within the CADEX framework, noting that when students were taught a comprehensive mathematical module, they were more likely to discuss mathematical and predictive models. Research efforts within the M&MP have focused around a mathematical modeling intervention called MEAs [4]. Some of this research has focused on how students develop mathematical model solutions to MEAs (e.g., [8] and [9]), MEA implementation strategies within engineering courses (e.g., [10] and [11]), and the improvement of MEA implementation strategies in large first-year engineering (e.g., [12]) and upper-division courses (e.g., [13] and [14]).
There are many concepts currently integrated in engineering curricula that implicitly teach modeling skills, but lack clear instruction around the underlying use of engineering models. This is especially true about many first-year engineering courses that focus on core concepts, such as problem solving, design, computer-aided design (CAD), and introductory computer programming [1], [2], [15], [16]. Even though all these concepts involve modeling, that involvement may not be definitively discussed or demonstrated. For example, in many CAD courses, the instructional materials focus on the specifics of how to use different tools and features in the CAD software rather than the ideas of modeling and their applications [17]. Similarly, teaching students how to develop an algorithmic solution (a type of This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ model) is fundamental to programming, although there is often a greater focus on syntax [16]. Most engineering education studies on computer programming focus on paired programming (e.g., [18]), extreme programming (XP) (e.g., [19]), or active learning teaching pedagogies [20] rather than the integration of modeling. These are examples of potentially missed opportunities for at least raising students' awareness of types of engineering models and even possibly their ability to develop, refine, and/or apply them. In addition to developing more interventions, tools for assessing students' awareness of models and acquirement of modeling skills are critical.
Some types of engineering models that should be explicitly taught are: mathematical, computational, physical, engineering drawing, CAD, financial, and business models. One framework for categorizing the core types of engineering models consists of five categories: 1) physical; 2) graphical/virtual; 3) mathematical; 4) computational; and 5) business/financial [21]. Physical models consist of prototypes. Graphical/virtual models consist of engineering drawings, hand sketches, and CAD models. Mathematical models are models utilizing mathematics (e.g., quantification and formulas). Computational models are computerized models based on mathematical models (e.g., simulations). Some examples of business models are risk assessment and project management systems. Financial models focus on financial aspects of assessing/predicting; typically, a context-specific mathematical model.
Student awareness of a model in this study is defined as a student's ability to recall and describe an engineering model. Based on Bloom's taxonomy [22] recalling and describing a concept, which falls into lower levels of Bloom's taxonomy, is necessary for achieving the higher levels, such as apply, analyze, evaluate, and create. This is supported by Henning and Keune's [23] framework for assessing students' mathematical modeling abilities; the first level focused on their ability to define, describe, and recognize models.

II. RESEARCH PURPOSE AND QUESTIONS
In this study, explicit instruction in engineering modeling was integrated into two first-year engineering courses (one in CAD and one in programming) and studied across two semesters. To evaluate how students' awareness of engineering models changed from the beginning to end of the semester, pre and post surveys were administered in each course. This study aims to answer the following research questions.
1) To what extent did student awareness of model types change after taking introductory engineering courses? 2) To what extent did student awareness of model types differ by course (CAD versus Programming) or semester (Fall versus Spring)?

A. Settings and Participants
A survey was administered in Fall 2019 and Spring 2020 in two required introductory first-year engineering courses at Embry-Riddle Aeronautical University, a medium-sized private university that only served science, technology, engineering, and mathematics (STEM) and business students. The two courses, a CAD course and a Programming course, were redesigned to incorporate modeling concepts throughout the course. Students typically take one of these courses in their first semester at the university and the other in their second; only in rare occurrences can they take them at the same time. The order of the courses is not predetermined and normally depends on their scores on placement tests for Calculus and the purdue spatial visualization test: rotations (PSVT:R). Calculus I is a required co-requisite for the Programming course and students must receive a passing score of the PSVT:R to take the CAD course. Fall 2019 classes were face-to-face. While the semester started face-to-face in Spring 2020, all the students were transitioned online after the COVID-19 pandemic interrupted schools. The majority of students in these courses were White and Male (See Table I). The two courses were taught in small sections of 11 to 27 students per section (Table II). There were 16 to 18 sections of each course in each semester. Six to eight professors taught one to four sections of the CAD course across the two semesters. Six different professors taught one to four sections of the Programming course across the two semesters. There was variation in how each instructor delivered the course content. Both courses had their own set of content, delivery, assignment, and assessment requirements, but each instructor had flexibility in how they implemented these in their sections.
The CAD course teaches 3-D visualization and parametric modeling using a combination of hand sketching and CATIA. Modeling language is a natural part of such a course, in particular the physical and virtual models associated with 3-D printing and CAD. The course design includes a final project to create a virtual model of a self-selected multipart object (e.g., stapler, skateboard, and lamp), allowing students the opportunity to measure and create virtual models of each part, their corresponding assemblies, and all of the appropriate engineering drawings for manufacturing. As part of the revised curriculum, language throughout the course was changed to more clearly emphasize that part and assembly design is modeling (e.g., "model the part" instead of "create the part"). A specific unit was added to the beginning of the CATIA portion of the course to contextualize modeling and how computational models underly the graphical/virtual models they are creating. It also included that CATIA has tools for applying mathematical models to the parts (e.g., computational fluid dynamics and stress/strain analysis).
The Programming course teaches students fundamental programming concepts, such as defining variables, understanding data types, logic statements, repetition, creating functions, array manipulation, string functions, and file input/output. The course also teaches engineering problem solving through a series of steps, including defining the problem, understanding assumptions, developing a solution and algorithm, and testing the solution. A significant portion of the course is an individual final project, where students must develop a program that incorporates all the core coding techniques covered in the course. These were assessed based on a common rubric. As part of the revised curriculum, the course incorporated concepts about developing a mathematical model, then applying this model and further developing it through a computational model. Each instructor had different levels of modeling concepts incorporated in their course beyond one common modeling problem assignment with four submissions.
For an example of these types of problems and the assessment tools used, refer to two previous publications [24], [25].

B. Survey Design and Implementation
A modeling survey was developed to investigate students' awareness of different types of models and how to apply different models to solve engineering problems. The survey prompted students to discuss their concepts of STEM-related models, answer questions about models, and present ideas about types of models they would use to solve two different engineering problems. The full survey is presented in a prior publication [21]. The survey was developed by the research team and reviewed by four additional model experts. Three open-ended questions related to students' awareness of types of engineering models were evaluated in this study.
1) What is a model in STEM fields? 2) List different types of models that you can think of.
3) Describe each different type of model you listed. These questions were designed based on data collected and analyzed in a previous survey at another university.

C. Data Collection
The modeling survey was administered online to all students via Qualtrics. Students' demographic information was also collected in the survey. The presurvey was administered at the beginning of each semester before student exposure to the modeling materials and the post survey was administered at the end of each semester. Table I shows the number of participants who completed pre and post surveys in each course, as well as the response rates. The survey was given to the students as part of an assignment that was graded based on completion. The response rates were much higher in the Programming courses than the CAD courses because the instructors were expected to use the completion as part of the course grade. CAD instructors were asked to voluntarily participate, but completion was not explicitly expected to be a course grade.

D. Data Analysis
Student responses to the three questions were analyzed to highlight the types of engineering models that were identified in students' responses. The students' responses were coded by two researchers based on an established coding scheme (shown in Table III). The intercoder reliability for the two researchers was more than 80% across the five coding categories; for more details about development and application of the coding scheme refer to our previous publication about this process [21].
The resulting codes were then quantified to determine the number of different types of models that students included in their responses (i.e., physical, graphical/virtual, mathematical, computational, or business/financial each counted as one type of model).
To determine whether the number of model types identified by each student changed from the beginning to the end of the semester (pre versus post survey), and whether there were differences across the two courses (CAD versus Programming) or semesters (Fall versus Spring), a mixed-design ANOVA was conducted. The survey (pre/post) was a within-subjects factor, while course and semester were between-subjects factors.

IV. RESULTS
Out of 867 students, 602 students completed both the pre and post survey in a given course/semester. Fig. 1 shows the mean number of models described by students on pre and post surveys in each course/semester. Overall, students identified more models at the end of the semester compared to the beginning of the semester (p < 0.001). Across all courses and semesters, the mean number of models increased from 1.73 to 2.18 or roughly half (0.45) of one model. The highest gains were in the Fall Programming course. Students also identified more models in Spring 2020 compared to Fall 2019 (p < 0.001). There was no significant difference across the two courses (p = 0.624). Improvements were seen in both courses, and students appeared to remember the model types between the Fall and Spring semesters (Fall post mean = 1.99 and Spring pre mean = 1.94) with continued increased awareness in the Spring (Spring post mean = 2.38).
Since the mean number of models increased in all sections from beginning to the end of the semester, we further investigated the types of models to understand which models were associated with increased or decreased numbers from pre to post-surveys (Fig. 2).
The first group of students, as shown in Fig. 2(a), were enrolled in the CAD course first, in the Fall, and the Programming course in the next Spring semester. These students mostly identified physical and graphical models in both pre and post surveys. Identification of physical and graphical models decreased in the consequent semester, in which students were enrolled in the Programming course. For the other three models, mathematical, computational, and financial, there was a gradual increase from beginning of the Fall semester to the end of the Spring semester. As students progressed through the academic year, the percentage of students identifying these three model types increased.
The second group of students, as shown in Fig. 2(b), were enrolled first in the Programming course in the Fall semester and CAD course during the following Spring. In the Fall semester, the number of students who identified physical and graphical models did not change substantially from the beginning to the end of the semester. However, more students identified mathematical and computational models at the end of the semester compared to the beginning of the semester. Unlike the first group (who took CAD in the first semester), these students had a more evenly distributed identification of different model types at the end of the Fall semester; they identified all model types (except financial) at similar rates. In the Spring semester-at the beginning of the CAD course, graphical models were identified more than other models followed by physical models. At the end of the semester, the number of students who identified each type of model increased for all, except graphical models.
At the end of the academic year (in the Spring 2020 postsurvey) more students identified mathematical models than any other model type for both groups. In contrast, financial models were identified the least compared to other models. Overall, the first group [ Fig. 2(a)] had higher post results for students identifying computational and financial models. The second group [ Fig. 2(b)] had higher post results for students identifying physical, graphical, and mathematical models. Based on the data, the Programming course appeared to have the greatest impact in broadening students' awareness of different types of models.

V. DISCUSSION
Many upper-level undergraduate and graduate engineering courses require students to apply and develop various types of models. Sometimes there is an expectation to do this without an established foundation. To ensure students are more prepared to apply and build various types of models, it is imperative to establish a common language and awareness of model types as early as possible in the engineering curriculum. There has been a significant amount of research that has shown first-year engineering students typically only show awareness of prototypical or physical models. This is a very limited viewpoint of engineering models. It is critical that we broaden engineering students' awareness of more model types and their purposes (e.g., demonstrate and test concepts/designs, interpret data, and make predictions) to ensure their success in their engineering education and careers.
This study takes a first step at assessing what first-year students "know" about types of engineering models. To consider something as known according to Bloom's taxonomy, someone must first remember a piece of information then demonstrate understanding [22]. A demonstration of remembering can be as simple as quoting an idea, but students were required to go beyond this. Within the administered Modeling Survey tool, students had to communicate their ideas about model types, along with descriptions and their purposes, without referring to resources; this aligns with the understanding level of Bloom's taxonomy [22]. In this article, we looked specifically about awareness of model types, but the survey included questions that assess other aspects of modeling (purpose of models and how models can be applied in specific problems) that will be evaluated in future studies.
The survey has broad application for engineering instructors, professors, and administrators seeking a tool to assess their students' perceptions of engineering models. For example, instructors implementing MEAs, emphasizing modeling language, or changing a project to utilize modeling in their courses can use this survey to assess how their students' understandings of model definitions, purposes, and applications change.
This study only focused on the first three modeling questions of the survey that were about students' definitional knowledge, but the other questions in the survey assess students' ideas about how models are used and how to approach two different modeling problems. Rodgers et al. [21] provided the full survey along with discussion about the implementation of the survey and some initial analysis of the data. Additionally, the authors developed a MATLAB tool for automated analysis of the open-ended responses [26].
There were some differences in the types of models students identified most frequently depending on which course they were in (CAD versus Programming). While the presurvey was conducted at the beginning of the semester before teaching any modeling content, there appeared to be a bias toward the students' anticipation of the course content in the survey responses, since the model types that are more closely related to the course were more commonly identified even in their presurvey responses (i.e., physical and graphical models were more frequently identified in CAD courses compared to Programming, while mathematical and computational models were more frequently identified in Programming courses compared to CAD courses). The students' ideas of what they should learn appeared to influence their presurvey responses.
Although the students in two different groups started with different profiles of models identified in the first semester, they ended up with a similar profile at the end of the academic year (after completing both courses). At the beginning of the Fall semester, all the students referred to physical and graphical models the most-although this was much more prevalent in the CAD course (presurvey). The students in the Programming course also identified other model types more than the students in the CAD course. The biggest difference between the two groups was seen at the end of the Fall semester. The results for the Programming course presented a more even distribution across four of the five types of models versus the CAD course which still saw a more skewed distribution heavily favoring physical and graphical models. At the end of the Spring semester, mathematical models were identified the most followed by physical, graphical, and computational models. Financial models were identified the least among all the models since this was not emphasized in the courses. The students were required to build mathematical models to calculate costs and analyze stock market data in the Programming course, but there should have been more explicit language embedded throughout these problems about financial models. In addition, neither course emphasized risk assessment nor project management (business-related models). These results may imply the order of CAD and Programming courses in first-year engineering does not have a significant impact on students' awareness of engineering model types.
Overall, there was an increase in the number of models students described over time (from the beginning to end of the semester and Fall to Spring) regardless of which course they were in. This shows some success in achieving desired learning outcomes (that students are able to identify and describe different types of models used in STEM) through these intentional, explicit modeling interventions. Additionally, since the number of models identified by students in the Spring was higher than the Fall semester, students likely retained some of the knowledge they gained in the first course.

A. Limitations and Future Work
This study had some limitations. First, the focus on the number and type of models is not indicative of whether students understand how these models are created and applied, only their awareness of the model type is assessed. In the statistical analysis, it was assumed that students in different courses were independent groups. However, there was some crossover in students who took both courses (18 students, about 2%, took both courses in either Fall 2019 or Spring 202). Planned future work will assess students longitudinally over multiple courses to see how their understanding of engineering models improves over time.
Another limitation of this study is CAD students' participation was voluntary; it was encouraged but not graded. This likely contributed to the lower response rate in the CAD courses compared to the Programming courses. In voluntarybased survey participation, there is a possibility that the students who were more self-confident and performed better in the course be more likely to respond to the survey.
There were many struggles across the nation in Spring 2020 with universities transitioning online in the matter of days or weeks. The impact of COVID-19 may have impacted the results, but this was not investigated. There were fewer responses in the Spring 2020 post-survey, which may have been an impact of the transition to fully online courses. As far as the content of the courses, the CAD course was designed to be a fully face to face course, but the bulk of the modeling intervention was at the front-end of the semester when the instruction was face to face. The Programming course was designed as a hybrid course with lecture materials already created as online videos, so the transition online did not make a significant change to materials delivered. Similarly, the in-class activity materials by the second half of the semester required primarily coding on computers and all required written components were already completed by this point in the semester. All in-class activities and homework assignments maintained the same building and applying models-no assignments were modified.
COVID-19 might have exacerbated the impact of some of the out-of-class factors on student learning. Follow-up studies can investigate this impact on students' understanding of the engineering models.

VI. CONCLUSION
The analysis of students' responses about models in STEM shown the students' awareness of different types of models improved over time (beginning to end of semester and Fall to Spring). Students' awareness of model types improved after taking either or both the CAD or/and Programming courses. Overall, the highest demonstrated awareness of types of models was seen in the post-surveys in the second semester (Spring 2020) for both analyzed groups. Although at the beginning of the academic year the students in two different groups started with somewhat different profiles in identifying the models, they ended up with a similar profile at the end of the year. Both courses had a positive impact on students. However, the Programming course seemed to have the greatest impact in helping students diversify their awareness of different types of models.
ACKNOWLEDGMENT Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation.