Utilizing Google Cardboard Virtual Reality for Visualization in Multivariable Calculus

In this study, we have developed a webXR tool that helps students visualise 3D graphs from functions of two variables through the use of simple, practical and cost-effective Google Cardboard for use in the classroom. Further, we have assessed Google Cardboard’s usability as a content delivery system in a mid-sized multivariable calculus class with 36 students, and 40 other students in another class as the control group. We also attempt to assess if Google Cardboard is better than PowerPoint slides, shown on flat screen computers, in terms of students’ attitudes and engagement towards the teaching and learning of multivariable calculus. Our results suggest that Google Cardboard functions better than PowerPoint slides when encouraging students’ attitudes and engagement towards learning multivariable calculus. At the same time, Google Cardboard as a content delivery system does not appear to differ from PowerPoint slides in terms of its usability.

ABSTRACT In this study, we have developed a webXR tool that helps students visualise 3D graphs from functions of two variables through the use of simple, practical and cost-effective Google Cardboard for use in the classroom. Further, we have assessed Google Cardboard's usability as a content delivery system in a mid-sized multivariable calculus class with 36 students, and 40 other students in another class as the control group. We also attempt to assess if Google Cardboard is better than PowerPoint slides, shown on flat screen computers, in terms of students' attitudes and engagement towards the teaching and learning of multivariable calculus. Our results suggest that Google Cardboard functions better than PowerPoint slides when encouraging students' attitudes and engagement towards learning multivariable calculus. At the same time, Google Cardboard as a content delivery system does not appear to differ from PowerPoint slides in terms of its usability.
INDEX TERMS Virtual reality, multivariable calculus, visualisation, Google Cardboard.

I. INTRODUCTION
A common problem faced by students in studying advanced calculus is in developing a graphical intuition for the many equations that they learn. Graphs are usually used as a visualisation tool as part of teaching and for developing such an intuition when students learn about single-variable functions. As graphs for such functions are presented in two dimensions, they are easily drawn on paper and calculators. This allows students to develop the required graphical intuition for singlevariable functions. However, graphs for two-variable functions, which students learn about in multivariable calculus, are presented in three dimensions. It is challenging to develop a graphical intuition for 3-dimensional graphs with still or flat images [1], [2]. While it is possible to draw 3-dimensional graphs on the computer, they are after all still being illustrated on a flat screen.
The associate editor coordinating the review of this manuscript and approving it for publication was John Mitchell .
It is worth noting that extended reality (XR) tools allow students to manipulate text, images, and other media within a virtual environment. A graph created in such an environment can be viewed from different angles and allow students to zoom in and out easily. XR tools have been used in various domains [3], including physics education [4] and industrial environments [5], [6]. Over time, computing power has increased, while the cost of XR tools has decreased. These changes mean that the resistance to development and cost to schools has also decreased. These changes coincide with the increased call for blended learning in Singapore [7], [8], [9], [10], [11] and many other countries globally.
Early research into the potential use of XR for teaching and learning multivariable calculus dates back to 2006 [12]. In their work, Orozco et al. developed an AR tool that allowed the instructor of the class to control the functions given as input that the student received on their personal learning devices. While the students may be perceived to be passive observers, the teacher's guided inputs meant that the visualisation was personalised for the contents of the class even though the graphs could not be manipulated by the student. Boggess and Harding [13] created a 3D visualisation tool on a desktop that translated into a haptic device for the student to manipulate. With advances in technology, both in wireless communication and viewing headsets, further research that delve into XR visualisation emerged [14], [15], [16], [17], [18].
Quintero et al. [14] and Ramirez et al. [15] both explore an AR tool, developed from the ground up across the span of 3 years, to teach the graphs of one real variable, solids of revolutions, and functions of two real variables. Similarly, [16], [17] explore the AVRAM (Remote Virtual Environments for the Learning of Mathematics) tool for augmented reality visualisation of graphs. Reference [18] details the creation of an AR tool using design-based principles, iteratively creating their prototype. However, despite the wide use of AR, there exists a research gap in XR research as there are limited studies that have dedicated to the use of VR for visualisation.
Google Cardboard is a very easy to use, practical and affordable virtual reality view-finder platform that uses compatible VR apps on a smartphone to create virtual environments. Google announced in November 2019 that it would open-source the platform's software development kit (SDK). The Google Store discontinued selling Cardboard viewers in March 2021 [19]. However, third-party companies can continue to sell compatible viewers as the Cardboard viewer specification is open source. Figure 1 depicts a typical third-party Google Cardboard with built-in lenses. By following a series of instructions, the Google Cardboard can be easily unfolded into a viewfinder as seen in Figure 2. It is worth noting that the quality of the lenses will make a huge difference to the user's experience and care should be taken to procure only cardboard with high-quality lenses through a series of trial-and-error with different vendors. The smartphone is placed on the back of a solid cardboard headset [20]. Figure 3 depicts a user with a Google Cardboard mounted on his head.  In our previous work [21], we investigated the effectiveness of using the Oculus Rift for visualization purposes. The results suggest that using VR can improve student performance in identifying the sign of partial derivatives to a certain extent, and has the potential for development into a future-ready smart classroom. Smartphones are becoming more ubiquitous in society [22]. The affordability of Google Cardboards coupled with the access to smartphones has tremendous potential to be a working solution to bring VR into the classroom at an accelerated pace [23], [24]. Therefore, this study seeks to evaluate the usefulness of VOLUME 11, 2023 Google Cardboard as an educational tool to create the desired virtual environment for the 3-dimensional graphs. Immersive and interactive learning experiences have been shown to positively impact students' attitudes and engagement in various educational contexts [25]. Given that Google Cardboard offers an immersive and interactive environment, it is reasonable to hypothesize that it could similarly enhance students' attitudes and engagement in the context of multivariable calculus. In this study, we employ a teaching approach that integrates technology-enhanced active and experiential learning. Students engage in active learning through the manipulation of algebraic expressions in multivariable calculus, and the use of VR technology for example to visualize contour plots and 3D graphs, providing an immersive, experiential learning environment that enhances spatial reasoning and visual learning. The students are also guided to tackle complex, abstract problems, fostering higher-order thinking skills. A sample lesson plan is included in the appendix. As part of our study, students are first guided on the use of the Google Cardboard for visualisation purposes in the classroom (as a companion tool to the class notes/examples) in the first 3-4 weeks, before they are asked to respond to a survey. This study specifically assesses the usability of Google Cardboard as a content delivery system to teach multivariable calculus. Google Cardboard will be compared to commonly used PowerPoint slides, shown on a flat screen as a medium of content delivery, in determining whether students' attitudes and engagement towards learning multivariable calculus can be enhanced.
In summary, this study has two aims: 1) To assess Google Cardboard's usability as a content delivery system in a multivariable calculus classroom with about 40 students, with a control group of another 40 students. 2) To assess if Google Cardboard is better than Power-Point slides, shown on flat screen computers, in terms of students' attitudes and engagement towards the teaching and learning of multivariable calculus. The alternative hypotheses are 1) Google Cardboard will work as a content delivery system, and 2) Google Cardboard is better than PowerPoint slides in terms of students' attitudes and engagement toward the learning of multi-variable calculus. The null hypotheses are that 1') Google Cardboard will not work as a content delivery system and that 2') Google Cardboard is no better than PowerPoint slides in terms of students' attitudes and engagement towards the learning of multi-variable calculus.

II. EDUCATIONAL RESEARCH INFRASTRUCTURE A. VIEWFINDER: GOOGLE CARDBOARD
Google Cardboard works by projecting dual screens from compatible smartphones into a single viewing experience when users bring the cardboard closer to their eyes. It is compatible with most modern smartphones. Most versions of Google Cardboard can only fit smartphones with screen sizes from 4.7 to 5.5 inches.
To develop apps for Google Cardboard, one can make use of the Google Cardboard SDK for popular game engines, such as Unity. The SDK would bootstrap the development as it comes with features commonly used in VR scenes, such as object selection or teleporting. However, apps developed using this method would need to be published in app stores, making them less universal since some smartphones will no longer be compatible to view the apps.
Another method of creating apps for Google Cardboard, which was later adapted in this work, is by utilizing webXR environment. Using this method, VR apps are developed to be published over the web, thus removing the necessity for them to be published in app stores. In order to access the app on their smartphones, users can go to a specific website that hosts the app and have the app played on the go without the need to install anything on their phones. This provides quick bug fixes and support for as many types of operating systems and smartphones as possible.

B. WEBXR: VIRTUAL REALITY APP
We have developed the app using Three.js, a Javascript library to specialise in creating, displaying, and manipulating 3D graphics over the web. With webXR as the chosen output, several considerations were made when selecting this method. First, there have been several math graphing libraries in Javascript that one can tap into while developing the app, thus reducing the time needed to develop codes from scratch. Second, Three.js library has been established over the years with good support and large communities, ensuring its stability and continuous improvements to have better capabilities, such as more compatibility with other Javascript libraries, smoother manipulation of the graphics using GUI (graphical user interface), dynamic texts, and expansions to webXR. Next, a more advanced Javascript library, Math.js, was used to parse advanced math expressions and to simplify the mathematical calculations in the code since native Javascript can only parse basic math expressions. This is coupled with MathBox, an open-source computational graphing library to create 2D and 3D graphs using math equations; thus, we can modify the codes to suit our needs for our VR app. Finally, using dat.gui, we can create a simple and lightweight user interface to manipulate/control variables in Javascript that is also compatible with Three.js.
Subsequently, publishing the codes over the web would ensure that the app can continuously be fine-tuned and replicated for further research. We chose to utilise webXR because it is known to be versatile, can be played in compatible browsers and various VR headsets, easy-to-use as no installation is required by users. For these reasons, it also allows for fast development, hence suitable for prototyping. However, we note that not all browsers can play webXR as it is meant to be deployed over the web; currently, it is an exclusively JavaScript API, and Javascript may not be suitable to code/create more complex math graphics. For the purpose of our study, we identify webXR as sufficiently suitable for our context. Before the app can be used, it must be hosted on the web that provides an SSL certificate. We chose to host them via Github as it is free and already comes with free SSL certificates. By uploading the codes to GitHub repository, we are able to then create a web page with the github.io address which is sufficient for our use with the webXR main menu illustrated in Figure 4 with the following topics which were developed into VR modules: 1) Contour plots 2) Partial derivatives 3) Directional derivatives 4) Multivariable chain rule 5) Critical points, Global extrema and the extreme value theorem These VR modules focus on functions with multiple variables. For instance, contour plots help students visualize functions in three dimensions in a two-dimensional format. The directional derivatives tool is useful for demonstrating to students the rate at which a function changes in a specific direction. Similarly, partial derivatives showcase how a function changes with respect to one of its variables while holding the other variables constant. In addition, critical points, as well as the concepts of maximum, minimum, and saddle points, are also illustrated. The course syllabus and specific learning objectives are given in the appendix. Specific examples from our VR modules can be found at this link. During the app development over the course of 9 months, the app has been continuously tested with PhD students and staff using iPhones and various Android phones. Following the feedback from each test via a survey form and interviews, the app was fine-tuned to increase its graphing accuracy and create better user interactions through a more user-friendly interface and presentations. All the app versions share the same base where the user inputs an equation and the corresponding graph is displayed. What differs between each version of the app is the special tools that are used specifically for each lesson. For example, the ''Critical Points'' version of the app contains a function to detect the maxima and minima of the graph that is currently displayed. Each lesson has its specified toolbox useful for the lesson so that students are not distracted from the other capabilities that the app might provide for an earlier or later stage of the course material. Next, we show an app example in Figure 5 (but not yet in VR mode) and Figure 6 shows how the app will look like when it is in VR mode (one display for each eye). Upon entering the app, the user ensures that the graph displayed is correct by entering the multivariable function required for the lesson before enabling VR mode. At this point, the user takes their phone and inserts it into the Google Cardboard set to view the graph and be immersed in the environment.

III. EXPERIMENTAL SETUP
We conducted a quasi-experiment; a posttest-only nonequivalent group design. All students taking multivariate calculus had been randomly assigned to one of the ten classes. Two classes were randomly chosen for the experiment, with one class of 36 students as the intervention group, and another class of 40 students as the control group. Both groups were given lecture PowerPoint slides that included 2D visualisations of 3D graphs. The intervention group was also given the VR intervention through the VR app designed by our research group, delivered on Google Cardboard. The study lasted 3 weeks with the same instructor for both classes, after which a posttest survey of 69 Likert scale items and 2 open-text items were administered. The instructor informed all students that the survey is anonymous and voluntary; by completing and doing whatever the respondent should do with the completed survey, they are voluntarily consenting to participate in the study. The instructor also explained to the students that the survey results will be used as part of an educational research.
In Figure 7, we can see students from the intervention group trying to set up the Google Cardboard for visualising certain examples in the class. Due to the national and institutional restrictions put in place on any in-person teaching during the COVID-19 pandemic (2021-2022) during which the data for the current study were collected, all students/instructors were required to wear a mask. Nevertheless, students were still very engaged in trying out the Google Cardboard. Some students can be seen in Figure 7 discussing their learning experiences with other classmates. VR engineers (as illustrated in Figure 8) were also deployed to provide assistance to students who faced issues setting up the Google Cardboard with their smartphones. Based on our experiences from this study, it is important to have some VR engineers on standby to assist with issues faced by the students in setting up their Google Cardboards. The survey questions that we utilised from [26], [27], [28], and [29], measured the constructs of ''attitude towards teaching and learning of multi-variable calculus'', ''engagement with multi-variable calculus'', and ''usability of the application''. Additional constructs could be included to compare the proposed approach of using VR with traditional methods. However, upon examining related literature, it appears that other constructs are either subcategories or related to the constructs of attitude, engagement, and usability [26], [27], [28]. Studies have found that students' attitudes toward mathematics strongly influence their learning ability and achievements [30], [31], [32]. Additionally, a wide range of research has shown that student engagement in mathematics is crucial for pursuing higher studies in the subject and achieving good grades [33], [34]. Therefore, we consider attitude and engagement to be important measures for comparing the two methodologies (VR and PowerPoint). Furthermore, the construct of usability is also included to compare the ease of use of VR and PowerPoint teaching techniques. The attitude construct is composed of 4 sub-constructs, self-confidence, value, enjoyment, and motivation. The engagement construct also had 4 sub-constructs, behavioral engagement, emotional engagement, cognitive engagement, and social engagement. The usability construct was not composed of any subconstructs. The 5-point Likert scale was identical for all items with 1 = strong disagree, 3 = neutral, and 5 = strongly agree. The survey was created with reference to individual surveys that measured each of the above constructs separately [26], [27], [28], [29]. However, modifications were made to the questions regarding usability of the application to make it applicable to both PowerPoint slides and VR on Google Cardboard.
The analyses used to test the hypotheses were to take the mean of the means of each relevant sub-construct to form the overall measurements for the attitude and engagement constructs. The mean was taken for the usability construct. Four statistical tests were conducted using the results of the survey to test the hypotheses mentioned in Section I. First, Shapiro-Wilk tests, which tests for normality, were conducted for each measurement. Second, a one-tailed one-sampled t-test was conducted, checking each group against a neutral response of 3 to see if the measurement was significantly different 75402 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.
from the mean of a Likert scale ranging from 1 to 5. Third, an independent samples t-test was performed to compare both groups. Finally, effect size was measured using Cohen's d. Figure 9 shows an example of the survey results for the first question related to their experience with the visualization tool that they have been encountered with. The other survey questions are compiled as part of the appendix.

IV. RESULTS
In this section, we discuss the results obtained by performing various statistical tests. We first compare the mean value of each construct for intervention and controlled group with the neutral response. Referring to Table 1, the mean value of the attitude construct for the intervention group was significantly greater than the neutral response of 3 confirmed by the significance of the p-value. On the contrary, in the case of control group, the mean is not different from the neutral response as p-value exceeds the threshold of 0.05. Google Cardboard has helped to enhance students' attitude toward these mathematical topics. Furthermore, the mean values of the engagement construct for both intervention and control groups are significantly higher than the neutral response of 3 captured by their respective p-values. Thus, student engagement in VR classes is not lesser than the power-point classes. For usability construct, Shapiro-Wilk test indicates the non-normality for the intervention group, hence despite being p < 0.05, we refrain from making any judgement. Finally, the mean value of the usability construct for the control group is not different with p > 0.05.
Next, we compare the mean of all the constructs for control and the intervention group by performing an independent sample t-test. For both attitude and engagement construct, the mean of the intervention group is significantly greater than the mean of the control group as appear in Table 2. The above observations provide some form of evidence that Google Cardboard may be better than traditional PowerPoint in terms of enhancing attitude and engagement in a mathematics class. As mentioned earlier, the usability construct is non-normal, so we refrain from making any judgement. However, we have also performed Wilcoxon Rank Sum test as a substitute and found that google cardboard and power-point are not signficantly different with regard to the usability.
We would also like to draw attention to the work of Geoff Norman [35]. Norman's research provides compelling evidence that parametric methods, such as t-tests, are robust against violations of normality assumptions. This robustness has been demonstrated in numerous studies since the 1930s, suggesting that even when data may not be normally distributed, the use of t-tests does not compromise the reliability of the results.

V. DISCUSSION
The medium effect sizes suggest that Google Cardboard does work better than PowerPoint slides when encouraging students' attitudes and engagement towards learning multivariable calculus. This effect size is smaller for the usability construct (d = 0.460), and might be negligible at the lower bound of the confidence interval. Furthermore, addressing our first research question, Google Cardboard as a content delivery system seems to not differ from PowerPoint slides in terms of usability.
The increase in attitudes and engagement despite the same usability could be because of the novelty of using VR in a classroom or because of differences in student perception. When asked to list the strengths or benefits of using Google Cardboard VR as a learning device for multivariable calculus, students in the intervention group remarked 'I am able to see [visualise] the graph, It is interactive and able to see complicated graph[s]', 'Helps to visualise the graphs better and overall makes the topic easier to understand', and 'VR helps me visualise many of the functions efficiently and easier, compared to when I see them on the whiteboard or on the slides. VR is a fun approach to learning math and this approach should be [encouraged]. I hope we can have more VR sessions now that we are familiar with the use of Google Cardboard.'. These students' feedback on the benefits of Google Cardboard has pointed to the potential usability of Google Cardboard as a learning tool; this motivates future work whereby we can consider implementing the study across all classes (typically 500 undergraduates) for further analysis. Familiarity with new technology is one of the hurdles that need to be tackled. Towards the end of the study period, students were starting to be familiarised with the tool, which could explain why the usability did not differ from that of PowerPoint slides. On inspection of the students' performance in their exams, we observe that the students in the intervention group performed better than the students in the control group for multi-variable calculus questions involving visualisation: the intervention group scored a mean of 68.7% while the control group scored a mean of 59.3% (both groups have the same instructors), with Cohen's d = 0.285, indicating that the effect size is small [36].
In the study phase, participants in the intervention group were also asked to provide feedback on the weaknesses of using VR, which some students feedback 'Due to different OS [operating systems] experience might be different.', and 'Hard to get used to, sometimes the software doesn't work properly on my phone, could take up a lot of time.'. The compatibility across multiple operating systems and versions could pose a potential hindrance to students' experience of the VR learning tool. Students' experiences with usability were the main weaknesses provided by certain students in the intervention group due to operating systems. As the aim is to deliver VR content in a cost-efficient manner, the use of smartphones remains our key focus in this project. Ensuring software compatibility across operating systems motivates future work.
Lastly, on the practical use of VR on the choice of topic, some students remarked in the qualitative survey that the use of VR might not be 'very useful'. For example, students felt that it 'is okay to [view graphs] from [the] laptop screen using the [multivariable calculus VR] website which is very clear.', 'It seems that using the same website on a computer without using the [Google Cardboard] works just as well in visualising the graph.', and 'I prefer not to use the [Google Cardboard] and just look at the graph and rotate it on the 2D screen to visualise it.'. There is a possibility that students assumed that they somewhat understand how the graphs looked like because they were provided with the 3D rendering of the graphs in the handout (instead of having to derive the shape for themselves). While we selected a subset of topics under the multivariable calculus syllabus, they may not be the topics where VR's full potential is realised. Thus, further investigation is required to study which topics under which VR may prove to be more useful as a teaching and learning tool.
We have some suggestions for the future iterations of this app: to ensure that it is compatible with most operating systems and smart device firmwares of the past 3 years; and to develop Google Cardboard VR tools for more advanced topics which may require greater spatial and dimensional awareness in the concept-formation stage of learning. To ensure better results that are independent of the novelty effect, our data collection period should require a longer experimental duration with more participants and a standardised multivariable calculus test to explore whether the increased engagement leads to better test results. We also plan to delve deeper into the specific design elements and pedagogical strategies that contribute to the enhancement of students' attitudes and engagement when using Google Cardboard Virtual Reality in the context of multivariable calculus. By incorporating a more in-depth investigation of the immersive and interactive aspects of this technology, we aim to provide a clearer understanding of its potential benefits and impact on the learning experience. This additional research will not only strengthen our current findings but also contribute to the growing body 75404 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.
of knowledge on effective virtual reality integration in educational settings.

VI. CONCLUSION
In summary, we have developed a webXR tool that enables students to visualise 3D graphs from functions of two variables through the use of Google Cardboard. The motivation for this study stems from the fact that students have difficulty developing a graphical intuition for the many equations that they learn, especially in the visualisation of 3D graphs from functions of two variables. The situation is made worse as 3D graphs are usually illustrated on a flat screen (2D) in the classroom. Second, Google Cardboard holds tremendous potential in being tapped as a visualisation tool. While VR is not a new technology at this point, it has not been widely adopted in mathematics classrooms despite being easy to use. In our study, we have attempted to assess Google Cardboard's usability as a content delivery system in a mid-sized multivariable calculus class with 36 students, and 40 other students in another class as the control group. Furthermore, our research findings suggest that Google Cardboard functions better than PowerPoint slides when in encouraging students' attitudes and engagement towards learning multivariable calculus. It is also worth noting that Google Cardboard as a content delivery system does not appear to differ from Pow-erPoint slides in terms of its usability. Our study has clearly illustrated the practical and cost-effective aspects of Google Cardboard and it is our hope that this study will inspire and encourage other mathematics instructors to adopt VR tools (eg. Google Cardboard) in their lessons if resources permit.

APPENDIX
The course syllabus, complete survey questions, and a sample lesson plan can be found at this link.