Socially Distanced: Have user evaluation methods for Immersive Technologies changed during the COVID-19 pandemic?

Since the emergence of COVID-19 in late 2019, there has been a significant disturbance in human-to-human interaction that has changed the way we conduct user studies in the field of Human-Computer Interaction (HCI), especially for extended (augmented, mixed, and virtual) reality (XR). To uncover how XR research has adapted throughout the pandemic, this paper presents a review of user study methodology adaptations from a corpus of 951 papers. This corpus of papers covers CORE 2021 A* published conference submissions, from Q2 2020 through to Q1 2021 (IEEE ISMAR, ACM CHI, IEEE VR). The review highlights how methodologies were changed and reported; sparking discussions surrounding how methods should be conveyed and to what extent research should be contextualised, by drawing on external topical factors such as COVID-19, to maximise usefulness and perspective for future studies. We provide a set of initial guidelines based on our findings, posing key considerations for researchers when reporting on user studies during uncertain and unprecedented times.