Toward VR in VR: Assessing Engagement and Social Interaction in a Virtual Conference

The pandemic brought about an unprecedented number of virtual conferences, given the heavy restrictions on travel to in-person meetings. Despite all the advances in technology, people still complain about virtual events. There is Zoom fatigue, confinement malaise, and a longing for personal social interactions. This paper discusses our experience organizing the IEEE Virtual Reality Conference (IEEE VR) as a virtual event. IEEE VR was a success with 1200+ registered paying participants, dozens of workshops and tutorials, and hundreds of technical papers. We used (1) a virtual environment platform, together with (2) discussion tools and (3) videoconferencing/broadcast/online tools to further provide effective social interaction and increase engagement. In this paper, we explore the synergies between virtual environments and other online tools and assess user engagement by analyzing the messages exchanged between participants across different genders and geographical regions. To this end, we apply diverse engagement metrics for online conferences. Our analysis shows that these metrics have the potential to highlight engagement, diversity, and inclusion by combining textual messages, participant geographic and gender information, communities of participants, and visitation patterns in a virtual environment. Drawing on our results and experiences, we propose guidelines for organizing technical virtual events to increase diversity and social interaction.


I. INTRODUCTION
For the past several decades, international academic conferences have been the primary modality for disseminating and The associate editor coordinating the review of this manuscript and approving it for publication was Lei Wei .
presenting new research in computing. The COVID-19 pandemic drastically disrupted this model, canceling in-person gatherings for subsequent years. The Institute of Electrical and Electronics Engineers (IEEE) Conference on Virtual Reality and 3D User Interfaces (IEEE VR), held in March 2020, was among the first such conferences to have the in-person gathering canceled and subsequently was one of the first to shift to an all-virtual online gathering [1]. As a result of this early adoption, IEEE VR 2021, held a year later, was one of the first conferences able to revisit initial concepts and technologies. This paper evaluates how the conference evolved to better support the needs of the community, the success of which is most evident by the subsequent re-use of nearly all elements for the 2022 iteration of the conference.
Though the IEEE VR 2020 conference is widely seen as successful [1], under the difficult circumstances, the organizers of IEEE VR 2021 had more time to prepare and had the benefit of prior experiences from the same conference and many subsequent conferences in various areas. As with IEEE VR 2020, the 2021 iteration took place during a significant increase in reported infections of the pandemic, the so-called ''second wave'' in late March/early April 2021. 1 Due to travel concerns in January 2021, the planned physical conference site had to be discarded, opting for a universitybased physical/virtual hybrid event instead. In early February 2021, the organizers decided to hold the conference in a purely virtual format. Following early experiences with IEEE VR 2020, 2 ACM ISS 2020 3 which took place virtually in November 2020, as well as ACM UIST 2020 and ISMAR 2020, IEEE VR 2021 had to prepare for a purely virtual event during a severe lockdown. During this transition, nearly all of the programmatic elements that were originally used were reconsidered and revised. As such, whereas IEEE VR 2020 could be classified as a translation of an existing, inperson model, IEEE VR 2021 was designed to transform the existing model to meet the needs of the existing community, to be as inclusive as possible to newcomers, and to continue to innovate in the delivery of an online event.
Our major changes focused on the program's structure, the online collaboration tools, and the virtual reality environment. For example, the IEEE VR 2020 conference featured talks with a length typical to in-person conferences (20+ minutes), multi-tracked into a typical workday of a particular timezone (Atlanta, GA, USA). However, the online format allowed for a longer, thinner program with shorter talks, allowing for more convenient access for presenters at disparate geographic locations. The conference also switched chat platforms from the pair Slack/Slido to the Discord usage for all messaging. A major consideration was that Discord did not impose significant limitations on the free version, unlike Slack. In addition, the community raised concerns about the number of different platforms used for IEEE VR 2020, and organizers had prior good feedback on their experience with Discord during ACM ISS 2020. Also, based on feedback from the community, the Mozilla Hubs virtual environment used in IEEE VR 2020 was replaced with Virbela 4 in 2021, which was used as the primary platform for ISMAR 2020 and received positive feedback. Virbela had many of the same features as Hubs but allowed more opportunities to socialize with support for more participants. These and many further details are outlined in Section III.
We also report on several broad engagement metrics that point to the utility of these innovations. For example, we considered the number of messages sent in various chat channels associated with event types (e.g., Keynote Sessions) relative to the potential population of registered members. In Virbela, we could capture several useful statistics, such as the unique visitors in each region each day. Finally, we conducted an anonymous survey of participants to gain qualitative feedback on the success of each tool. These are reported in Sections IV and VI.
Last, the paper closes with a set of guidelines and recommendations in Section VII that may aid future conference organizers, especially those who want to include a virtual social environment in the event.

II. RELATED WORK
The past few years have witnessed a lot of conferences and scientific gatherings moving from in-person exchanges to virtual events motivated in part by environmental considerations and foremost by the pandemic that severely curtailed most forms of travel. While this has entailed lower travel expenses, reduced carbon footprint, and readily accessible content, many pitfalls emerged, from decreased social networking, fewer impromptu discussions and resultant collaborations, decreased interactions and engagement and missing social elements from conference attendance. Woodruff et al. [2] question how the transition to virtual settings has impacted interest in medical conferences and propose hybrid approaches. Indeed, common patterns emerge across many different fields. Kenrick [3] discusses visitor engagement during virtual events at an Academic Art Museum. Remmel [4] reports a Nature poll among hundreds of attendees to conclude that while online conferences have brought significant benefits, blending those with in-person meetings will bring significant challenges. Virtual meetings have yielded lessons learned from a broad spectrum of research areas, ranging from astrophysics [5] to child education and orthopedics, among many others [6], [7]. Rundle et al. [8] discuss logistics, troubleshooting, and conversion of an NIH-funded conference from an in-person to a virtual conference. Gishora et al. [9] propose guidelines to organize a virtual event based on best practices. Rubinger [10] presents a review of virtual meetings and conferences and derives best practices in the field of Orthopaedics.
A growing number of organizers have experimented with conferences and events in 3D virtual worlds or desktop VR game-like environments. MacIntyre & Outlaw [11] discuss the social experiences using Mozilla Hubs during IEEE VR 2020. Ahn et al. [1] also provide a survey report around media choice (Twitch, Hubs, and Slack) and appropriateness at IEEE VR 2020. Their study indicated that for the participants who experienced the Hubs platform, perceived social presence was highest when compared with other platforms. Jauhiainen [12] discusses Entrepreneurship and Innovation Events during COVID-19 using Virbela, an immersive platform, for the SHIFT Event in October 2020. The study revealed that more reserved and less experienced 3D virtual platform users were more reluctant to interact, and participants' opinions differed regarding the 3D digital platform as a trustful and secure site. Mulders & Zender [13] present the findings of the IEEE iLRN 2021 conference, held in Virbela, presenting results of an explorative study of 75 conference participants. Kirchner & Forsberg [14] explored small social gatherings using virtual reality (22 participants in three Nordic countries). Remacle [15] proposes hosting conferences in Minecraft. Williamson et al. [16] discuss proxemics and social interactions in an instrumented virtual reality workshop during ACM CHI 2020.
One key issue is keeping attendees interested and engaged with the content. There is a growing body of literature on participant engagement in conferences, either online, or faceto-face [17], [18], [19], [20], [21], [22] Raby & Madden [23] present a case study of a one-day online event held in July 2020 and explore delegate engagement through registrations and retention, website analytics and monitoring attendance, and a post-conference survey. Wu et al. [7] used data analytics to explore attendee behaviors and psychology in a Virtual event (TWELF2020 in Taiwan). Shanley [24] discusses engagements and interactions during the EASST virtual conference in early 2020. An interesting work from Christopoulos et al. [25] discusses, within an educational context, how virtual worlds can engage learners without using immersion.
While Social VR applications are getting more momentum, most services regarding Social VR focus on animated avatars. Gunkel et al. [40] introduce Social VR services based on photo-realistic video recordings. Their paper focuses on two parts: the communication between multiple people and integrating new media formats to represent users as point clouds. Further, the paper presents a study with 54 people evaluating a three-people communication use case and technical analysis to move towards 3D representations of users.
Johnson [41] emphasizes that the beginning of the COVID-19 pandemic forced the abrupt cancellation of faceto-face conferences. However, as they created a fully new virtual meeting format, they realized that online access during the pandemic had expanded participation in scientific conferences for women, young scientists, and low-and middle-income countries. The author reiterates that the virtual format alleviated hurdles for those with conflicting childcare responsibilities, health issues, teaching responsibilities, restricted travel, economic barriers, and other impediments to in-person meeting attendance. So, as they return to inperson events, they intend to continue offering virtual access to all meetings. Although a hybrid approach to conferences may include both in-person and virtual formats, it will be necessary to better connect the two audiences and to fully integrate the attendees.
Gupta et al. [42] presented a review suggesting the advantages and limitations of conducting virtual events and discussing future trends of holding such events in uncertain times. Among the main advantages mentioned are financial savings, accessibility, increased opportunities for education, reduction of infectious risk, and reduced carbon emissions. On the other hand, the biggest limitations listed by the authors are the lack of networking opportunities, technical difficulties, requirements for support teams, lack of academic tourism, health issues such as computer vision syndrome, distraction, and unavailability of computing resources or networks.
Wu et al. [43] highlight the advantages of technology over the past decade and the ability to attend a virtual conference from any device without the need for custom hardware. Additionally, the authors declare that virtual conferences help alleviate environmental challenges. Furthermore, the authors noted that despite the increase in the number of attendees at virtual conferences, social media engagement during those conferences did not increase as the number of participants did. Finally, the authors conclude that conference organizers must adopt a novel, comprehensive approach to ensure increased accessibility, diversity, and inclusion of post-pandemic conferences.
Much research has focused on discussing the perceived experiences using virtual environments during online conferences. However, we apply diverse engagement metrics for online conferences to explore synergies between the virtual environment and online tools to assess participants' engagement. Our analysis shows that these metrics have the potential to highlight engagement, diversity, and inclusion by combining textual messages, participant geographic and gender information, communities of participants, and visitation patterns in a virtual environment. Moreover, we propose guidelines for organizing technical virtual events to increase diversity and social interactions.

III. IEEE VR 2021 ORGANIZATION AND SOCIAL ACTIVITIES
Due to a diversity of access conditions, bandwidth limitations, and timezone and human resource constraints, the conference decided on a format organized around the conference website for program navigation 5 using Twitch 6 and YouTube for video broadcast and dissemination. In order to avoid attendant fatigue and disengagement, the traditional program was modified to shorten research presentations (seven minutes per presentation, with a backup video previously recorded and uploaded in advance). In contrast to the traditional in-person presentation formats, this would allow for shorter presentations, longer and livelier exchanges, and fewer parallel tracks during the conference technical program. For social interactions, message exchanges, and Q/A activity, the conference used the collaboration tool Discord 7 because of its multimedia capabilities and more informal look and feel compared to similar tools such as Slack. Discord allowed us to set a dedicated server for the conference, Independent Categories, or Groups for each activity (Workshops, Paper Sessions, Talks, Panels, Plenary Sessions, Demos, Posters, etc.). Additionally, each presentation received its own dedicated channel, where participants could access the conference proceeding documents and place questions to presenters. In order to facilitate navigation amongst the hundreds of papers, posters, and 3DUI contributions, each channel could be accessed via a pointer in the conference program.

A. VIRBELA/iLRN
While these tools are well suited to support most communication tasks, in the context of textual exchange and collaboration, for all social events, Demos, Poster Sessions, and 3DUI contest entries, we decided to use a 3D desktop virtual environment (Virbela) hosted on the Immersive Learning Network (iLRN) 8 campus, inspired by the experience of ISMAR 2020, where the organizers had run the full event. We aimed to provide an immersive environment where par-7 http://discord.com/ 8 http://immersivelrn.org/ VOLUME 11, 2023 ticipants were able to perceive other participants and follow another person's elaboration, reactions, and resistance. The Virbela platform allows participants to move avatars in large digital event areas, follow events, and communicate verbally with other participants' avatars. The 3D game world includes buildings, open public outdoor and indoor spaces, and private (sound-isolated) places. Virbela supports live and recorded video broadcasts mapped onto special surfaces in the virtual world. In addition, the VE supports digital twins of regular activities, such as information desks, and PowerPoint presentations on walls, among others. These features persuaded us to include auditoria in Virbela to mirror the technical sessions. The drawback is that immersion from Virbela is a strongly hardware-related property that depends on the technological mode of delivery. As such, immersion relies on the performance of the infrastructure and improves as the audio-visual signals between meeting participants gain in quality, smoothness, and speed.
Regarding the streaming pipeline, the platforms involved were Zoom, Virbela, Twitch, YouTube, Restream, and Open Broadcaster Software (OBS). Initially, we decided to use only Twitch as the official streaming platform. However, Virbela did not support twitch streams directly, and a YouTube Livestream is currently the only way people inside Virbela can watch something live. Since we still wanted to use Twitch, we opted to use Restream, allowing streaming to several platforms simultaneously, including Twitch and YouTube.

B. READY PLAYER 21
One of the challenges to widespread participation and adoption of purely virtual conferences is the lack of compelling reasons to ''go'' there. Additionally, a significant amount of non-trivial setups and tool familiarization must occur before participants feel comfortable and confident enough to join virtually. In short, people are looking for the value added by something beyond the now-familiar desktop conferencing tools, such as Zoom.
In order to attract more people, we decided to create an in-platform game for attendees to play. The game was designed to promote: (a) on-boarding by being available only inside Virbela, (b) familiarity with the virtual conference venue by requiring exploration of and interaction with it, (c) socializing by allowing people to collaborate and communicate; (d) diversity by not requiring heavy (English) language skills; (e) a sense of accomplishment by becoming progressively harder; (f) and fun usage.
We also needed to work within Virbela's framework, which provided some constraints to the design, and within the timeframe of the conference so players could complete the experience before the conference was over. We decided to create a treasure-hunt kind of experience, based loosely on the plot of the book Ready Player One. 9 We called our experience ''Ready Player 21'' (RP21) since this was to be run at IEEE VR 2021. We created a series of four puzzles, and the solution to one puzzle unlocked the hint and instructions for the next puzzle. A running public ''Leaderboard'' was also set up in the world so that all Virbela conference participants could keep track of everyone's progress. We utilized the badge feature of Virbela and awarded RP21 badges for each puzzle solved. These would appear on the avatar of the solver 2. RP21 was designed by an experienced geocacher, 10 and consequently, the creation of the puzzles was relatively straightforward.
A small team of volunteers worked to set up the necessary technical infrastructure to support the RP21 experience and create the artwork (badges). This work included creating Webpages for people to log answers that would be automatically checked and the leaderboard updated. The clues had to be set up and distributed throughout the Virbela world. Secret Virbela rooms needed to be created for participants to access once they solved each puzzle.
The conference organizers announced the start of the game via a recorded message from James Holiday (a riff on the game creator character from Ready Player One 11 ), announcing that he had placed several puzzles throughout the environment. This message started the counter on the leaderboard, and attendees (both inside Virbela and on Zoom) were made aware of the large QR code signboards that were available around the Virbela campus to start the game.
The game ran throughout the entire conference, with players taking part on the last day. In terms of numbers, the main game landing page received 970 hits, and the first puzzle page received approximately 450 hits. A total of 35 people solved at least one puzzle, and 31 solved the first three out of four puzzles. In the end, 13 completed all four puzzles.
In general, the feedback provided by this experience was positive and meaningful. Several times during the conference, we could see people scouring the campus to find the clues for the first puzzle, and the puzzle creator was approached numerous times in the world about clues or other thoughts on solving the puzzles. Often, groups of avatars could be found to be running around together throughout the campus. The game appealed to some segments of participants, though the design did not appeal to everyone as with any game.

C. FLASH MOBS
Another way we tried to encourage socialization was to organize virtual flash mobs. 12 Like their real-world counterparts, people would move their avatars to an appointed location within the campus at a given time and then perform a coordinated set of dance moves. Since dancing was supported within Virbela, this was relatively easy to carry out, with little practice. We organized several of these at the close of the conference, and people seemed to enjoy them 13 (Figure 2).

D. SCIENTIFIC SPEED DATING
One of the critical aspects of a conference is to improve research networks. Multiple funding agencies and research supporters already use the level of collaboration as a metric to evaluate high-impact research by measuring co-authorship in papers or project developments. Carr & Ludvigsen [44] argue that online conferences have a specific challenge in supporting informal and social interaction: ''Online conference designers can face some challenges supporting informal and social interaction among participants. However, these forms of interaction may be essential to develop the safety and trust required for effective engagement in formal conference activities and the formation of professional relationships that last longer than the conference.'' This necessity was perceived by organizers of IEEE VR 2021 when multiple attendees complained about the lack of social interaction moments during the event.
As a quick response, during the first day of IEEE VR 2021, the general chairs met with the online experience chairs to define a strategy to foster the missing social interaction. Scientific Speed Dating (SSD) was pointed out as a great possibility, as it provides an environment to meet new people quickly and to increase the feeling of ''being in a conference.'' After research on SSD formats, the Scientist Speed Dating 14 proposed in 2012 by the Nanoscale Informal Science Education (NISE) Network from the Science Museum of Minnesota provided an initial guide for the activity. Described as ''a facilitated, yet informal and high-energy, social activity to encourage a large group of people to speak with one another, ask questions, and learn about specific areas of research and practice within the field of nanoscale science and engineering, as well as the related societal and ethical implications of work in this field.'', NISE's main goal was to foster discussion between Scientists and Society. Thus, the main goal of an academic conference is to foster discussion between scientists. Further, IEEE VR 2021 was held online, while the original SSD was an in-person event. In this scenario, IEEE VR 2021 organizers kept the main structure of SSD with minor changes.
Before the SSD session, organizers provided a form to gather people's intention to participate in the SSD and their preferences as it was not on the original event program. The time structure remained the same: The SSD in IEEE VR 2021 took great advantage of the Virbela virtual platform capabilities regarding space. Therefore, it was possible to keep the same spatial distribution of an in-person event and quickly expand the offices available if it reached the maximum limit ( Figure 2). After analyzing the multiple spaces of the iLRN space, organizers defined the ''Office Rooms'' as the most adequate for the SSD and renamed it to ''Scientific Speed Dating Square.'' It was a continuous space composed of multiple offices with private sound areas as in a standard university floor, each office containing a round meeting table and boardrooms on the walls.
According to the space setup, the SSD dynamics begin with all numbered offices side-by-side. After the setup instructions, every attendee is asked to walk from the first office to the last, following simple rules: 1) If the office is empty, take a seat and wait for another attendee to talk; 2) If the office has one attendee inside, take a seat and start talking to each other; 3) If the office has two attendees, keep moving to the next office. 4) After each round, the first attendee who reached the office stays, and the attendee who arrived second moves to the next office to the right.
Some attendees will always be seated, and some will always move to a new office to meet the next. By using these simple mechanics, organizers had a warranty not to repeat pairs. Attendants arriving late would find the last office occupied and sit to wait for another late attendee, expanding the group while there are empty offices.
Each attendee should get to know its pair at each discussion round by quickly sharing information such as name, affiliation, scientific interests, and primary objectives at IEEE VR 2021. Then, if the discussion flows, they are supposed to exchange contacts or keep the discussion later. To control the round duration and communicate to all attendees, organizers broadcast text messages to start each round, provide a one-minute warning, and end each round. After all rounds, organizers share final remarks and acknowledge the audience, inviting participants to get to know each other better and to use the conference to exchange new ideas.
The SSD event took place on Day 5 of the conference, with 54 attendees. After sound positive feedback in Discord messages, an extra event took place on Day 6, allowing more people to engage and interact. An analysis of how SSD fostered engagement, and social interaction is presented in the following sections.

IV. DATA ANALYSIS FROM ONLINE TOOLS
To understand the user participation patterns at the IEEE VR 2021 conference, we made a thorough data analysis where we analyzed textual message interactions from Discord and user participation in Virbela. This analysis aims to gain insights into how the choice of these tools can promote VOLUME 11, 2023 synergies between a virtual environment. More specifically, we are interested in understanding the impact of our conference program for the different participants across the globe and understanding behavioral patterns between male and female participants to organize better conferences in the future that attract more females and contribute to the reduction of the diversity gap in the Virtual Reality domain.

A. DATA AUGMENTATION AND ANONYMIZATION
To ensure a robust data analysis, we had to intersect different data sources, such as participants' registration data, with their interactions in Discord. This was a challenging task, because most of the conference participants were registered in Discord with an email different from their institutional email. In these circumstances, the identification process difficult, because we could not directly correlate registered participants with their respective Discord activities. Therefore, we developed a framework to (1) associate conference registered users to their respective Discord / Virbela activities, (2) integrate data from these different platforms into one single dataset, and (3) anonymize participants.
We would like to highlight that both Discord and Virbella collect textual messages from registered users with their consent, which are made public and visible to all IEEE VR 2021 participants. We did not collect any additional data beyond the data that is already provided by these tools. Although the information was already public at Discord and Virbela, we made an effort to anonymize it. Figure 3, presents the anonymization framework that we used for this study.

1) DISCORD DATA COLLECTION
For this study, we used three primary sources of information: participants' conference registration information, message exchanges in Discord, and Virbela visitation patterns. We extracted participant information such as user discord username, role, server registration time, and textual messages, using an open-source API to extract all the textual messages from the IEEE VR 2021 Discord server.

2) MATCH PARTICIPANTS' REGISTRATION TO DISCORD's USERNAME
The online format of IEEE VR 2021 enabled the collection of textual interactions between participants in Discord. However, a problem we faced when trying to analyze the participants' message interactions is that for more than half of the conference participants, the username used in Discord did not match the participant's registration name. Not having this association would imply limitations in investigating the topics discussed between the different gender groups.
To address this problem, we applied the following steps: (1) assuming that some participants would use slight variations of their names as Discord usernames, we applied text similarity distance metrics such as Jaro, Jaro Winkler, and Levenshtein [45] to determine these pairs. In the end, we matched approximately 2/3 of the registered participants using the combination of these metrics; (2) for the cases where the Discord username was very disparate from the participant's name. We applied text mining techniques to the conference's registration desk in Discord. IEEE VR 2021 participants were required to communicate their name when registering at the conference, where a role would be given to them (such as author, attendee, etc.). The interactions were usually of the form ''<DiscordUser-name> Hello! My name is [XXX] and I am the author of . . . ''. These interactions enabled us to capture the remaining associations using text mining techniques and with a final manual verification check. In this stage, we identified duplicate profiles due to participants registering in Discord with different emails, which we consolidated into a single user. From a total of 1,212 registered participants, we could find message exchanges in Discord for 962 users (approximately 79%), which served as the basis for this study. For Virbela, this task was simplified since most participants used their conference registration email to also register on the platform.

3) EXTRACT PARTICIPANTS' GENDER IDENTITY
The gender information of each participant was extracted directly from the registration forms of IEEE VR 2021. Of 962 users, 53 chose not to disclose their gender identities (approximately 5%).

4) EXTRACT PARTICIPANTS' TIMEZONE
This study required information about the participants' time zones to understand how the conference program affected participants in different regions. We assumed that participants were attending the conference from a region close to their affiliation. We extracted geographical coordinates, such as Latitude and Longitude, by using an open-source API called OpenStreetMap [46], which is highly used in research studies [47]. From the latitude and longitude, it is possible to compute the timezone of the participant. The automatic extraction of time zones and continents resulted in noisy and poor-quality data that had to be manually adjusted. Additionally, since Discord textual messages were extracted from Australia, these messages were all associated with the GMT+10 timezone, which needed to be converted to the european timezone where the conference took place. Since this would not enable us to compare and analyze user interaction patterns throughout the conference, which was in Lisbon time, we had the additional task of correcting all the timestamps associated with each Discord message and Virbela visitation patterns in the corresponding timezone.

5) ANONYMIZE DATA
After connecting all the information about the participants, we had to proceed to the anonymization of the data in such a way that it would be impossible to identify the participants from this dataset. For this, we randomly generated IDs for each of the 962 participants. Given that the Discord message interactions also referred to the participants' names or Discord's usernames, we then applied text-matching techniques where we replaced all the mentions of the participant name, email, or Discord username with the randomly generated ID. We then made a final manual verification to check if any mention of the participant could still be found in the message interactions. Finally, we deleted the participants' names, email, DiscordID, and Virbela ID and worked uniquely with their generated AnonID. In the end, the anonymized participant dataset we used for this study had the AnonID, gender information, timezone, and abstract Discord conference role (whether the person was a conference organizer, an author, an attendee, or a speaker). The participants could not be identified from the Discord message interactions; however, one could sometimes infer whether the participant was part of the conference organizing committee by the content of messages, but still with no possibility of identifying the person. We did not make any of this information publicly available, and the anonymized data were used solely to support data analysis for this study.

B. GENERAL DESCRIPTIVE STATISTICS
The conference had 1,212 registered participants from 49 different countries. The organizers offered 14 scholarships to cover registration costs to participants from under-represented countries to promote a more diverse and inclusive attendance. These scholarships provided registration fees for four full paper authors, one poster author, one workshop author, and eight general participants. Table 1 presents the participant gender distribution. Additionally, it shows a male-dominated conference in a proportion of approximately 2:1 male-to-female ratio, including other gender identity categories.

C. PARTICIPANT DEMOGRAPHICS AND INTERACTION PATTERNS
An important concern in online conferences is how to propose a program schedule that can accommodate participants from different timezones. In this paper, we are interested in understanding how the chosen conference program affected different participants. Figure 4 presents user interaction patterns on Discord from different continents per hour across all the days of the conference.
As expected, the users who were highly engaged in exchanging messages in Discord were in a timezone compatible with Lisbon time (UTC+1 due to daylight savings time). Countries in North America also had a significant presence. Two factors also need to be considered to avoid a biased reading: many of the large research centers in VR can be found in these two continents, namely research centers in France, the USA, and Germany that had a strong presence at the conference. Additionally, participants in Oceania found it hard to engage with others at the conference, since the time difference exceeded 10 hours. Figure 4 also highlights a few participant interactions from under-representative countries in South America or Asia.

D. ENGAGEMENT METRICS
To understand user engagement, we propose a set of metrics that range from textual interactions in Discord to visitation patterns in Virbela while trying to identify correlations between the two.

1) TEXT-CENTRIC METRICS
These metrics focus uniquely on general descriptive statistics of the messages exchanged in a channel of a specific category in Discord.

Number of Message Exchanges:
Measures the average number of messages exchanged per channel of a given category. It is assumed that the more messages exchanged in a channel, the more engaged the user is with that topic. Length of Messages Exchanged: Measures the average length of the messages exchanged per channel of a given category. By length, we define the number of sentences in a message from a given user. Since many participants exchange quick messages such as thank you, great presentation, etc., with this metric, we are interested in understanding if the topics in the different Discord channels promoted continuous and long discussions. We assume that longer messages correlate with more engagement. Table 2 shows that keynotes, posters, tutorials, and workshops had more user engagement and participation than the more technical sessions, such as Paper Discussions. This is expected since keynote speakers had more time to present than the paper presenters did. The same applies to workshops, tutorials, and panel sessions. The Keynote sessions had the most user participation, and detailed user interactions can be found in Section IV-E. Table 2 also suggests that Paper discussions seemingly had less engagement as compared to all the other sessions presented at the conference. There is the preconceived idea that an online environment that makes virtual conference presentations more engaging than in-person paper discussions [10]. However, this ignores the important role that expectations play in driving audience engagement. When presenting a paper discussion, audiences typically expect to listen to a dry, academic presentation with little interaction or opportunity for questions. In contrast, when participating in an interactive conference session, audiences expect to be engaged and participate in the discussion. This difference in expectation can account for much of the difference in engagement levels between these two presentation types.
Recent research on virtual conferences has shown that presenter expectations play a significant role in determining audience engagement. For example, Pedaste and Kaemets [48] found that participants in web-based conferences were more likely to be engaged when they felt their contributions would be ''heard'' by the presenter. Thus, it is not necessarily true that the online environment inherently supports interactive conference presentation formats over paper discussions. Rather, the audience's expectations drive engagement levels in these two presentation modalities. Our experiences seemingly corroborate this. The speakers were not present in the virtual world during presentations, but during keynote talks, the presenters were engaged in the chat in Discord. As a result, there was very little interactivity in the virtual environment, where the presenter was rarely available (due to holding forth on Zoom). Still, many resulting posts were made on Discord, where the presenter could announce their engagement beforehand (e.g., by sending a message indicating that they were interested in taking questions ahead of the session).

2) PARTICIPANT-FOCUSED MEASURES
These metrics focus on the textual exchanges by participants in each Discord channel. Table 3 presents the results, using the following metrics: Participants per Channel : Measures the number of participants that interacted in a channel. It is assumed that the higher the number of participants in each channel, the more engaged they were about its respective topics.

Participants' Messages per Channel@N (PartMsg@N):
Measures the number of participants who shared more than N messages in a channel from a given category divided by the total number of participants in that channel. In this study, we computed N = {2, 5, 10} to determine if participants continuously engaged in discussions in Discord rather than just contributing with just one message. Participant engagement in channel: Measures how many messages have been exchanged in a channel from a given category averaged by the number of participants in that channel. We assume that categories with high numbers of messages per participant may indicate topics of interest. Table 3 presents the results of the above metrics where one can see that the most interactive categories were the technical sessions, the workshops, and Keynote sessions. Demos and Birds of a Feather (BoF) took place mainly in Virbela, which explains the low engagement in Discord. Additionally, only in the longer sessions, such as tutorials or workshops in we see the engagement of users who shared at least ten messages in a workshop, suggesting that the sessions were highly appealing to the participants.

3) GENDER-CENTRIC METRICS
These metrics are similar to the participant-focused metrics, but are sorted by different gender identities. With these metrics, we are interested in understanding how diverse and inclusive the conference was. Figure 5 presents participant engagement by gender identity across different categories in Discord. Participants per gender and per Channel: Counts participants by their gender identities that interacted in a channel normalized by the total number of participants of a given gender identity. Participants' Messages per gender and per Channel: Measures the number of participants by their gender identities who shared more than N messages in a channel from a given category, divided by the total number of Female/Male participants in that channel. Female/Male engagement per channel. Measures how many messages have been exchanged in a channel from a given category averaged by the number of participants in that channel with a given gender identity.
Although the female-to-male ratio was 1:3, we can see similar interaction and engagement patterns between males and females when normalizing the results over all the identified female participants in Discord. An interesting aspect that Figure 5 highlights is that female participants engaged more in Keynote Sessions, while males seemed to be more active in Tutorial or Workshop sessions. To better understand this difference in engagement between males and females, we analyzed the topics of each Discord channel and computed the percentage of messages exchanged between these two groups. Figure 6 presents our findings.
We can confirm previous results in terms of male participants being more engaged in Workshops and Tutorials, but also see that female participants seemed to dominate certain discussions in topics concerned with Training & Learning, Accessibility in VR, Embodiment and in general Panel and Keynote discussions. To reduce the gender gap for future conferences, organizers could propose more activities or sessions related to these topics.
We complemented this analysis with messages exchanged throughout the conference days between these two gender classes. Figure 7 suggests similar participation patterns between males and females throughout the conference except on the second, third and fifth days of the conference. Day 5 corresponds to the day where the technical sessions were more related to topics that male participants did not engage in, but females did: Accessibility, Haptics, and Training & Learning. A lack of interest in males in these topics may explain the decreased engagement on Day 5 in Discord. However, social activities also took place in Virbela, which may explain the lower Discord participation rates.

E. PARTICIPANT INTERACTION DURING KEYNOTE SESSIONS
One of the sessions that contributed to the high engagement of participants was the Keynote speaker session. IEEE VOLUME 11, 2023 VR 2021 had four keynote sessions ranging from avatar in immersive technologies, bending realities, AR/VR, and data science. All keynote speakers presented online in real-time, except one who could not give a live talk from her location due to work constraints, so she made a pre-recording of her presentation, which she then shared. We analyzed all the interactions in the keynotes sessions and made a network analysis that is presented in Figure 8.
We analyzed the messages sent directly to a specific person in Discord. For all other messages, we mapped them to a central node since all other participants followed these messages  or even interacted with them. This provided a clear graph for analysis. Figure 8 shows three main clusters that correspond to interactions centered in the keynote speaker. One of the keynote speakers did not register in Discord and therefore does not appear in the graph. In this session, the questions were mediated via the session chair in Virbela. One can also see that the cluster belonging to Keynote θ is the largest and promoted more interactions. This keynote speaker pre-recorded the video and did not make a live presentation. While the presentation video was playing, the keynote speaker was at the same time engaging in Discord, providing clarifications about the talk and also answering participants' questions as they made them. This setting enabled maximum interaction between keynote speakers and participants that highly benefited user engagement.

F. VIRBELA/iLRN MEASURES
We wanted to assess usage of Virbela by meeting participants. To this end, we were to collect two main engagement measures. Unfortunately, the granularity of visitation patterns does not discriminate between areas in a room, which would have allowed us to visualize most accessed zones in posters/social/meeting spaces. We therefore present two main engagement metrics: Based on the analysis of unique visitors and the average time spent per room by day in Virbela/iLRN rooms, it is possible to identify multiple indications of engagement and social interaction. For example, by analyzing Day 5 in Figure 9, it is possible to see the high average time spent in Ready Player 21 and Scientific Speed Dating (SSD) rooms. Furthermore, the duration of the visits to these two events is comparable to the time spent in the main event rooms as the ''VR21 Auditorium A'' (Full papers/Keynotes) and ''VR21 Expo Hall A/B'' (posters/demos/3DUI). In addition, SSD appears as one of the most relevant activities on Day 5 on the virtual platform regarding unique visitors, indicating relevance for social interaction on the platform. By looking at all event day data, the Puzzle rooms for Ready Player 21 had the highest average time spent, above all rooms at the virtual platform, due to the difficulties of the challenges.

FIGURE 8.
Interaction patterns between attendees and conference committee members during keynote and after keynote sessions. Each keynote speaker is associated with a random Greek letter to avoid identification.

V. FINANCIAL CONSIDERATIONS
Conferences moving to online formats have also needed to adjust to the vastly different financial model. The VR conference in 2019 had author registration rates of approximately $750 USD (after conversion from 112,500 JPY), with the vast majority of funds allocated to conference centers and social activities. VR 2020 had an interim model, where only authors paid a registration fee of $450 USD and attendees were free of charge. VR 2021 adopted a compromise of author fees of $360 USD, largely paying for the publication costs, and online attendee fees of $60, largely associated with the cost of the Virbela online environment, which scaled directly with registered participants. One key lesson learned was that charging a small fee for remote participation, seemingly increasing participant engagement while allowing the venue to scale with attendance figures.

VI. POST-CONFERENCE QUESTIONNAIRE
In order to assess participant attitudes and opinions regarding conference organization and suggestions for improvement, we conducted a post-conference questionnaire online, 15 answered by N = 90 people in September/October 2021, which represents around 7.5% of the conference's registered attendees. Unfortunately, it is difficult to engage a high number of attendees in post-conference questionnaires. As an example, ISMAR 2020 had 625 attendees but its post-conference questionnaire was answered by just 93 attendees. Even with the low number of answers, our demography 15 See the questionnaire at (https://bit.ly/IEEEVR21) analysis in the next section shows a coherent distribution when compared to the registration data, indicating a representative sample of the conference's attendees. The answers can also be seen online 16 anonymously. We reproduce the main findings in this section together with individual anecdotal feedback to better motivate and inform our general findings and recommendations. Demography: Of the 90 respondents, 67.8% were male. The

VII. GUIDELINES AND FINAL RECOMMENDATIONS
The analysis made throughout this study enabled us to collect guidelines and final recommendations for the organization of conferences in a fully online setting. The guidelines are insights directly supported by this research data analysis, while the final recommendations arise from our experience.
A. GUIDELINES G0: Offer Scholarships to promote inclusion and diversity. Offering scholarships to cover registration costs from under-representative participants can contribute to a more diverse and inclusive range of participants. These scholarships may also promote and inspire young researchers to contribute to the area of VR (from Section IV-B). G1: Pre-record presentations to increase synergies between speaker and attendees. If a speaker uses a pre-recorded presentation video, the speaker's availability to interact with the participants can be maximized. Participants can watch presentations in advance and the speaker can have more opportunities to provide clarifications along with the presentation (from Figure 8.) G2: Prepare activities for online participants to promote engagement. Ready Player 21 and SSD analysis showed significantly increased engagement and social interactions. However, the SSD last-minute planning and publicity profoundly impacted the participation rate with multiple attendees reporting not knowing about it or not making the time to participate. To avoid that, social activities must be planned and widely publicized before and during the event.
G3: Organize the conference program around time zone differences. Making an event work across different timezones can be quite challenging. One approach is to cluster paper and technical sessions such that most presenters are in adjacent timezones. While this can increase attendance in some sessions, the conference can become more fragmented. Another possibility is to replicate the discussion using recorded content and different moderators (from Figure 4.) G4: Keep presentations short and sweet to reduce fatigue IEEE VR 2021 featured seven-minute presentations in a panel discussion format. We required each presenting author to prerecord, rehearse and send their presentations in advance. Having a backup video for all presentations provided insurance against connection failures. The short presentations and panel discussion format kept discussions lively, and Discord allowed for a level playground regarding discussions and questions. Furthermore, having a separate channel for each presentation allowed for persistent discussions that continued long after the session was over in many cases.
G5: Virtual environments are better to socialize and can minimize the number of online tools. Multiple attendees complained about the high number of tools that were used for the conference. Using virtual platforms can be an option as it can integrate live/recorded presentations and social interaction inside the same environment i.e. Ready Player 21 and Scientific Speed Dating. However, from experience, less social events, such as keynote talks and paper presentations, are better accessed and archived via dedicated streaming platforms.

B. RECOMMENDATIONS
R0: Increase sessions on topics where female participants are more active. The female participants of the IEEE VR 2021 conference seemed to dominate certain discussions in topics concerned with Training & Learning, Accessibility in VR, Embodiment and in general Panel and Keynote discussions. More research is needed to better understand topics of interest in the VR community so organizers can propose more activities related to topics that generate more engagement from underrepresented participants (from Figure 6). R1: Make sure multimedia and conference hosting work. Critical to the conference success is to make sure that the conference hosting site has reliable and supports fast internet connections. Make sure that there are backup computers to cope with media/hardware failures during the event.
R2: Engage Student Volunteers as early as possible. Dedicated volunteers are the key to any academic conference's success. Engage the best and most enthusiastic students, creating a team of inspired and enthusiastic people. Student Volunteers, the Multimedia Team, and Active Subcommittee Chairs were a key to the conference's success. Plus, the teams built around the event have endured and gone on to other successful academic endeavors.
R3: Increase participant interactions by setting expectations in advance. Increase participant interactions in virtual environments by setting expectations in advance. Before the conference, set expectations about how authors and presenters will interact with the audience virtually. Let them know in advance if you are available during the chat during presentations. If you are interested in taking questions, let them know how, when, and where they can submit them. This will help increase interactivity and engagement in the virtual environment.

VIII. CONCLUSION
The pandemic has forced drastic changes in scientific gatherings, given the strict restrictions on travel and strenuous challenges to face-to-face meetings. We have described the experience of hosting the IEEE Virtual Reality Conference. Despite being planned as a hybrid event, IEEE VR 2021 had to be recast as a purely virtual conference, given the second wave of the COVID-19 pandemic. The conference was a success due to a combination of enthusiasm, careful planning, flexibility, and improvisation to cope with lastminute changes. We combined a Virtual Environment with Discussion Tools and Video Conferencing, Broadcast, and Online Tools to facilitate and engage effective social interactions. We assessed user engagement by examining participant exchanges in the VE and online tools and analyzing the messages exchanged between participants of different gender and geographies. Our analyses and metrics emphasize engagement, diversity, and inclusiveness by combining text messaging, participant geographic and gender information, attendee communities, and VE traffic patterns. Based on our results and experience, we proposed guidelines for organizing technical virtual events to increase social interaction. While the future remains uncertain, the events in the past two years seem to have forever changed the organics of academic gath-erings. Our experience suggests that virtual environments powered by the next generation of collaborative tools and robust multimedia and communications infrastructure will usher in the future generation of technology-enhanced scientific events.
CATARINA MOREIRA is currently a Senior Lecturer with the School of Information Systems, Queensland University of Technology, where she also holds the role of the Deputy High Degree Research Academic Lead. She is a Computer Scientist and passionate about investigating data science, machine learning/deep learning technologies, and their impacts on society. She is the Co-Founder and the Co-Leader of the XAMI research initiative (eXplainable Analytics for Machine Intelligence), where she focuses on innovative human interactive probabilistic models to promote more fair, transparent, and trustworthy AI models. She is also a pioneer in quantum probabilistic technologies for cognition and decision-making where she developed models to accommodate paradoxical human decisions in scenarios with high levels of uncertainty.
FRANCISCO P. M. SIMÕES received the M.Sc. and Ph.D. degrees in computer science from the Universidade Federal Rural de Pernambuco (UFRPE) with internships at Lagadic/INRIA and CVLab/EPFL. He is currently an Adjunct Professor with the UFRPE, and a Research Leader at Voxar Labs/UFPE. He has a solid experience in R&D&I for industry and applied research, with contributions in high-impact journals, conferences, and patents. His research interests include spatial intelligence, using computer vision (CV) and extended reality (XR) to better understand spaces and to make a bridge between real and virtual experiences. He has been the Vice-President of the Special Committee on Virtual Reality (CE-RV) of the Brazilian Computing Society (SBC), since 2018. ROBERT W. LINDEMAN is currently the Director of the Human Interface Technology Laboratory NZ (HIT Laboratory NZ), University of Canterbury, and has been doing research in the field of VR, since 1993. He has also done significant work on the effective use of 360-degree video for science outreach, including work in Antarctica. He is also on Sabbatical in Europe, splitting his time between working with Prof. Anthony Steed at University College London (August/September), and Prof. Frank Steinicke at the University of Hamburg (October/November). His research interests include immersive, multi-sensory feedback systems for VR, AR, gaming, and long-term VR immersion research, specifically non-fatiguing and non-sickness-inducing experiences.