Design and Experimental Validation of an Augmented Reality System With Wireless Integration for Context Aware Enhanced Show Experience in Auditoriums

The development of multiple cultural and social related activities, such as shows related with the performing arts, conferences or presentations rely on facilities such as auditoriums, theatres and conference sites, which are progressively including multiple technological features in order to enhance user experience. There are still however situations in which user experience is limited owing to lack of environment adaption, such as people with disabilities. In this sense, the adoption of Context Aware paradigms within auditoriums can provide adequate functionalities in order to comply with specific needs. This work is aimed at demonstrating the feasibility in enhancing user experience (e.g., improving the autonomy of disabled people) within auditorium and theatre environments, by means of an Augmented Reality (AR) device (HoloLens smart glasses) with wireless system integration. To carry out the demonstration, different elements to build AR applications are described and tested. First, an intensive measurement campaign was performed in a real auditorium in the city of Pamplona (Baluarte Congress Center) in order to evaluate the feasibility of using Wi-Fi enabled AR devices in a complex wireless propagation scenario. The results show that these environments exhibit high levels of interference, owing to the co-existence and non-coordinated operation of multiple wireless communication systems, such as on site and temporary Wi-Fi access points, wireless microphones or communications systems used by performers, staff and users. Deterministic wireless channel estimation based in volumetric 3D Ray Launching have been obtained for the complete scenario volume, in order to assess quality of service metrics. For illustration purposes, a user-friendly application to help hearing impaired people was developed and its main features were tested in the auditorium. Such an application provides users with a 3D virtual space to visualize useful multimedia content like subtitles or additional information about the show, as well as an integrated call button.

disability [1]. Within these disabilities, the latest report of October 2019 [2] estimates that at least 2.2 billion people suffer from some kind of visual impairment or blindness and 1 billion of them have a moderate or severe visual impairment (i.e., uncorrected refractive problems, macular degeneration, cataract, diabetic retinopathy or glaucoma, among others). Hearing loss is another cause of impairment that more than 466 million people suffer in the world [3]. Such a loss may be congenital or acquired, and the level of hearing loss varies, with deafness being the most serious.
Regarding visual disabilities, it should be noted that vision is the sense that is more developed in humans and, in fact, 80% of the information required in our daily living is captured through our eyes. Serious visual shortcomings lead to restrictions in having an autonomous life (e.g., getting around or carrying out daily activities). In addition, according to the latest data from the National Statistics Institute of Spain (INE), it causes a decrease in participation in areas that are not accessible, such as education, work, leisure or culture [4]. An example of a leisure or cultural activity is the attendance to theaters or auditoriums, which is essential for anyone interested in certain cultural activities, whether or not they have a disability. Unfortunately, such environments are usually poorly adapted for people with disabilities, both in their ability to move freely inside the building (which is essential for visually impaired people) and during shows (which affects both visually and hearing-impaired people).
This article presents a system that was deployed in Baluarte Palace of Congresses and Auditorium of Navarre [5], located in the city of Pamplona. Baluarte is a palace that holds events like conferences, exhibitions, shows or congresses of multiple disciplines. Such events need for providing different human and technological services, including qualified personnel to help and accommodate the people who come to the shows. On the technological side, the auditorium has lighting and audiovisual cabins where scenography is controlled. There are devices available for people with mild hearing impairments (hearing devices) and high hearing impairments (transmitters based on Modulated Frequency systems for users wearing cochlear implants). However, no guidance is available for people with visual disabilities, and no additional tools are used to improve their QoE (Quality of Experience).
There are different popular technologies like smartphones or tablets that may help to enhance impaired people experience by delivering certain content (e.g., contextual videos or other multimedia content), but they provide a worse user experience (i.e., they are less immersive) than other technologies like Augmented Reality (AR) and can be difficult to use in auditoriums, since they have to be hold continuously (which is a critical problem for people with certain physical disabilities) and they can interfere with other spectators' experience (i.e., tablets and smartphones pollute the auditorium with light and can interfere other spectators view of the scenario). Devices like cardboard glasses based on smartphones may help to recreate part of the AR experience, but their visualization hardware is currently rather limited in comparison to dedicated AR glasses [6]. In contrast, the latest dedicated AR devices, thanks to their computing power, embedded displays and audio outputs (i.e., speakers, earphone jacks), provide really useful tools for easing middle and long-distance reading/watching and are able to improve hearing, thus enhancing human perception and interaction [7], [8]. Due to such benefits, this article proposes to take advantage of AR systems and thus improve the user experience of people with mobility, visual or hearing disabilities during live shows. Specifically, and as an example of the capabilities of the presented system, this article describes the design and development of an AR application for Microsoft HoloLens smart glasses to assist hearing-impaired users and to provide them a better user experience during the performances in Baluarte Auditorium by showing additional virtual information and multimedia content.
In order to demonstrate the feasibility of implementing AR-based solutions in an environment as complex in terms of electromagnetic propagation as a large auditorium, this article evaluates the performance of a Wi-Fi enabled AR solution in a realistic scenario. Specifically, a radio characterization has been performed by means of the 3D Ray Launching (3D-RL) simulation software. Such simulations consider the dimensions of the auditorium together with certain properties of the obstacle materials, like their dielectric constant and their conductivity. Thus, measurements were performed at 2.4 GHz and 5.5 GHz frequency bands, where different communications technologies (e.g., Wi-Fi, Bluetooth or ZigBee) can operate while being used by devices like the Microsoft HoloLens smart glasses. In addition, for illustration purposes, this article describes the design and development of an AR application for Microsoft HoloLens smart glasses to assist impaired users and to provide a better user experience during the performances in Baluarte Auditorium by showing additional virtual information and multimedia content.
This article extends the initial work described in [9], which compared measurements taken in Baluarte Auditorium with preliminary 3D-RL simulations and analyzed the radio frequency spectrum of the two aforementioned bands. Due to interference levels within the auditorium shown by the results of such work, optimized radio planning is key aspect to consider. Following this conclusion, in this article simulations are carried out with AR devices located at the impaired person's head height (to emulate Microsoft HoloLens location) and in different auditorium positions (to emulate Wi-Fi access point locations). Moreover, how the number of people in the audience affects the device communications is analyzed, introducing human body models in the simulations. Thus, the 3D results obtained with ray-launching software help to decide the number of devices to be deployed as well as their location in the auditorium, therefore enabling the optimization of the deployment of different devices such as HoloLens, access points or wireless sensors. As a summary, the following are the main novel contributions of this article, which have not been found together in previous works: • The article demonstrates the feasibility of implementing Wi-Fi enabled AR applications in scenarios as complex in terms of electromagnetic propagation as large auditoriums. Thus, an extensive radio characterization of a real scenario is presented through simulations based on 3D Ray Launching software (3D-RL). Such simulations consider the dimensions of the auditorium together with certain properties of the obstacle materials, like their dielectric constant and their conductivity. Thus, measurements were performed at 2.4 GHz and 5.5 GHz frequency bands, where different communications technologies (e.g., Wi-Fi, Bluetooth or ZigBee) can operate and therefore can cause interference.
• A novel edge computing-based architecture was devised, implemented and tested with the objective of supporting low-time response decentralized AR applications that do not rely strictly on remote clouds on the Internet, which usually suppose a single-point-offailure.
• To illustrate the potential of the proposed system and thus illustrate its feasibility, an AR application was developed and tested in a real scenario. Such an application was devised to be user-friendly and to help essentially hearing impaired people, but other people with other visual and physical disabilities can take advantage of it as well. Specifically, the application provides users with a 3D virtual space to visualize useful multimedia content like subtitles or additional information about the show. In addition, a call button is integrated within the AR application to ask the auditorium stewards for assistance.
The rest of this paper is organized as follows. Section 2 describes the most relevant related works. Section 3 describes the radio characterization of the auditorium under analysis, while Section 4 presents the design and implementation of the AR/MR applications proposed. Finally, Section 5 is devoted to discuss the obtained results.

II. RELATED WORK A. WIRELESS COMMUNICATIONS AND LOCATION SYSTEMS FOR IMPAIRED PEOPLE
In the literature, numerous publications of wireless communication systems related to visual impairment are found for both indoor and outdoor environments. Regarding the latter, some studies focus on accessibility and mobility on urban buses [10], [11]. In addition, individual guidance and location systems have been developed through sensors, mobile applications and GPS, which provide audible help messages [12] along with family alert systems when assistance is required [13] or even a task management application [14].
Previous literature includes, for indoor environments, reviews of navigation and location systems [15], [16], and also of WBAN [17], for VIP (Visually Impaired People). In the case of hearing-impaired people, reference [18] includes a review on assistive listening devices. Among autonomously guided and indoor location studies, references [19]- [21] focus on the deployment of WSNs that deploy ZigBee or RFID (Radio Frequency Identification) nodes in household environments for obstacle detection. In [22], INSIGHT is introduced, which is an indoor navigation system that uses different wireless technologies to locate users in a public building. Other previous works focus on the detection of obstacles [23] and, specifically, on obstacles above waist height [24] (e.g., fire extinguishers or phones that protrude out of walls). In [25], a diversified shared latent variable model is proposed to use the availability of Wi-Fi for location. Reference [26] explores the idea of navigation by means of a hybrid system, using different signals to increase accuracy due to the difficulty of localization in indoor environments. In [27], an assistance application is developed that uses MEMS (microelectromechanical system) sensors from a Smartphone. Reference [28] analyzes the effects of the deployment of BLE (Bluetooth Low Energy) beacons in a mall to improve VIP location accuracy. Reference [29] presents a self-guided system for VIP in a museum where several location technologies were evaluated before deciding on using a combined WLAN (Wireless Local Area Network) and BLE beacon system.
Among the studies that analyze the use of intelligent devices for the detection of obstacles in indoor environments, smart canes and smart glasses are the most popular. With regard to the former, in [30] the authors describe a smart cane composed of ultrasonic sensors that warns VIPs about their distance from obstacles. In [31], a cane with RFID tags is used (it is able to determine the posture of the user (seated or standing) through the Received Signal Strength Indicator (RSSI)), while in [32] an alert was added so that the person with disabilities can find it in case of losing it. Finally, it is worth citing reference [33], which shows a prototype glove that uses sensors for obstacle detection in combination with a WSN for location.

B. AR SOLUTIONS FOR IMPAIRED PEOPLE
AR is a field that has evolved significantly in the last years thanks to the progress of embedded electronics, data processing techniques and wireless communications. Since the first pioneering works in the 1960s [34], [35], AR/MR regained interest first in the 1990s [36], [37] and later in the 2000s [38], [39] thanks to mobile developments and initiatives carried out by industry and academia [40]. In fact, it is currently considered one of the essential technologies of the Industry 4.0 paradigm [41], [42]. Thus, multiple AR applications have been developed for diverse fields like design [43], [44], product assembly [45], nano-manufacturing [46], staff training [47] or maintenance [48]. The application in the field of leisure related activities is still in initial stages.
In relation with users with some kind on impairments, AR can enable a wide range of user-friendly and real-time applications, such as information and assistance provision (e.g., vibrations for haptic feedback, sign-reading assistance [49]), location, guidance and navigation, augmented collaboration or interaction with Internet of Things (IoT) applications. There are studies that make use of smart glasses, such as references [50], [51], which have developed glasses with ultrasonic sensors for obstacle detection. In [52] the reaction speed in obstacle detection is analyzed by using glasses with the Gablind tool, which reproduces navigation instructions through an app. In [53] an AR system is developed with glasses and smart bracelets for mobility in the city for people with disabilities. Reference [54] presents smart glasses that provide obstacle information to the person with disabilities through audio warnings. In [55] an intelligent stick together with smart glasses give fall detection alerts, sending the GPS location to a predefined contact.
The following references have in common the use of Microsoft HoloLens smart glasses for indoor localization, using different techniques to improve their accuracy. A preliminary step towards localization is the work described in [56]. Such an article focuses on quantifying gait parameters like step length, cadence of walking speed in healthy adults and people with Parkinson. In [57] an evaluation of smart glasses is presented in terms of their functionalities and mapping in a multi-room environment. In [58] the authors make use of smart glasses to solve the location issues that signal-based methods such as Wi-Fi or BLE offer. In [59] a wearable system is proposed by combining smart glasses with RFID and QZSS (Quasi-Zenith Satellite System) to improve GPS coverage in Japan for indoor and outdoor environments. In [60] smart glasses are used together with high speed techniques, spatial mapping and point-cloud matching, to solve problems of mobile location in indoor environments. A different approach is proposed in [61], where the authors develop a proof of concept that focuses on the use of spatial sound system in HoloLens to guide a visually impaired person. The obtained results show a consistent localization of the spatial sounds with a deviation less than 10 • (often less than 5 • ) in an unfamiliar building.
With respect to human-centered AR approaches, although many research efforts have been devoted to QoS (Quality of Service), there are still a limited number of solutions centered on QoE (Quality of Experience). For instance, in [62] the authors propose a QoE evaluation model for Microsoft HoloLens in order to guide developers and designers to improve the usability of developed applications.

C. SOLUTIONS FOR IMPAIRED PEOPLE IN AUDITORIUMS
As it can be concluded from the previous subsections, the research studies focused on the deployment of WSN in combination with AR devices for people with disabilities are very scarce. Moreover, the analysis of the radio channel in environments like auditoriums are pretty limited. The following references are related, since they are based on measurements performed in the UHF (Ultra High Frequency) band in a concert hall to assess statistically the channel in such an environment. References [63]- [65] are aimed at analyzing the behavior on the radio channel of applications such as PWMS (Professional Wireless Microphone Systems) and C-PMSE (Program Making and Special Event applications), respectively. Reference [66] analyzes, in terms of delay spread and by applying ray-tracing based methods, the impact of the presence of 100 people inside a concert hall. References [67], [68] present location systems for auditoriums based on the collection of RSSI levels received in the WLAN. Reference [69] describes a Wi-Fi live audio streaming system to connect microphones that is tested in an auditorium.
As it was previously stated, buildings such as auditoriums and theaters are not usually adapted to VIPs. Nonetheless, such environments are better adapted to people with other disabilities such as reduced mobility, since the solutions usually only requires to remove physical barriers. On the other hand, visual and hearing impairments derive into issues related to mobility and to the show quality of experience. To the knowledge of the authors, the only ongoing AR solution in operation for people with hearing difficulties in auditoriums and theaters is the so-called Smart Caption Glasses [70]. Such a solution has been used at the National Theater of London during the last years and more recently in cultural and civic centers in the United States [71]. The Smart Caption Glasses solution provides live transcripts of a play dialogues and descriptions on the captured sound. The information is presented on Epson Moverio BT-350 smart glasses [72], which are cheaper than Microsoft HoloLens ($1100 versus $3500), but, from the experience of the authors of this article, the latter offers a better user experience and additional capabilities [40].

D. ASSESMENT OF AUDITORIUM APPLICATION FOR THE HEARING IMPAIRED PERSONS
As previously stated, one of the potential end users of the proposed wireless-AR system are hearing impaired. In order to gain insight in relation with the needs and the required specifications among this collective, feedback has been obtained from the main organization within the region of Navarra, called ASORNA (Asociación de Personas Sordas de Navarra-Association of Hearing Impaired Persons of Navarra [73]). This non-profit association has been representing the interests and needs of the hearing impaired within the region for over 50 years, working to provide equal opportunities among the hearing impaired. The proposed solution has been analyzed by ASORNA in terms of implemented functionalities and to foresee new requirements that can be implemented in the near future. It is worth noting that owing to Covid-19 related restrictions, full measurement trials with a wide array of users aren't feasible and will be studied once full mobility and social interaction are allowed. In relation with the requirements in order to enhance the experience of hearing-impaired persons with auditorium related activities, the following items have been identified: • If available, there is only 1 person performing sign language translation, which can be limited in the case of large shows with multiple actors in stage.
• Subtitles are available and different functionalities should be available, such as color change or the use of VOLUME 9, 2021 icons in order to consider different performers as well as elements such as mood or tone of the conversation.
• There is the risk that the sign language translation can become effectively a monologue, which strongly depends on the skills of the sign language translator. This is related with the need to provide high quality sign translation services in order to provide true and universal accessibility.
Taking into account the previous considerations, the proposed application is seen in qualitative terms to provide an alternative to existing solutions, which in the case of auditoriums is limited to the presence of one person to perform sign language translation and/or fixed subtitle settings.

III. RADIO CHARACTERIZATION OF THE AUDITORIUM
In this section, the auditorium of the Baluarte Congress Center building is presented and analyzed in terms of radio electric characterization. RF measurements have been performed for both the radio channel characterization (at Wi-Fi in the 2.4 GHz and 5.5 GHz bands) and interference level assessment, ensuring the good performance of the required wireless communication. Once the HoloLens' operating frequency band is selected based on the measurements, simulations by means of the in-house developed 3D-RL tool have been performed. For that purpose, a realistic scenario has been created for its simulation. The obtained simulation results have been validated comparing them to the measurements. Then, the performed radio planning tasks are presented, where the RF propagation within the whole volume of the scenario is analyzed, and the selected location for the dedicated Access Point for the HoloLens is validated. The fact of providing 3D results gains importance in such a complex scenario morphology, where the floor is not at the same level. Finally, the effect of the presence of human beings is assessed in terms of radio propagation.

A. AUDITORIUM UNDER ANALYSIS
The scenario in which the solution will be tested and the wireless channel characterization performed is Baluarte Congress Center and Auditorium of Navarre [5]. It is located in the city of Pamplona, consisting in a set of different rooms and lounges. Among the different locations, the main auditorium stands depicted in Fig. 1 is the largest, with a total capacity of 1,568 seats, divided into 1,036 in the room and 532 in the auditorium box. The number of seats varies depending on the event since some seats can be removed in order to place wheelchair users in the last row of the main room. It has dimensions of 65.26 m length × 30.58 m width × 16 m height. The seats have leather tapestry and the room is lined with beech wood. Baluarte Auditorium has lighting and audiovisual cabins in which the scenography is controlled, where the former is located in the high zone of the auditorium (in the auditorium box), and the latter is located behind the last row of the main room (see Fig. 1b). From a technological point of view, there are several Wi-Fi networks deployed within the building and the auditorium, both for the audience and for the building/event workers. Additionally, it has some mobile antennas to strengthen the reception throughout the whole building. Baluarte Auditorium is already accessible, but only for people with hearing impairments who use a Modulated Frequency system transmitter with cochlear implants.

B. MEASUREMENT CAMPAINGN IN THE BALUARTE AUDITORIUM
Before deploying any wireless device, several spectrograms were measured in the auditorium in order to assess potential interfering systems and to ensure a correct selection of the operating frequency band of the HoloLens' Wi-Fi. The spectrograms were obtained with an Agilent's FieldFox N9912A portable spectrum analyzer. In Fig. 2 the measured spectrograms for 2.4 GHz and 5 GHz bands are shown. As can be seen, the 2.4 GHz band (from 2.4 to 2.5 GHz) is almost full mainly to the existing Wi-Fi signals in the building (see Fig. 2a). On the other hand, the 5 GHz band presents less potential interfering signals (see Fig. 2b).
Since the employed Microsoft HoloLens glasses can operate at both 2.4 GHz and 5 GHz bands using IEEE 802.11ac, the information provided by the measured spectrograms could be a key factor in order to choose the operating frequency band at any specific scenario where the presented system is deployed. In the present study, the measured spectrograms seem to lead us to opt for the 5 GHz band for the developed application. But before making such a decision, measurements for the characterization of both 2.4 GHz and 5 GHz bands have been carried out in order to characterize the radio propagation within the scenario (e.g. for the assessment of the coverage).
Once the potential interfering signals present in the scenario under analysis have been detected, specific measurements for the characterization of the wireless channels have been performed. For this purpose, the Audiovisual Cabin (see Fig. 1b  Oscillator) have been used as transmitters. For 2.4 GHz, the ZX95-2500 model from Mini-Circuits has been used. Its maximum transmitted power is 8.38 dBm. For 5.5 GHz, the ZX95-5400 model (also from Mini-Circuits) has been employed. In this case, the maximum transmitted power level is much lower: -4.3 dBm. For both cases, a dual band monopole antenna has been used, which works in the ranges of 2.4-2.5 GHz and 5.2-5.8 GHz. The setup for the measurements is presented in Fig. 3.
39 different measurement points have been set within the auditorium in order to measure the RF power level at 2.4 GHz and 5.5 GHz. The same Agilent's FieldFox N9912A portable spectrum analyzer has been used for these measurements. In Fig. 4, the schematic upper view of Baluarte Auditorium is presented. The transmitter (represented by a green square) is located at the audiovisual cabin shelf, 1.55 m above the ground. The measurement points (represented by blue circles numbered from 1 to 39) are distributed along corridors and among seats, in order to characterize the whole surface and casuistry of the auditorium's room. Note that the auditorium box (the upper zone of the auditorium-the right part in Fig. 4) has not been considered (no measurements in it) since the developed application is devoted to impaired people, whose seats are always in the main room. The receiver antenna's height for the seats is 1m over the ground level, and 1.2m for the measurements at the corridors. It is worth noting that the corridors to be measured have been chosen in order to cover and characterize the both cases present in the scenario: one of the central corridors and one of the corridors near a wall. Fig. 5 shows the RF power level measurement results for both the 2.4 GHz and 5 GHz bands corresponding to the measurement points presented in Fig. 4. The detected signal level difference is due to the different transmitter power level VOLUME 9, 2021 of the VCOs (lower for the 5.5 GHz measurements) and the fact that at higher frequencies the propagation losses are higher. In general, as expected, the measurement points nearer to the transmitter receive higher RF signal levels. This phenomenon can be seen in the 5 different linear paths (from the transmitter) of measurement points that can be observed within the scenario, with numbers 1, 10, 17, 24 and 33 as initial points. In these linear paths the tendency of the detected RF power level decays with the distance, but the variations due to the multipath propagation can be also seen: there are farther points where higher power levels are detected (such as point 9, which is the furthest point of the first linear path), making the radio planning tasks complex, since it does not depend only on the distance between transmitter and receiver, but also on the specific location of the transmitter/receivers.

C. DETERMINISTIC RADIO PROPAGATION ESTIMATIONS BY 3D RAY LAUNCHING ALGORITHM
As it can be deducted from the previous measurements, radio planning within such a complex environment is not an easy task. The multipath propagation and the special morphology of the auditorium make the study very site-specific. In order to perform an accurate radio planning studio and obtain reliable estimations to validate the proposed system, an in-house developed 3D-RL algorithm has been used. This subsection presents the employed 3D-RL algorithm, the created auditorium scenario for the simulations, and the validation of the obtained results.

1) 3D RAY LAUNCHING ALGORITHM
For the successful implementation of wireless communication systems in the aforementioned scenario, a deep understanding of the physical channel by means of reliable and accurate channel models is necessary. In order to take into account, the presence and distribution of all the obstacles in the environment, as well as user distribution within it, realistic three-dimensional (3D) environments are necessary. On the one hand, different approaches in the literature are based on statistical channel modeling under certain hypothesis to analyze interference and propagation prediction in large areas with high density of nodes [74], [75]. On the other hand, stochastic geometric models have also been proposed in IoT scenarios deployment, by means of spatial and temporal techniques [76]. Besides, measurement-based techniques can also be used, with the aid of supervised learning [77]. Nevertheless, these techniques do not perform a full analysis and system operation evaluation considering the whole morphology and topology of the considered scenario. For that purpose, an in-house deterministic 3D-RL algorithm has been used in this work, based on Geometrical Optics (GO), Geometrical Theory of Diffraction (GTD) and its extension the Uniform Theory of Diffraction (UTD). The algorithm foundation is that hundreds of rays are launched from the transmitter, which propagate within the environment according to physical and optical rules. The complete volume of the scenario is considered, with all geometries and dimensions of all the obstacles in the space. Electromagnetic phenomena such as reflection and refraction are considered by means of the GO basis, while the shadow areas that arise by the edge obstacles are predicted with the UTD, by means of the diffracted rays. The presented algorithm has the potential to simulate multipath propagation, and thus the spreading characteristics of the radio channel in both time and space at the same time.
Parameters such as the transmitters' location, angular and spatial resolution, scenario discretization, system operation frequency and number of reflections/diffractions are considered as input parameters in the algorithm. When analyzing large scenarios, a commitment between results accuracy and required computational time must be achieved, due to the fact that smaller discretization of the scenario leads to more accurate results as well as higher computational time. For large scenarios, the computational time can be unaffordable if a high accuracy is required. In this sense, in order to have enough accuracy with the 3D-RL tool maintaining an adequate computational time for simulations, the optimal parameters for angular and spatial resolution have been obtained for large scenarios and it is presented in [78]. These parameters have been used for the simulations presented in this work. A detailed description of the 3D-RL algorithm can be found in [79], along with its validation for different complex environments such as urban environments [80], vehicular communications [81] or complex indoor environments [82]. Furthermore, novel approaches have also been proposed in the literature combining the 3D-RL software with different techniques to obtain accurate results while maintaining an affordable computational time for large complex scenarios. These hybrid techniques are based on hybridize the 3D-RL approach with neural networks [83], collaborative filtering [84] or the diffusion equation methodology [85]. The 3D-RL software is based on a modular structure, where the adequate methodology can be selected for the scenario under analysis.

2) MODELING OF THE SCENARIO UNDER ANALYSIS
The first step to obtain accurate simulation results consists on creating a scenario as close as possible to the real one, in terms of size, objects and their constitutive material properties. Fig. 6 shows the created Baluarte Auditorium scenario. The real dimensions and geometries of the objects within the auditorium have been taken into account, including the material properties of all the elements. The three different areas where spectators can be present are: The main room with 28 rows and 1036 seats, the box with 16 rows and 532 seats, and the VIP box with usually 8 seats. The lighting and audiovisual cabins as well as the stage have been also modeled.
As mentioned, the specific electric parameters of the materials (i.e. permittivity and conductivity) of all the objects and elements within the auditorium have been considered for the simulations. Table 1 shows the summary of the most significant materials used for the creation of the scenario.

3) MEASURED RESULTS VALIDATION
After creating an accurate scenario model for the 3D-RL simulator, a set of simulations have been launched in order to validate the provided estimations. For that validation, the simulation results are compared to the RF power level measurement results showed in Section 3.2. For that purpose, the transmitter has been placed at the audiovisual cabin's shelf at 1.55m height (see Fig. 7). RF power estimations by the 3D-RL tool at the same points (see Fig. 4) have been compared to the measured values. The simulation parameters have been set in order to comply with the values of the VCOs employed in the measurements campaign. Table 2 shows the configured 3D-RL parameters.
The obtained simulation results for both 2.4 GHz and 5.5 GHz are presented in Fig. 8. The RF power distribution in a bi-dimensional plane at the height of the row 28 is shown, as an example. The transmitter is represented by a white circle (on top of the audiovisual cabin shelf). The results show how the received power level decreases with the distance, but again, as noticed in the measurements, the rapid variations due to the multipath propagation can be observed, leading to a higher power levels in farther points. The 2.4 GHz results show better coverage of the auditorium, which is due to the higher transmitted power level (+12.7dB) and the fact that it has lower propagation losses than at 5.5 GHz. Note that the results within walls and floors have not been included to facilitate the understanding of the graphs.
As previously mentioned, the aim of these simulation results is to validate the created scenario and the provided simulation results. Fig. 9 shows the comparison between the VOLUME 9, 2021    error is higher than the 2.4 GHz error, it can be considered low, and therefore, the 3D-RL is validated for further radio planning analyses.

D. RADIO PLANNING RESULTS
Once the 3D-RL has been validated, and taking into account the existing interfering systems at 2.4 GHz band, the Wi-Fi (IEEE 802.11ac standard) at 5.5 GHz band of the HoloLens smart-glasses has been selected for the development of the application and further radio planning analysis. This higher frequency band will increase the propagation losses, but taking into account the high power level that a Wi-Fi system can provide (around 20 dBm), the performance in terms of coverage and sensitivity is not a problem at this stage.
In order to perform an in-depth radio planning study of the auditorium to check and find problems regarding the wireless channel at 5 GHz, this section presents the assessment of three radio propagation-related issues: 1. The impact of the Auditorium's morphology.
2. The impact of human beings' presence.
3. The impact of wearing the HoloLens. The assessment of these three issues has been carried out with the 3D-RL simulation tool, and is presented in the following three subsections.

1) AUDITORIUM'S MORPHOLOGY IMPACT ON WIRELESS PROPAGATION
As previously discussed, the auditoriums present a very complex morphology in terms of radio propagation. Apart from being an indoor scenario with a lot of obstacles (mainly seats), they present multiple height levels. This last fact is a very important issue, since radio channel models usually provide the estimated values at specific points, in a 1D path (dependent on distance), and some in a 2D planes. The presented 3D-RL simulator provides, with a single simulation, the RF power distribution of the whole volume of the scenario, including, if needed, the propagation inside the objects and elements within the scenario.
In order to show the impact of the morphology of the scenario, a simulation has been launched setting the parameters as shown in Table 3. The transmitter has been located on top of the audiovisual cabin shelf (marked as a white dot in the results graphs), corresponding to the potential placement of the dedicated Wi-Fi access point. Fig. 10 shows bi-dimensional planes (XY) of the estimated RF power level distribution at three different heights. Fig. 10a shows a lateral view of the created scenario, where the dashed yellow lines indicate the planes that are represented in Fig. 10b, c and d. The impact of the morphology of the scenario can be clearly seen, and how the steepness of it has a direct impact on the propagation of the radio wave.
In the same way, Fig. 11 shows (YZ) bi-dimensional planes of the estimated RF power distribution, but for (YZ). As expected, for the planes nearer to the transmitter (Fig. 11d), higher power levels are detected, but like in the other cases, the distribution is not constant, and it presents rapid variations and even zones with different power levels. It is worth noting how the power distribution within the VIP box is shown in Fig. 11c, while this cannot be seen in Fig. 11b and d due to there is not VIP box at those X-distances (in fact, it is wall, as can be seen in Fig. 10a).
All these results show how the received power level at each point depends strongly on the morphology of the scenario and the chosen location of the transmitter device. This fact leads to the necessity and the importance of using an accurate tool in order to obtain a reliable radio planning results for this kind of complex environments.

2) HUMAN PRESENCE IMPACT ON WIRELESS PROPAGATION
Another key factor to take into account in order to perform radio planning tasks in this kind of environments is the presence of human beings. The human body absorbs a significant amount of electromagnetic waves, leading to the well-known shadowing effect. In this particular type of environments, a big number of persons can be present. Therefore, the effect on the radio propagation could be very important. So far in this work, all the simulations have been performed with the empty scenario. Now, two new scenarios have been created in order to assess the impact of the presence of human beings: • Full scenario, with all the seats occupied (see Fig. 12a).
• Half-full scenario, with half of the seats occupied (see Fig. 12b). Note that the distribution of the persons follows the typical one occupying the central seats. Simulations have been performed for full and half-full scenarios, using the same configuration of the empty scenario simulations, and the same parameters of Table 3. Fig. 13 VOLUME 9, 2021 presents the RF power level difference between the empty and full auditoriums, in the surroundings of the transmitter (at the height of row 28). As can be seen, the differences in some zones can be as high as 20 dB. These differences in the received power level are very important, and could be the difference between a good performance of the wireless communication and a fall of the system. In order to have a broader view of the effect of the presence of human beings, Fig. 14 shows the comparison of received power at the measurement points shown in Fig. 4, for the cases of empty, full and half-full scenario. As expected, in general, the empty scenario presents the higher RF power levels, but there are cases where the empty and half-full scenarios present similar values, which is due to the distribution of the human beings (i.e. there are areas in the half-full scenario which are empty). On the other hand, the points located at the central block of seats present very close RF power level for the cases of full and half-full scenarios. Again, this is due to the similarity of both scenarios in such areas (i.e. full of human beings), as can be seen in Fig. 12.

3) TRANSMITTING FROM THE HOLOLENS SMART GLASSES
After studying the effect of the morphology of the scenario and the presence of persons, this subsection presents results when the transmitter is the HoloLens and the signal is received at the access point of the audiovisual cabin. This is an important point of the radio planning study, since the access points send data to the users, but the users also interact and need to send data to the dedicated access point, as will be seen in next section.
For that purpose, the transmitter antenna has been placed at the user's head, emulating the HoloLens transmitter antenna. Two different user locations are presented as examples. The first user is located at one end of row 2 (impaired people usually sit in this row because it is close to the exit doors where attendants of the assistance services are present). This case can be taken as the worst case in terms of wireless propagation (longest distance from the transmitter). The second  user under analysis is seated in row 28 (people with wheel chairs and hearing impairments usually sit in this row), which is the closest row to the audiovisual cabin and therefore, to the access point (RX). Fig. 15 shows the chosen user locations. Both cases have been simulated for the three human density cases: empty, half-full and full scenario. Fig. 16 shows the results when the user is at row 2 (transmitter is represented by a white dot). Fig. 16a presents the bi-dimensional plane at the height of row 2 (i.e. the height of the HoloLens-transmitter). Fig. 16b presents the results at the height of row 28 (i.e. the height of the access point at the audiovisual cabin). Fig. 17 shows the bi-dimensional plane at the height of row 28, which corresponds to the height of the transmitter itself and the receiver at audiovisual cabin.
The most relevant data from these simulations is the received power level at the dedicated access point located at the audiovisual cabin. These results are summarized in Fig. 18, where the received power level is presented for the two user locations (TX1-row 2 and TX2-row 28) for empty, full and half-full scenario. Due to the proximity, higher values are received from TX2. The effect of the presence of human beings can be seen again, lowering the received power (10dB or more) as the human density increases. An important fact is that although the received power is much lower when the user is seated at row 2, the obtained values are enough to provide good wireless communications with the auditorium cabin. This fact was verified when using the developed system and the application at the real scenario.

IV. DEVELOPED AR SYSTEM
The previous radio propagation analysis allows for implementing AR applications on the auditorium by using different Wi-Fi enabled AR devices. It is important to note that such AR devices differ significantly on their hardware capabilities [6], what impacts significantly the quality of the user experience. Specifically, the user experience obtained through dedicated AR devices is significantly superior to other platforms due to their dedicated visual display and processing hardware.   Among the available dedicated devices, the implemented AR system is based on the Microsoft HoloLens smart glasses, which are currently the most advanced commercial AR device and have been previously used in multiple scenarios [56], [57], [59], [86]. Microsoft HoloLens contains a custombuilt Holographic Processing Unit (HPU 1.0), 2 GB of RAM and 64 GB of Flash, embedded sensors (ambient light sensor, accelerometer, gyroscope, magnetometer), see-through holographic lenses, automatic pupillary distance calibration and human-machine interfaces (voice support, gaze tracking, spatial sound and gesture recognition). It's worth noting that the smart glasses can be connected to other devices through Wi-Fi (IEEE 802.11ac) or Bluetooth 4.1. The previously mentioned components are depicted in Fig. 19.
The main objective of the developed application is mainly to help hearing impaired people during shows and thus enhance their experience. The next subsections provide details on the design and implementation of the proposed system.

A. DESIGN OF THE SYSTEM
The developed system makes uses of the communications architecture depicted in Fig. 20, which is composed by three layers: • AR Node Layer: it is composed by the AR devices, which communicate with the other layers through a wireless access point (AP).
• Edge Computing Layer: it consists of edge computing devices located in the auditorium so as to provide local processing and fast responses to the AR devices. There are two types of edge computing devices in this layer:  fog computing gateways and cloudlets [87]. Fog computing gateways respond fast, but their computing power is rather limited (e.g., their usually run on Single-Board Computers (SBCs) like Raspberry Pi). Due to their resource-constraints, fog gateways often provide services to a limited number of users, so, for a large number of attendees, several fog gateways can collaborate to distribute their computational load. In contrast, cloudlets act like local clouds, making use of hardware that can be usually found in high-end PCs, which are required for fast video/audio processing tasks like the ones that may be needed in AR applications.
• Cloud: it is usually a server or a server farm located on the Internet that is designed to process the requests that cannot be handled by the edge computing layer. In addition, the cloud provides access to remote users, who can monitor and manage the content delivered to the AR devices.

B. DEVELOPED APPLICATIONS
In order to implement the communications architecture described in the previous section, two main applications need  Web control panel to synchronize video streaming and to receive assistance calls. The video content projected through the HoloLens app is available from YouTube: [88]. The legal rights belong to their owners.
to be developed: an application that runs on the AR devices worn by the attendees and a web application for the edge computing or cloud server that provides and manages AR content. Fig. 21 depicts the software architecture of such components, emphasizing their most relevant software modules. The developed HoloLens application is based in Universal Windows Platform (UWP) in order to display the delivered content as a floating window in front of the user (as it is shown later in Fig. 24). Such a floating window can be easily scaled and placed at the most appropriate position for each user, who can adapt its size to his/her needs and to the configuration of the scenario.
The HoloLens application exchanges data with the remote server application by making use of websockets, which allow for creating bidirectional communication channels. Thus, the server not only streams content to the smart glasses, but also allows the AR user can send messages to the server (e.g., to ask the auditorium assistants for help without bothering the rest of the audience or the performers) through the developed Alert System module, which makes use of HTTP requests.
To guarantee that the videos sent from the server to the smart glasses are always synchronized with the performance, a web control panel (shown in Fig. 22) was created to allow a remote assistant to manage the videos streamed to the audience smart glasses. Specifically, the web control panel provides multiple buttons to pause, resume, go forward/backwards or change the video source. The system allows for playing both stored videos and stream live video to all the AR devices that operate in the auditorium.
The HoloLens application is internally composed by three screens or pages: WelcomePage, MainPage and VideoPage. In the WelcomePage the user chooses his/her preferred language (English, Spanish, Basque or French). Then, a short video welcomes the user to Baluarte Auditorium and then the MainPage is displayed. Such a page contains the main menu, where the user can access the sign language video service, the show information menu (which usually includes a short video, a summary and the program of the show) and the assistant call button. Finally, the VideoPage is used to display a previously stored video (which may include subtitles in different languages) or to stream live video. Fig. 23 shows a sequence diagram that illustrates the internal operation of the smart glasses when they exchange data with the server. Such a diagram describes the main steps of the moment when a user interacts with the HoloLens application to watch a previously stored video. Thus, the user first clicks on the 'View simultaneous translation' button in the VOLUME 9, 2021 MainPage, which loads the VideoPage. While a video presentation on the auditorium is played, a websocket connection to the server is created on the background. Once the connection is opened, the application asks for the video information on the show, which may indicate that the HoloLens application has to load a video from a specific source (and its corresponding subtitle files) or that the AR device has to request a live video streamed from a specific URL. When loading a stored video, an assistant can make use of the video control web panel to select the video source or to control video playing time, which is performed by sending messages through the websocket connection.

C. IMPLEMENTED FUNCTIONALITY
The developed AR system was tested in Baluarte's Auditorium positioning the user in different locations (among them, at row 2 and row 28) in order to test how the HoloLens app perform from different view angles (one of the moments during the experiments is shown in Fig. 24a). Once the user starts the app, it presents the menu language (in Fig. 24b). The language can also be selected later by using a dropdown menu that is available in the main page (this is shown at the top on the right in the floating windows in Fig. 24c). The application provides users with a 3D virtual space where they can visualize useful multimedia content. For instance, a video stream of a simultaneous sign language interpreter can be displayed (as illustrated in Fig. 24d, where the display is placed on the right of the stage) or subtitles can be enabled and shown at the bottom of the virtual screen (an example is shown in Fig. 24e). In addition, the application is able to provide dynamic information about the performance (like the one shown in Fig. 24c on the right). Thus, the traditional show program that is handed over to the attendees when entering the auditorium can be replaced by a digital and therefore greener alternative that can be modified dynamically even during the show.

V. CONCLUSION
This article presented the design and implementation of an AR system dedicated to help and enhance the user experience of impaired people in auditoriums. As an example of the capabilities of the presented system, an AR application for Microsoft HoloLens able to help hearing-impaired people during shows in Baluarte Auditorium has been developed, although the application is versatile and can be easily modified for mobility and visual impaired people necessities. To evaluate the proposed system, an assessment of the wireless channel for both 2.4 GHz and 5 GHz operation frequencies were carried out. Measurements within the auditorium exhibit high interference levels at both frequency bands due mainly to the Wi-Fi networks of the auditorium. In order to gain insight in the radio channel behavior, simulations by means of the in-house developed deterministic 3D-RL algorithm as well as RF signal propagation measurements have been performed with the aim of analysing the effect of different relevant issues related with the radio propagation within this complex environment, such as the impact of the morphology of the scenario, the presence of human bodies and the effect of wearing the AR device (Microsoft HoloLens smart glasses) at different locations of the auditorium. Although the experiments performed show that the presented system and application work is feasible, as a future work an analysis of the system performance with several HoloLens operating at the same time is required. This realistic fact will increase the number of users connected to the dedicated access point, augmenting the wireless data traffic. In order to deploy the system in a real use case scenario, the AR/MR application should be tested empirically on impaired people with a special focus on their user experience and improved autonomy. Current COVID-19 pandemic has derived into relevant restrictions that involve strict safety measures when making empirical trials which prevented us from carrying out thorough tests on the AR/MR proposed application, which will be further evaluated in the future.
Anyway, the obtained overall results provide useful guidelines to deploy novel AR/MR systems in scenarios that are as complex in terms of wireless propagation as large auditoriums.
Future work will be conducted in order to assess different aspects such as non-uniform user distributions, heterogeneous interference sources, the impact of bandwidth handling or end to end latency measurement, among others.
I. PICALLO received the bachelor's degree in telecommunications engineering from the Public University of Navarre (UPNA), in 2016, and the master's degree in telecommunication engineering from the Charles III University of Madrid (UC3M). He is currently pursuing the Ph.D. degree in communication engineering with UPNA. He worked with Zener Group Company as a Developer and is also working, on several public and privately funded research projects. He has over ten contributions in indexed international journals and conference contributions. His research interests include radio propagation in inhomogeneous environments, human body interference analysis, wireless sensor networks, IoT networks, and 5G communications systems. He has been awarded the ISSI 2019 Best Paper Award. He has worked in ten different public and privately funded research projects. In 2019 partly worked as a Researcher for Tafco Metawireless. He has over 150 contributions in indexed international journals, book chapters, and conference contributions. He is currently affiliated with the Institute for Smart Cities (ISC), UPNA. His research interests include radio propagation, wireless sensor networks, electromagnetic dosimetry, modeling of radio interference sources, mobile radio systems, wireless power transfer, IoT networks and devices, 5G communication systems, and EMI/EMC. He has been awarded the ECSA 2014 Best Paper Award, the IISA 2015 Best Paper Award, and ISSI 2019 Best Paper Award.