OmniScent: An Omnidirectional Olfaction-Enhanced Virtual Reality 360° Video Delivery Solution for Increasing Viewer Quality of Experience

The introduction of multiple sensorial effects to improve viewer multimedia experiences has resulted in the novel field coined mulsemedia. Virtual Reality (VR) is one of the recent enabling technologies for mulsemedia distribution and solutions can be enhanced to include scent dispensers, haptic devices and other equipment, for creation of immersive user experiences. In the past, a number of research works and real-life implementations have integrated smells and visual media. However, the scents are generated from a single location around a user, which can be inaccurate in the case of 360° VR experiences. This paper presents the architecture, principle, implementation and a feasibility study for OmniScent, an omnidirectional olfaction-enhanced VR 360° video delivery solution. The approach integrates four olfaction devices placed around a user, integrated with 360° videos. Scents are released during the video playout in the same direction the related objects appear in the video (e.g., front, back, left or right). An omnidirectional olfaction-enhanced video player was implemented using the WebXR technology, allowing the videos to be seen on both Web browsers and VR headsets. User testing demonstrated that over 70% of experiment participants noticed that the diverse scents, including apple and spices, were presented from multiple directions. Performance tests in terms of Round Trip Time (RTT) and resource consumption indicate that the solution performs well in commercial PC setups.

media is becoming part of our daily life.Numerous works highlight that experiences integrating multiple human senses increase users' perception of immersion and perceived Quality of Experience (QoE) in VR applications [4], [5].While the sight and hearing senses have been part of multimedia content delivery for decades, other senses such as touch and smell can be integrated to VR experiences via a number of recent haptic and olfactory technologies [6], [7].These technologies can be employed not only to the delivery of immersive audiovisual experiences but also to educational projects [8], product design, manufacturing and marketing [9].
The human nose is a very sensitive organ, which can identify a varied range of smells.This process of discriminating scents triggers defense, fight or flight and awareness mechanisms in humans, also helping in remembering events, as scents are connected to human memory [10], [11].
Even though olfactory devices have been integrated with VR experiences in the past, scents are released to users from a single dispenser, regardless of the direction that the triggering scene occurs in the video [12], [13].For instance, if a scent is released in front of the user while he or she is turned back to the dispenser (e.g., looking to a VR scene in the back) the intensity of the scent effect will be decreased.In order to mitigate this issue, multiple olfactory devices must be used around a user in VR experiences, replicating the directions of objects in the 360 • videos and also increasing the number of available scents.Even though wearable olfactory devices exist, they are uncomfortable and provide a limited number of scents at a time [14], [15], [16].
Omnidirectional scents can also be used as a cue to help users to look into important areas of the videos.For instance, a specific scent might be generated by a dispenser situated in a particular location to attract attention to the user to a direction they otherwise would not focus on.
Based on these principles, this paper introduces OmniScent, a novel approach for omnidirectional olfactory immersive VR experiences consisting of 360 • videos, VR technology and scents being diffused from the multiple directions surrounding users (e.g., front, back, left and right).OmniScent aims to address the limitations of employing only one olfaction device when watching 360

II. RELATED WORK
This section presents a review of works and their limitation, which inspired the design of OmniScent.
A new concept called as neurally-inspired VR, which combines VR and mulsemedia was described in [17].The authors highlight limitations such as bandwidth constraints when there are multiple sensory devices to be synchronized at the same time with the 360 • videos.Authors in [18], [19] also affirm that playback of 360 • content in high quality can cause network latency due to limitations in bandwidth.Adaptive network mechanisms are identified as potential solutions for 360 • video content streaming, however the compression of video content and decrease in video resolution can negatively impact users' QoE levels [20], [21].
In order to increase QoE when streaming compressed videos due to their large file sizes (e.g., 360 • videos), authors in [22] investigated the use of scents.The experiment demonstrated that users' perceived QoE is maintained even when video quality is decreased if there is the presence of scents.In [23], authors presented the positive subjective evaluation of the level of enjoyment experienced by users regarding an adaptive multisensorial olfactory media solution.
A 360 • mulsemedia solution was proposed in [24] containing a prototype to let users experience the immersive 360 • content with a single dispenser.During the experiments on the users, it was found that higher video resolutions do not always mean high user perceived QoE and sensorial effects benefit QoE metrics.
The work presented in [25] introduces an architecture and a framework for mulsemedia systems.The framework supports communication and connectivity protocols, multiple standards and the use of design patterns.The deployment contains a single olfactory device in the front of users.
In [26], a testbed was introduced for the evaluation of the QoE in a mulsemedia-enhanced 360 • video experience.Authors acknowledged that unidirectional olfactory scent diffusion (i.e., using only one dispenser) is a limitation when videos are omnidirectional, misrepresenting the position of objects in the 3D space.Participants preferred the video clips with sensory effects, even with only one dispenser being used.
A newly-designed multi-sensory-based adaptive content delivery solution was deployed in an educational learning environment [27].The solution was tested when delivering real educational content and the evaluation demonstrated improved learner satisfaction.
The authors in [28] proposed a portable system that includes a VR headset, a low cost wearable electroencephalography (EEG) headband and an olfactory necklace in order to improve users' perception of relaxation when watching 360 • videos.The necklace, however, only supports one scent, which was lavender in the experiment.The study demonstrated that there was an approximate 25% increase in the level of user relaxation based on the EEG headband.
Convolutional Neural Networks (CNN) were applied in scene and action recognition for the automatic generation of scents in [11], [29].These works process 360 • video frames, identifying the exact times that certain elements appear in the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
videos, relating these elements to smells or haptic feedback.The authors aimed to simplify the process of adding timestamps to videos in order to add sensorial effects, leaving this task to the CNNs.Comparative studies demonstrated that ResNet18 provides the best balance between accuracy and complexity.The testbeds of these works, however, only contain a single scent dispenser and were not tested with users or in terms of QoE.
In order to optimize bandwidth consumption for 360 • videos, a method proposed in [30] reduces video quality in areas where the viewer is not looking at.Authors employ the Equirectangular projection (ERP) as the 360 • video projection mechanism in the tests.This is because the cube map projection, the other major 360 • video projection mechanism, causes an over-sampling issue within the edges of tiles that form the 360 • video.The proposed method reduces the average video bit rate by 75%.
WebXR technology is a key enabler for the streaming of 360 • content on PCs over Web browsers and also on VR headsets [31], [32], [33].It is able to render 3D scenes with motion and orientation tracking.WebXR detects when a VR headset is connected to a PC with an active WebXR application, and streams the content to the headset.VR headsets with built-in browsers can also open WebXR application directly, without the need of an external PC. 360 • video content can be streamed over Web browsers and VR headsets with a dedicated WebXR-based player [34] which supports ERP video projection.
The papers presented in this section indicate that the dissemination of 360 • videos is challenging due to file size, network limitations and multitude of existing devices.Two ways of addressing this issue include the use of ERP video streams via Web browsers, and the use of multisensorial effects to increase user perceived QoE and mask eventual decreases in video quality for seamless streaming.The introduction of scents as the sensorial effect for immersive videos is another challenge to be overcome, as multiple scents must be supported and the scents must come from the same directions that the triggering objects or scenes appear in the 3D space.Therefore, this paper addresses these challenges by introducing OmniScent, a solution which employs WebXR for the delivery of 360 • ERP content over the Web, supporting multiple devices, and increases user QoE by adding multiple omnidirectional scents via a novel architecture and testbed.

III. SOLUTION DESIGN
OmniScent's architecture is illustrated in Fig. 2. The architecture of the solution consists of the following components: the rich media content delivery process involves an Application server hosting the 360 • video player and the olfaction controller; the olfaction devices, which include scent dispensers and their respective PC controllers; and the client devices (i.e., standalone VR headsets or PCs) with their Web browsers supporting WebXR.Even though not part of the solution, the architecture supports the use of a Quality of Service (QoS) based traffic control unit for adaptive delivery of content based on network metrics, while a slice-based dynamic traffic control

TABLE I APPLICATION SERVER SPECIFICATIONS
can be deployed in the level of the provider (e.g., via 5G) for prioritization of data packets containing the omnidirectional mulsemedia content.
The following subsections detail each component of the architecture.

A. Application Server
The application server runs Apache Tomcat, which hosts the 360 • video player and the olfaction controller, both developed with JavaScript and Java.The server also hosts the 360 • video files (e.g., in MP4 format).The server technical and network specifications are presented on Table I.
1) 360 • Video Player: The WebXR-based 360 • video player makes use of the VideoJS-XR plugin [34], therefore client PCs automatically detect and output immersive content to VR headsets (e.g., Oculus Rift) connected to them.Videos can also be played on regular monitors via Web browsers.The player communicates with the olfaction controllers, which trigger the scent dispensers.
The player keeps track of users' actions, such as play, pause, fast forward and rewind, as well as the current timestamp in videos, via the JavaScript event "ontimeupdate".Based on the actions and pre-defined timestamps, the player invokes the olfaction controller via Java Servlets.Timestamps are located in a separate JavaScript file, and they trigger the appropriate olfaction controllers even if users fast forward or rewind videos.Timestamps also contain information of the specific dispenser that must be triggered and the effect duration (e.g., activate left dispenser at 1m03s for 10 seconds).
The 360 • videos are located at the same server which runs the player.These videos are encoded in the ERP format.Platforms such as YouTube store immersive videos in the Equi-Angular Cubemap Projection (EAC), requiring a conversion process.For any conversion, FFMPEG [35] can be used.
FFMPEG is an open source video/audio converter software capable of converting videos from the EAC format into ERP.
2) Olfaction Controller: The olfaction controller is defined in a JavaScript file located in the application server.It controls all olfaction devices located in the local network.
The scent timestamps file located in the 360 • player indicate which dispenser must be triggered and the effect duration.This process is illustrated in Fig. 3.When the video timeline reaches a point containing a scent timestamp, a JavaScript function performs an Asynchronous JavaScript And XML (AJAX) HTTP request (as seen on Fig. 4) to the specific dispenser indicated in the timestamp.The request also contains the URL of the dispenser, the effect duration and the fan number within the device, as each device contains four fans, allowing four different scents.The dispensers are located in the same local network of the application server and invoked via Java servlets.

B. Olfaction Devices
The olfaction dispensers employed in the solution implementation are provided by Inhalio, model SBi4v2.These devices are connected to PCs via USB.The specifications of these PCs and their network are presented on Table II.Four devices are employed in the solution, as seen on Fig. 5.Each dispenser supports four scent cartridges, which means sixteen individual scents are supported.
Each PC contains a Java servlet, running as a Tomcat Web app, that receives commands from the olfaction controller located in the application server.The servlet receives the fan number to be activated, as well as the duration of the effect.
Inhalio provides a Windows Dynamic-link library (DLL) which is installed on each PC connected to a dispenser.This DLL is responsible for activating the devices when requested by the Java servlet.
An object of a Java class is instantiated, executing the DLL of the USB olfaction device and turning on the requested fan.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.The wind generated by the fan towards the cartridge releases the scent in the direction of the user.
As seen in Fig. 5, the left, right and back olfaction fans are connected via USB to PCs which communicate with the application server via Wi-Fi.The Tomcat server handles the front device, with a Java servlet on the localhost.

C. Client VR Devices
Web browsers are the main component of the client devices.The browser-based 360 • video player, which renders ERP content, can be visualized on PC Web browsers and auto detects and streams content to VR headsets, such as the Oculus Rift, via a HDMI cable.The URL of the player can also be accessed via built-in browsers on standalone VR headsets (e.g., Oculus Quest), as long as they are compatible with WebXR.

IV. PERFORMANCE TESTING
In order to test the network performance and processing requirements of OmniScent, two different experiments were performed in terms of Round Trip Time (RTT) and CPU usage of all devices.Wireshark [36] was used to monitor RTT from the application server to the left, right and back PCs that control the olfaction dispensers.
OmniScent was tested with six 360 • videos with highly diverse content.They present a city scene (Video A), an ice cream shop (Video B), shots taken in a forest (Video C), images from an apple orchard (Video D), a food market (Video E) and footage taken on an ocean beach (Video F), respectively.The six 360 • videos were downloaded from YouTube [37], [38], [39], [40], [41], [42] in the EAC format and were converted to ERP using the FFMPEG converter.The videos were presented to viewers sequentially, with a total duration of 4m55s.Table III   Screenshots of the six videos are presented in Fig. 6.Each video is related to a predominant scent: (a) diesel for the city video; (b) tutti frutti for the ice cream shop video, (c) oak for the forest video, (d) apple for the orchard video, (e) Spices for the food market video and (f) ocean for the beach video.OmniScent was employed and the corresponding scents were released from the appropriate dispenser.This selection is made such as the smells will approach the viewers from the direction the main elements related to these scents are located in the 360 • space of the VR video.Table III lists the type of scent associated with each video and the direction from which the scents are presented to the viewers.
The scents employed in the tests did not overlap with other scents and enough time was allowed for the dissipation of each smell after their release.Synchronization between scents and video was performed following the methodology presented in [43].The authors noted a delay between the time scents were released from the dispenser until they reached the user, which influenced user QoE.This latency varied according to scent, distance between dispenser and user, ventilation and room size.Therefore, ahead of the tests reported in this paper, each scent was tested individually.Based on our findings, during the tests, the scents were released approximately 10s to 20s before the related scene was presented to the users, as part of the synchronized video playout.

A. Round Trip Time (RTT)
In order to measure latency and network quality, RTT was measured from the application server to each of the left, right and back olfaction controller PCs, respectively.The RTT was not measured for the front olfaction PC controller, as it was co-located with the application server itself.
1) RTT Between the Application Server and Left Olfaction Controller PC: Fig. 7 shows two main spikes in RTT, related to the packets containing the requests to activate the olfaction dispensers.As indicated in Table III, the left olfaction device was requested to release tutti frutti and spices scents for 40 and 55 seconds at 00m50s and 03m06s, respectively.These scents were released during video B and video E. The RTTs for the associated data packets were approximately 40ms and 46ms, respectively.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

2) RTT Between the Application Server and Right Olfaction
Controller PC: Fig. 8 includes three high peaks in RTT, in line with the three times the right olfaction dispenser was triggered, according to Table III.Scents of diesel, oak and spices were released at 00m08s, 01m28s and 03m12, respectively, during the playout of video A, video C and video E. The RTTs of the packets related to these requests were approximately 52ms, 54ms, and 47ms, respectively.
3) RTT Between the Application Server and Back Olfaction Controller PC: Two main peak RTT values can be noted in Fig. 9.They happened at 02m40s and 04m00s, when the apple and ocean scents were released during the playout of video D and video F. The RTTs of the packets related to these requests are approximately 165ms and 135ms, respectively.
Overall, the recorded RTTs were reasonably low, indicating that operating the olfaction dispensers over a wireless network did not impact the synchronization of scents with the videos, or negatively affected the users' QoE.The RTTs of the data packets transmitted between the application server and the back olfaction controller were higher than those for the left and right PCs due to the location of the user in front of the dispenser.This position was affected by interference from the communications with the VR headset and the other two dispensers.

B. 360 • Player CPU Usage
Two additional scenarios were tested to demonstrate the resource consumption of the 360 • video player of OmniScent.The Web application was exposed to the public network via ngrok [44].This test was considered in order to measure the CPU impact of the player on a PC with or without a VR headset.The tested PC was only running its operating system and the Web browser with the video player.The specifications of the PC employed are presented in Table II.The measurements were performed on a PC outside the testbed rather Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.than the ones running Tomcat and other resource-consuming applications and APIs.Fig. 10 illustrates the CPU utilization of the player with the PC sending the video content to the Oculus Rift connected via HDMI.The average CPU usage was approximately 16%.
The CPU usage was approximately 6% (as seen in Fig. 11) when the 360 • videos were watched directly on the PC screen, without the use of an Oculus Rift VR headset.This indicates a CPU overhead decrease of 10% when the VR headset was not in use.

V. USER TESTING
User testing was conducted in order to assess the perception of viewers and their associated QoE in relation to the directional exposure to scents.This section details the assessment protocol, questionnaire used and presents the results and results' discussions.

A. Assessment Protocol
The user study was performed according to the ITU-T Recommendation P.913 [45], adapted to consider directional olfaction stimuli.Ethical approval was obtained from the Dublin City University Ireland's Research Ethics Committee in advance of running the subjective tests.
A total of 27 participants (5 females and 22 males) watched the six 360 • videos illustrated in Fig. 6 with an Oculus Rift.As already mention, the total duration of the six videos was 4m55s.
All participants were recruited via e-mail and have filled consent forms before participating in these subjective tests.A plain language statement and data management plan document detailed the testing scenario, research purpose and how data is anonymized, processed, analyzed and protected.After   signing the consent forms and clarifying any questions about the experiment, the participants answered a demographics questionnaire, followed by the videos and finally the QoE questionnaire.The proposed OmniScent approach was used as described in the previous sections.Scents were released to users according to Table III, in synchronization with the videos.The timings indicated in the table already consider the delay scents take to propagate to the users.

B. Questionnaires
A demographics questionnaire was answered by participants before the experiment started.This questionnaire contained 15 questions related to user profiling and familiarity with the technology.
In terms of demographics answers, 51.9% of the participants stated that they have experienced VR for the first time, while the remaining 48.1% mentioned they were already familiar with this technology.
Fig. 12 shows the age distribution of participants.The age of the 44.4% of participants was between 26 and 29, while Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.the age of the 29.6% of participants was between 22 to 25 years.18.5% and 7.4% of test participants were aged between 30 and 35 and between 18 and 21 years, respectively.Fig. 13 illustrates the occupation domains of the participants.84.6% of the people involved in the tests have a computer science or engineering background, while 11.5% work in finance/business and 3.8% in the field of education.
The participants represented different nationality groups, as indicated by Fig. 14. 51.9% of the test subjects were Indian, 14.8% of the participants were French and 14.8% were Pakistani.Iranians were 7.4% of the total of participants, and the remaining nationalities (i.e., Brazilian, Irish and Nepalese) represented a percentage of 3.7% each.
The following questions were answered by participants: 1) I was able to notice different smells during the videos; 2) The smells are relatable to the visuals/scenes; 3) I enjoyed the use of different smells during the show; 4) The smells were a distraction or hindrance from the visual experience; 5) The smells helped me look into the direction from where they were coming from in the videos; 6) I felt dizziness or motion sickness during the virtual experience; 7) I was able to identify transitions between the smells while watching the videos; 8) The smells helped in making the VR experience more immersive; 9) Were you able to smell apple from the front and back directions while watching the 360 • videos?; 10) Were you able to smell SPICES from the left and right directions while watching the 360 • videos?; 11) Did you, at any point in the video, notice scents were coming from multiple directions?; 12) The smells helped in making this experience enjoyable; 13) I would recommend this experience to other people.

C. User Tests Results and Discussion
Fig. 15 depicts the percentage of answers given by users to each of the 13 questions related to the QoE achieved when employing OmniScent.

TABLE IV PARTICIPANTS' ANSWERS -AVERAGES AND STANDARD DEVIATIONS
Based on Q1 it is possible to determine that 45% of participants agreed and 48% strongly agreed that they were able to notice different smells in the video.Regarding the scents being relatable to the visual contents, 60% of participants agreed and 33% strongly agreed with the statement.
Regarding specific scents and their directions, 41% of participants agreed and 32% strongly agreed that the smell of apple came exactly from the front and the back directions.Regarding the spices scent, 45% of participants agreed and 28% strongly agreed that the scent came exactly from the left and right directions.
73% of participants agreed or strongly agreed that the smells helped them to look into the correct direction they were coming from in the video.This indicates that the omnidirectional scents indeed provide valuable cues for navigation in an immersive space.
Approximately 88% of participants agreed or strongly agreed that the smells helped in making the VR experience more immersive, with over 95% of them agreeing or strongly agreeing that scents helped in making the experience enjoyable and would recommend this type of experience to other people.
Table IV presents the average scores and standard deviations of the answers provided by all users and separated according to gender, respectively.The answers are presented as a score ranging from 1 to 5 expressed on the Likert scale, which ranges from strongly disagree -represented by Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
1 to strongly agree -mapped to 5. The total average score difference between males and females is 0.44.Question 8 answers recorded the largest score difference between genders (i.e., 0.91), whereas Question 5 answers had the smallest difference (i.e., 0.14).In general, it can be concluded that on overall, the answers were consistent across males and females.

VI. CONCLUSION AND FUTURE WORKS
This paper presented OmniScent, a solution that integrates 360 • videos to multiple olfaction dispensers, mirroring the directions of immersive scenes and their related scents.
The olfaction dispensers, which represent the four horizontal faces of immersive videos (e.g., front, back, left and right), are connected via a local network.RTT analysis demonstrated low latency in communications, indicating the feasibility of such approach.The immersive player employed in the solution projects ERP 360 • video content, with a satisfactory CPU usage with or without the use of a VR headset.
A user test with 27 individuals indicates that participants are able to perceive the direction of scents and that helped them to look into the relevant areas of the 360 • video.Participants also demonstrated to enjoy experiencing OmniScent.
Future work includes expanding OmniScent to automatically detect the scenes and actions in the videos and generate the timestamps.This is considered to be achieved with an innovative solution which combines neural networks and computer vision approaches.

Fig. 7 .
Fig. 7. RTT Between the Application Server and Left Controller PC.

Fig. 8 .
Fig. 8. RTT Between the Application Server and Right Controller PC.

Fig. 9 .
Fig. 9. RTT Between the Application Server and Back Controller PC.

TABLE II RIGHT
, LEFT AND BACK PCS SPECIFICATIONS