High-Resolution Stereoscopic Visualization of Pediatric Echocardiography Data on Microsoft HoloLens 2

Three-dimensional ultrasound offers volumetric images and detailed anatomical data for medical diagnosis and treatment planning. It is a key tool in the medical field to obtain a comprehensive view of the body. Ordinary two-dimensional displays do not provide depth perception and are not suitable for representing volumetric data, necessitating the use of more sophisticated visualization methods. Virtual and augmented reality (AR) displays can be used to improve the visualization of medical images, allowing for more natural interaction with the environment. This study proposes custom software developed using the Unity3D platform to render high-resolution 3D echocardiography (3DE) on the Microsoft HoloLens 2, providing an immersive AR experience for medical professionals. This research focuses on three-dimensional echocardiography in children and uses a phantom heart model to mimic a pulsating heart. The volume rendering algorithm utilizes the ray-marching technique, enabling direct volume rendering of high-quality volumetric models. To maintain a satisfactory frame rate, a Holographic Remoting approach is employed to reduce latency and enhance network transmission speed, utilizing the resources of a personal computer (PC). The custom software developed offers an intuitive and interactive user interface that allows medical professionals to manipulate and explore 3DE images effectively. The interaction includes the ability to slice, modify the intensity range, and alter the voxel density. The experimental evaluations demonstrated that it is possible to produce high-quality real-time display with HoloLens 2 and a PC-based remote rendering system, allowing intuitive control and exploration of 3DE. Overall, this research highlights the potential of AR rendering offered through Microsoft HoloLens 2 to advance pediatric 3DE rendering for medical professionals to enhance their decision-making and understanding of medical datasets.


I. INTRODUCTION
Medical imaging plays a crucial role in diagnosis and treatment planning, providing volumetric images from various modalities such as computed tomography (CT), magnetic resonance imaging (MRI) and 3D ultrasound (US).
The associate editor coordinating the review of this manuscript and approving it for publication was Ravibabu Mulaveesala .
US imaging is a portable, noninvasive, widely available, and real-time capable imaging modality that utilizes highfrequency sound waves (2 to 18 megahertz) to generate images of internal organs [1].It is a safe alternative to imaging techniques involving ionizing radiation, making it a preferred choice for medical diagnosis and evaluation [2].At the core of the US devices lies the US transducer, responsible for transmitting sound waves and receiving echoes, which are then converted into image data [3].In the field of US imaging, accurate diagnosis and treatment depend on skilled sonographers who can mentally align 2D US images within the 3D patient anatomy.This requires exceptional proficiency and hand-eye coordination to achieve successful imaging procedures and favorable patient outcomes [4].Among the various types, the three-dimensional (3D) US, also known as the volumetric US, which is illustrated in Fig. 1, offers detailed anatomical information and visualization of volumes using orthogonal planes [1].Nowadays, both 2D and 3D US are commonly employed in clinical settings.Volume visualization, an essential technique in interactive graphics and imaging, enables the extraction of meaningful information from volumetric data, such as data representation, modeling, manipulation, and rendering.
Volume rendering encompasses a set of techniques used to create a 2D projection of a 3D discretely sampled data set.This process enables visualization of volumetric data, providing valuable insight and representations of complex structures and relationships in a three-dimensional space [5].The 3DE voxel data obtained from a US scan are composed of individual data points that signify the density at a particular location.The information is read from a file and stored in a 3D texture.Although current 2D screen displays have limitations in conveying the true 3D nature of data, recent advancements in VR and AR displays offer better depth perception and natural interaction tools.In AR, virtual objects are overlayed on the real-world environment [6], unlike VR, where a purely artificial environment is generated with or without full user immersion.
A technique is required to render virtual objects and obtain data about the actual world to experience an augmented reality application.Ray marching is a fundamental technique for direct volume rendering, which allows the production of high-quality virtual volumetric models in HoloLens 2 [7].For optimal performance on HoloLens, 3D models are often optimized for supported formats such as FBX, GLB, glTF, STL, and PLY.This helps to improve the overall performance and user experience in rendering the object.During the conversion of Cartesian DICOM files to supported polygonal formats, there is the possibility of data loss.To mitigate this, we export the raw data that retain intensity information for all voxels.The process of creating the beating heart involves rendering a sequence of raw files that is then synchronized according to the frame rate or heart rate to achieve the desired visual effect accurately.Unity3D is a popular development environment for VR and AR applications, including medical simulations.However, its native visualization capabilities for medical images are somewhat limited, focusing primarily on surface rendering.In this research, we aim to develop custom software using Unity 3D to render high-resolution 3DE stereoscopic image datasets at up to 60 Hz.We also construct user interface elements, including cutting the 3DE volume in all directions, using color functions, modifying the visible intensity range, and altering the voxel density to generate 3D images.Also, we enable volume rendering to be interactive by incorporating various transformations.

II. BACKGROUND
Augmented reality is increasingly being applied for applications in the medical field, particularly for diagnostic purposes, as numerous studies explore its potential.Prior research in the domain of augmented reality has paved the way for our current exploration.Rezende et al. [5] provided the basic idea to start the procedure as they analyzed volume rendering in HoloLens 1, which led us to work with HoloLens 2, a standalone device that does not require additional equipment.The HoloLens 2 is a head-mounted display where the holograms are generated by projecting images onto its lenses, creating an augmented reality environment with a display resolution of 2048 × 1080 pixels per eye.It boasts specific hardware specifications, including 64 GB onboard memory and 4 GB RAM, along with the 2nd Generation Custom-built Holographic Processing Unit (HPU).
Palumbo et al. [8] provide a state-of-the-art overview of Microsoft HoloLens 2 applications in the medical and healthcare context.The review thoroughly examines various studies conducted since 2019, focusing on the applications of the HoloLens 2 subfield, the functionality of the device, and the software used.
Another review summarizes the results of 44 papers on HoloLens applications in several industries, including medical and surgical aids and systems, medical education and simulation, industrial engineering, architecture, civil engineering, and other engineering fields.The review concluded that among current applications, HoloLens was the most widely applied in medical and surgical aids and systems, accounting for 43% of the existing applications [9].Furthermore, Park et al. [9] highlight that the utilization of Microsoft HoloLens 2 applications for medical auxiliary devices and systems remains relatively low as of 2021.Mojica et al. [10] introduced a holographic interface (HI) capable of visualizing 3D MRI data to plan neurosurgical procedures.The HI immersed users in a comprehensive MR scene, integrating actual MRI data and virtual renderings, enabling interactive interactions with objects within the MR environment.
Currently, there is a substantial amount of ongoing research focused on the implementation of visualization tools for medical purposes.However, most articles [10], [11] rely on CT or MRI scans due to their clarity and ease of identification of targets.These modalities require fewer image processing techniques such as filtering, edge smoothing, and segmentation [12].Conversely, in the case of the US, its greater computational complexity has limited its usage for rendering on devices like the HoloLens 2. Although some researchers have used US [4], [11], [12], [13], [14], [15], [16], it comes with certain limitations.
Costa et al. [13] explained that their application involves slicing a 3D breast volume template [12] with the input orientation.The resulting 2D US image is then transmitted back to the HoloLens 2 at a rate of 35 frames per second.This implementation is based on the HoloUS [14] application, which transmitted 2D images to HoloLens 1 at 25 frames per second.Another study [15] used 3D ultrasound rendered on a custom-made hybrid optical/video see-through headmounted display.
Background analysis reveals a scarcity of research in the US with HoloLens 2 for cardiac images [16] or HoloLens 2 combined with cardiac MRI [15].This situation prompts the undertaking of a study on framerates during the rendering of 3D and 3D animated cardiac US volume using HoloLens 2. Recent work by Maddali et al. [16] indicates that the study has been applied to 3DE images using HoloLens 2. However, specific minor issues remain unaddressed or unexplored.For instance, the performance statistics of the holographic rendering approach for different resolutions were not thoroughly examined.The application incorporates ray tracing; however, studies show that ray marching [17] enables efficient rendering of 3D fractals and faster calculation of surface normals compared to traditional ray tracing.The authors have created an application that is specifically designed to work with data from GE vivid E95 scanners and utilizes the same transfer functions or color maps as the scanner.
The approach proposed in this study offers the option to edit the color and opacity transfer functions through a user interface, which allows the end user to customize the rendering prior to displaying the images through HoloLens.This study also explores the performance statistics of the holographic rendering using the ray-marching technique and animates the heart phantom to mimic the actual heart function to give a more realistic 3DE rendering.

III. MATERIALS AND METHOD A. DATA
This study focuses on pediatric cardiology data, and no actual patient data was used in Phase I of the analysis.Instead, a phantom model was created with multiple internal   [18] used in this study to simulate heart motion that is captured using an echocardiography scanner and rendered through HoloLens 2.
targets representing papillary muscles and the leaflet-annular atrioventricular interface of different sizes, as shown in Fig. 2. The 3D phantom heart model was made from silicone rubber and molded using heated plastisol, red dye, and chalk powder.It was designed with inflow and outflow to fit a phantom dynamic heart phantom system at the Servier Virtual Cardiac Centre.Fig. 3 shows the dynamic heart phantom system set up with a pneumatic regulator that is used in a model that simulates a beating heart up to 110 bpm.Ultrasound images were acquired using the X3-1 probe on a Philips system.The acquired data are in a Cartesian Dicom format and then converted to individual raw images to load into the application.

B. AUGMENTED REALITY HARDWARE
The Microsoft HoloLens 2 was chosen as it is a stand-alone device that does not require additional equipment.HoloLens 2 utilizes a Microsoft Holographic Processing Unit 2 (HPU 2) with 64 GB of onboard memory and 4GB of random access memory.For the development environment, Unity3D 9778 VOLUME 12, 2024 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

C. Unity3D-BASED SOFTWARE RENDERING
Unity3D is widely used for creating 3D games and graphics applications and offers relative ease in building prototypes.It is also the most commonly used platform for HoloLens applications due to Microsoft's support.Unity3D development is done using the C# programming language.Unity is a cross-platform environment for developing 2D, 3D, VR, and AR games on various platforms, supporting popular VR APIs such as Oculus and OpenVR, as well as other 3D display devices through manufacturer plugins.In the medical field, Unity is used for VR training environments [19], [20], and scientific/medical visualization [21] using surface rendering techniques.Direct rendering of 3D medical images using volume rendering in Unity is desirable and has been implemented through fragment shaders [22], [23].Unity's low-level native plugin interface enables multithreaded rendering with external volume rendering libraries [22], [23], [24] for direct display in Unity.

D. DEVELOPMENT OF THE APPLICATION
The system developed in this study is an application module that acts as an interface between the input DICOM US images and the output HoloLens 2 device that would be used by a surgeon.The developed system is depicted using a simplified architecture diagram in Fig. 4. The DICOM images are converted to individual raw images and provided as input to the system.This information is processed to provide a mesh using Unity 3D, which can then be visualized on the output device, a Microsoft HoloLens 2. In addition to providing the user with the ability to view the 3D surface, a plane is developed to view the cropped version of the phantom to see internal structures.Moreover, hand tracking and object interaction are enabled to perform spatial orientation to register 3DE in physical space.Additionally, to support the precision of object manipulation and noise removal, a UI interface is developed, and voice commands are integrated.

E. 3D+T RENDERING
It is challenging to generate an accurate surface representation of data containing high-noise components, as in the case of echocardiography.Instead, a volumetric rendering approach that does not require the surface to be extracted can be utilized for the problem.Thus, a direct volume rendering (DVR) approach based on ray marching was adopted with one of the newer approaches to the ray model.It attempts to render faster than ray tracing by jumping (or marching) in fixed steps along the ray, making the time until an intersection occurs shorter.
The volume rendering algorithm used in this study is based on ray marching, which is available as a Unity plugin [21] to visualize static medical images.The image processing and rendering algorithms were implemented using C# and used the free asset of the plugin [21] from the Unity Asset Store.The plugin offers an option to import raw datasets, where the data is stored sequentially, and the header file is read to determine how the data is organized (e.g., content format, dimension, endianness, number of bytes to skip, etc.) for static volume rendering.Once imported, the datasets are converted into game objects to be compatible with the Unity platform, and a script is attached to them to render them as volumes in Unity.In this study, to implement the beating heart phantom model, a new script was added to support timeseries rendering of the echocardiography sequence through an array of data textures.The static rendering plugin [21] is then used to individually render the textures in each time frame.Visualization of the beating heart phantom model was achieved by importing all raw files in the sequence.Then, a dataset is periodically selected from the array with a wait time based on the frame rate, allowing for the animation of the beating heart.

F. REMOTE RENDERING
Initially, the static volume data was transferred from a PC to HoloLens 2 using a wireless network.However, because of the large image size, the frame rate dropped significantly, leading to a loss in performance.To maintain an acceptable frame rate, the data had to be resized.The frame rate dropped to an unacceptable 16 fps when the rendering of the beating heart phantom was enabled, which was not suitable for the application.One solution to address this issue is to increase the network transmission bandwidth and perform remote rendering to transfer the results to the display device.To achieve this, a higher computing power device, such as a PC, was utilized to calculate the view and then transfer the final view to the HoloLens 2. This approach helped  minimize latency and optimize network transmission speed.To implement this solution, the Holographic Remoting option was chosen to utilize the PC's resources for faster calculations in the app.The PC was connected to the network via Ethernet, while the HoloLens 2 was connected through a mobile hotspot to support wireless network transmission and ensure faster data transfer.

G. USER INTERFACE
One of the key strengths of HoloLens 2 in the medical field is its intuitive and interactive user interface.This software allows medical professionals to manipulate and explore medical images in a novel way.For example, the user interface provides options to cut off the data in all three dimensions, adjust the visible intensity ranges, and change the density of the voxel.These interactive tools facilitate a deeper understanding of complex medical datasets and improve the precision of diagnosis and treatment planning.Fig. 5 shows the interface developed to interact with the rendered phantom to change its transfer functions.
The cross-sectional plane is a versatile tool that can be moved and rotated to clip the volume at any arbitrary position and angle, providing a comprehensive view of the volumetric data from different perspectives.This approach allows for a more comprehensive exploration and analysis of the 3D data, enhancing the visualization and understanding of the medical images in the application.Fig. 6 shows a rendering of a 3DE of a heart phantom and a sequence of the beating heart phantom while the cross-section plane is moved towards the phantom.
The slider threshold control illustrated in Fig. 7 allows developers to adjust the appearance of the mesh in real time, enhancing user interaction and exploration.This feature could be utilized to reveal internal structures gradually as the user interacts with the 3D model.When the threshold value is adjusted for the upper and lower bounds through the slider control, specific regions of the mesh can be made transparent or opaque, providing a detailed view of complex anatomical structures.

A. PERFORMANCE METRICS ANALYSIS ACROSS DIVERSE HOLOGRAPHIC SCENARIOS
The performance assessment of the proposed rendering system included different scenarios, each designed to thoroughly examine its capabilities.These scenarios included static and animated representations of heart phantoms derived from echocardiography data, which included image volumes in their original dimensions and scaled-down versions.A tabulated summary of the results of these diverse scenarios, along with their corresponding performance metrics for holographic rendering and remoting, is presented in Table 1 with the movement of HoloLens 2.
The findings from these evaluations reveal intriguing insights into the system's behavior.In particular, it was observed that resizing the phantom data and reducing its inherent complexity led to tangible improvements in the performance of the holographic rendering.However, it is imperative to acknowledge a specific warning encountered during the assessment of the animated phantom scenario.At a data transfer rate of 150 Mbits/s with device connectivity during holographic remoting, occasional issues were reported, including instances of ''Device not found'' and ''Tracking lost.''It is essential to underscore that the WiFi setup in use boasts an expected capacity of 832 Mbits/s connection TABLE 1.The performance of holographic rendering and remoting in different scenarios is evaluated using metrics that measure the quality of the holographic experience, such as the frame rate, latency, and image quality.
on the HoloLens 2. This underscores the importance of maintaining a consistent WiFi connection with a stable bandwidth, especially within the sterile environment where these visualizations are deployed, to prevent any potential disruptions in the rendered scenes.This is particularly critical as user mobility during the visualization process can further compound the demands on network stability.

B. ANALYSIS OF GPU RESOURCE UTILIZATION FOR STATIC AND DYNAMIC PHANTOM RENDERING
Table 2 provides a detailed breakdown of the maximum GPU memory usage, GPU computational unit usage, and video engine load during both static and animated phantom rendering.These critical performance metrics were precisely measured using the GPU-Z tool, affording a broad view of the system's resource consumption dynamics.In particular, the data disclosed that the animated phantom rendering approach consumed a maximum GPU memory of 1117 MB, around 200 MB higher than the maximum GPU memory consumption observed during the rendering of the static phantom.This divergence in resource usage underlines the additional demands imposed by animated phantom on the GPU, emphasizing the importance of managing memory resources efficiently for optimal performance.Moreover, the original-sized animated phantom utilized a GPU computational unit usage rate averaging 74.2%, which is more than 2.3 times the utilization of the original-sized static phantom.This shows that rendering multiple frames will not have much significant difference in rendering using holographic remoting.

C. PERFORMANCE EVALUATION OF DIRECT VOLUMETRIC RENDERING ON HoloLens 2 WITH VARIED DISTANCES AND MEASURES
For a more comprehensive discussion of the performance analysis performed on the algorithm, the direct volumetric rendering on HoloLens 2 was rigorously examined in various scenarios involving different volumes and measures.A pivotal aspect of this evaluation was determining the highest achievable frame update rates for the HoloLens 2, taking into account the dynamic interplay between the   Based on the graphs in Fig. 9 and Fig. 10, it can be seen that the fps rate of the lowest volume is very accentuated, contrary to the larger volumes, which presented performance below 16fps.The performance of the rendering is adversely affected by the increase in volumetric data size and any sudden movement of the user wearing the HoloLens 2.

V. CONCLUSION
This study proposes a HoloLens 2-based augmented reality rendering of echocardiography datasets for volumetric visualization of the data.The proposed approach renders static and dynamic versions of the echocardiography images, allowing animations to be rendered.Through its advanced volumetric rendering capabilities, interactive user interface, and stereoscopic image rendering, HoloLens 2 offers an excellent option to visualize echocardiography datasets in an augmented reality environment.The evaluations showed that the Holographic Remote Rendering option offers better frame rates than rendering the volumes in the head-mounted hardware itself.

FIGURE 1 .
FIGURE 1.A volume rendered 3D echocardiography scan of a heart phantom visualized in 3Dslicer.

FIGURE 2 .
FIGURE 2. A 3D heart phantom made of silicone consisting of internal heart structures was used for the experiment.

FIGURE 3 .
FIGURE 3. Dynamic heart phantom system[18] used in this study to simulate heart motion that is captured using an echocardiography scanner and rendered through HoloLens 2.

FIGURE 4 .
FIGURE 4. Overall components showing the acquisition, processing, and display of the 3D echocardiography data used in this study.The proposed approach uses Unity3D for rendering the volumetric data on a Microsoft HoloLens 2 augmented reality device.

FIGURE 5 .
FIGURE 5. Implemented interface for HoloLens 2 to interact with the phantom and adjust its transfer functions.

FIGURE 6 .
FIGURE 6. Illustration of the phantom's appearance during movement across the cross-sectional plane.

FIGURE 7 .
FIGURE 7. Example images of a static phantom of size 224 × 208 × 208 rendered with various minimum and maximum intensity values set through a user interface for holographic remote rendering.

Fig. 8
shows the system memory usage in MB for the original static phantom size of 224 × 208 × 208 and the original animated phantom size of 224 × 208 × 208 × 12.

FIGURE 8 .
FIGURE 8.The amount of system memory used in megabytes for the static and animated phantom data for the original echocardiography data sizes.

FIGURE 9 .
FIGURE 9. Maximum frame rate attained at different distances for holographic rendering.
rendered volume's proximity to the device and the utilization of holographic remoting and rendering techniques.The data obtained with performance tests are shown in Fig.9and Fig.10, which are, respectively, the tests for holographic remoting and holographic rendering with the movement of HoloLens 2.

TABLE 2 .
The maximum amount of memory used by the GPU, the amount of computing power used by the GPU, and the workload of the video engine for the static and animated phantom renderings of the original size echocardiography data.

FIGURE 10 .
FIGURE 10.Maximum frame rate attained at different distances for holographic remoting.