A Web-Based Synchronized Architecture for Collaborative Dynamic Diagnosis and Therapy Planning

Clinical treatment delivery often involves multiple medical professionals, where collaborative teamwork is crucial to ensure the quality of diagnosis and therapeutic decision making. With the development of Internet of Medical Things (IoMT) and its applications in mobile health (mHealth), healthcare services can be delivered to remote users. However, a challenging situation arises when the data are volumetric and dynamic, it will be difficult to achieve real-time performance in information streaming and data dynamic rendering and synchronization over Internet, as is the case with cardiac procedures, where both the anatomy and dynamic function characteristics of the organ must be considered. To address this issue, we have developed new algorithms and a web-based collaborative software architecture to display dynamic volumetric data in a web browser to enable dynamic data sharing with remote healthcare professionals. We illustrate our system using the diagnosis and treatment planning of an atrial septal defect (ASD) repair as an example to evaluate the capabilities of our platform. In this example, a series of three-dimensional (3D) ultrasound images are registered and fused with the high-quality magnetic resonance (MR) cardiac data to provide complementary information; virtual septal occluder patches are modeled and integrated into the beating heart views to facilitate the planning of the procedure to occlude the ASD, and feedback is streamed amongst all participating professionals. Our algorithm runs on a Node.js server with WebSocket protocol to synchronize dynamic heart rendering, so all the connected users can observe the same four-dimensional (4D) display of the dual-modality heart data during the process of examining cardiac anatomy, performing functional analysis, and sharing treatment strategies across distributed geographic locations. The presented computational models and software architecture create a new vehicle for collaborative exploration and manipulation of dynamic volumetric medical datasets.


I. INTRODUCTION
Complex medical procedure planning often requires input from multiple clinicians [1], who may not be physically colocated -a situation that has recently been exacerbated during COVID- 19. Under these conditions, to optimize medical The associate editor coordinating the review of this manuscript and approving it for publication was Lei Wei . resources while making full use of the clinical experience of multiple medical practitioners and clinical doctors, a webbased collaborative teleradiology system is critically important for data exploration and feedback streaming among team members [2], [3].
Internet of Medical Things (IoMT) connects various physical spatial-restricted medical equipments and personals through Internet and can access to distributed medical VOLUME 11, 2023 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ services and resources. The rapid progress of IoMT makes personalized mobile health (mHealth) and distributed medical treatment available to remote patients [4]. Moreover, the widely accessible medical devices and senors can collect patients' data to generate insights about their health information through IoMT. The ubiquitous data collecting and computing environment constructs a smart space for medical diagnosis and information sharing [5]. These Internet-based technologies create a cost-effective way to connect the remote patients with a personalized continuous monitoring system for semantic analysis of the acquired dynamic dataset and delivery of smart space based approach for emergency situations in personalized mobile healthcare [6], [7]. However, when the medical data are volumetric and dynamic, such as the case of cardiac procedures, it is still a challenging task to provide physicians who are in multiple geographical locations with real-time synchronized visual feedback and information streaming using the algorithms currently available in the traditional IoMT environment and smart space. Furthermore, due to the complex anatomical structure of the human heart and the limitations of network and visualization hardware and software, rendering large and dynamically varying datasets (such as multi-modal cardiac data) and streaming diagnostic information via the Internet remains a topic of on-going research [8]. Current web-based visualization algorithms are mainly restricted to single modality data rendering on an HTML5 canvas using WebGL [9], [10], [11], [12], where the dynamic cardiac functions and detailed structures of the atrium, ventricle, aorta, and myocardium cannot be clearly revealed [13], [14], [15]. No research has been reported on integrating virtual tools into complimentary images in conventional cardiac image rendering methods, making it difficult to deliver satisfactory visual feedback related to the exploration of internal cardiac structures [16], [17], [18], [19]. To address this need, we developed new algorithms and a software platform to synchronously display fused multi-modality dynamic medical images over the Internet [20], [21]. A particularly challenging situation arises when rendering dual-modality four-dimensional (4D) heart data with virtual tool integration, where both the anatomy and dynamic function characteristics of the organ must be considered. To the best of our knowledge, this problem has not so far been addressed in the literature. This paper is an extension of our research to address the above described limitations and challenges, which focuses on developing new medical data visualization algorithms with feature of interest enhancement in web browsers, information sharing and streaming strategies over Internet, graphics processing unit (GPU)-accelerated computational methods for parallel computation in data processing and volume rendering, mathematical models, and a unique web-based dualmodality medical data visualization and synchronization architecture for collaborative atrial septal defect diagnosis and therapeutic planning. Our platform can display dynamic cardiac images which have been pre-processed to identify their various anatomical sub-structures, along with registered 3D ultrasound images and a modeled virtual septal occlusion patch tool. All such data can be synchronized across all connected visualization workstations and mobile devices. The virtual septal occlusion tool is modeled using a raymarching algorithm and merged into the volumetric data rendering pipeline. The presented software platform employs a Node.js framework and a WebSocket protocol to create bidirectional connections and visual streaming of cardiac data display among the server and all connected client computers. Moreover, a data management algorithm runs on the Node.js server to provide users with real-time data access and information sharing without geographic restriction. The developed algorithms and models can be considered as an extension to the existing computational methods available in the Internet of Medical Things (IoMT).
The organization of this paper is as follows. Section I introduces the project background and its connections with the current medical applications as well as the novelty and importance of the presented research. Section II illustrates the related work and Section III introduces the approaches related to data acquisition and processing, as well as texture generation. In this section, the methodologies employed to generate implicit surfaces and model the virtual septal occlusion tool are described, dynamic rendering with region of interest enhancement and virtual patch integration are illustrated, and the techniques employed for visual synchronization and information sharing are introduced in detail. Section IV discusses experimental results using the presented system for Internet-based collaborative atrial septal defect diagnosis and therapy planning, and concluding remarks are outlined in Section V.

II. RELATED WORK
Traditional teleradiology systems have been developed to address the problem of transferring radiological images between medical image workstations to facilitate diagnosis, treatment and patient follow-up, training, and consultation [22]. Abrardo and Casini [23] developed a web-based solution for remote access to radiologic data and diagnostic services, while Gomez et al. [24] presented a telemedicine platform that was based on an asynchronous transfer mode (ATM) for cooperative video conferencing, image digitizing, as well as local and remote transparent database access. A similar web-based environment was presented by Lasierra et al. [25] for teledermatology support of a dynamic evaluation process with clinical personnel. Moreover, a webbased teleradiology management system for automatically transferring images and radiologists' reports was reported by Caffery and Manthey [26].
To facilitate rapid development of a non-proprietary, lowcost teleradiology solution, Puech [27] introduced a software framework for reading and working on DICOM (Digital Imaging and Communications in Medicine) images, while Shen et al. [28] developed a medical image access and presentation system (MIAPS) for remotely accessing and presenting DICOM images. In addition, a multi-functional telemedicine system that supported both telediagnosis and teleconsultation services was proposed by Lin et al. [29]. For the purpose of integrating DICOM and common Internet services into an operational domain that allowed access to all levels of the DICOM information hierarchy, Koutelakis et al. [30] implemented a modular multiprotocol teleradiology architecture, while Mitchell et al. [31] introduced a client-server teleradiology system for the diagnosis of acute stroke, which could interactively visualize 2D and 3D brain images on a smartphone device.
None these teleradiology systems however took advantage of modern Internet protocols and were limited to processing and displaying a single image modality, static volumetric data or dynamic 2D DICOM images. They all lacked the capacity to provide synchronous real-time 4D visual feedback, which restricted their application in dynamic 3D cardiac imagebased applications such as the planning of mitral, aortic, and tricuspid valve repair, ablation procedures for the treatment of atrial fibrillation, and the repair of septal defects (ASD).
Using 4D cardiac data sets, medical image computing and visualization methods can be developed to display different aspects of cardiac motion and extract quantitative motion parameters [32]. For example, Kim et al. [33] implemented a software platform for visualizing the myocardium and quantitatively evaluating cardiac MR data for ASD therapy, and Linguraru et al. [34] used real-time 3D ultrasound to guide atrial septal defect closure. Meanwhile real-time 3D echocardiographic images were employed by Suematsu and his colleagues [35] to guide beating heart atrial septal defect closure. Zhang et al. [36] designed a stereoscopic cardiac image dynamic visualization platform to provide cardiologists with real-time feedback regarding heart anatomical information, Kidoh et al. [37] reported a rendering method for motion analysis of cardiac computed tomography, and the impact of visualizing 4D flow on diagnosis and surgical planning was demonstrated by Christopher et al. [38]. However, all these described software platforms were limited to local computers without Internet-based synchronization or medical information streaming, so were unsuitable for collaborative analysis and treatment planning [39]. Moreover, while the self-gated 4D whole heart imaging technique can potentially allow cardiac anatomy and function to be assessed from a single measurement, the published literature lacks the ability to enhance various relevant sections of the anatomy (cardiac chambers for example), making it difficult to strike a balance between rendering internal structures and displaying their dynamic behavior during a network-based collaborative assessment [14].
Internet technologies can create advanced and rich webbased applications that allow radiologists to readily access teleradiology systems and view medical images remotely [8]. Through high-speed Internet connections, medical data can be communicated between distinct clinical departments, making it is possible to connect remote health care professionals. The concept of Cyber-Medicine System (CMS) is introduced and applied to research and development of Internet-based medical information systems, which can connect medical devices and healthcare services with patients and medical professionals, and provide support for assisting patients in the case of emergency and first aid [7]. In particular, the visualization of multi-modality cardiac data over the Internet can convey a significant amount of image information from various sources into a single and meaningful display [16], [40], [41], and the complementary image can facilitate timely diagnosis and identification of patients in need of surgical management [17], [18]. For instance, Qiao et al. [10] proposed an HTML5-based solution to access large-scale 3D cardiac volume, and Arumugham's research group [42] built a user interface to Local Area Network for storing and accessing medical records, while Min et al. [12] reported several applications for viewing radiological images remotely. Korzun [5] presented a detailed overview of the applications of the Internet of Things (IoT) in mobile health (mHealth) system for providing patients with customized smart services though Internet.
The concept model of a semantic layer and smart space based approach were introduced by Korzun and his colleagues for connecting patients with their surrounding equipments and for providing digital assistance for emergency situations in personalized mobile healthcare, and the introduced architectural model was used in developing the first aid assistance system for helping people with chronic cardiovascular deceases [6]. Moreover, Zavyalova and her colleagues [7] added the derived knowledge from analyzing the end-user activities to the CMS for supporting medical professionals and for delivering intelligent services to remote patients. Furthermore, an online visualization framework for cardiac data was implemented by Li et al. [15], and a webbased diagnosis system for improving the quality of cardiac surgery was introduced by Yano's team [11].

III. MATERIALS AND METHODS
In the description of our platform, we make extensive reference to the planning for an atrial septal defect (ASD) procedure as an example case of its application but emphasize that this platform is not specific to this procedure and can be readily adapted to any 3D dynamic image-based diagnostic/therapy planning procedure. ASD occurs when there is a failure to close the communication between the right and left atria, which is one of the most common type of heart defect around the world [43]. Surgical closure is a major treatment solution and can prevent functional cardiac deterioration, and is often accomplished using interventional cardiac techniques [44]. Due to the complexity and morphologic variations of the heart between patients, ASD is usually suspected clinically, so a web-based collaborative treatment platform is a valuable architecture in the confirmative diagnostic procedure for surgical repair.

A. DATA ACQUISITION
A set of T1-weighted 3D MR cardiac images of a healthy subject was acquired using a 1.5 T GE CVi scanner. The VOLUME 11, 2023 FIGURE 1. Flowchart describing the scanning of the patient and reconstructing 4D MR and ultrasound cardiac images gated with ECG signals, data storage scheme, dual-modality image registration and processing, cardiac chamber enhancement, texture construction, and dynamic texture binding.
imaging protocol employed an ECG-gated gradient echo pulse sequence and a 115 × 170 image matrix. As shown in Fig. 1, the dataset consists of twenty 3D images equally spaced throughout a cardiac cycle, each identified numerically from 0 to 19. For each image, the in-plane resolution was 1.48×1.48 mm and the slice thickness 1.5 mm.
The end-diastolic (ED) image from the 4D dataset was chosen as a reference for segmentation. This image was manually segmented to outline the morphological structures of interest: left atrium and aorta (LAA), myocardium (myo) of left ventricle, as well as right atrium and ventricle (RAV). The voxels of interest are labeled by setting the mask value to unity, while zero is used to mark the background voxels, ensuring an unambiguous differentiation during visualization. To reduce the intermixing artifacts along the edges of the segmented sub-volume, binary mask labels resulting from the explicit segmentation were filtered with the spherical Gaussian kernel to achieve a compromise between image quality and accuracy. Next, a nonlinear registration algorithm was used to register the ED image to each of the remaining images in the 4D dataset [45]. The acquired 19 transformation matrices were employed to assign the three segmented subvolumes of the ED image, corresponding to LAA, myo, and RAV, respectively, to the each of the remaining 19 heart volumes throughout a cardiac cycle.
The next step was the acquisition of the ultrasound image, which is demonstrated in the top left of Fig. 1. Using a Polaris optical tracking system to continuously track an US probe while ECG signals are simultaneously acquired from the patient, fourteen 3D ultrasound images of the same subject were acquired on a Philips SONOS 7500 US machine. The pose of the ultrasound probe is fixed during acquisition of the US images, so we can assume that each tracking transformation is represented by the identity matrix. The dimension of each acquired ultrasound volume is 160 × 208×144 with a voxel size 1.24×1.26×0.8 mm 3 .
Even though these data were acquired some time ago, using imaging equipment that is definitely not state of the art today, both systems nevertheless represent the kind of dynamic images that are acquired using current MR and ultrasound technology.

B. IMAGE REGISTRATION AND PROCESSING
As described in Fig. 1, for the 20 images in a cardiac cycle, the first one and its corresponding ultrasound image are used to initialize the registration, and the remaining images are employed for intra-procedural registration. A mutual information (MI) algorithm is applied to align the two images [46], which is illustrated by the formula in Fig. 1, where A and B are two discrete data volumes with individual probability density distributions p A (a) and p B (b).
Based on maximization of the mutual information metric, the MI algorithm iteratively transforms the floating template image until it is optimally aligned with the reference image. Due to artifacts, clutter and low signal-to-noise ratio (SNR) in the ultrasound images, when using the MI method to register 3D ultrasound to dynamic 3D MR images, the MI metric considers both the heart and background information in its FIGURE 2. Workflow and system architecture of modeling virtual septal occlusion tool using implicit surfaces and displaying them with 4D cardiac images in web browsers, during which graphics hardware accelerated algorithms including raymarching, raycasting, and optical mapping running on a Node.js server. The beating heart and fused virtual patch operations are streamed amongst all the connected computers using WebSocket protocol over ethernet or wireless network for maximum collaborative interaction.
calculation, which often results in a mismatch. Moreover, due to the large amount of information that must be processed, the convergence of MI algorithm is usually too slow for dynamic beating heart applications.
To address the issue of incorrect registration associated with the MI algorithm for registering ultrasound and MR images, and to accelerate the registration speed, a median filter is first applied to the ultrasound images to reduce the image speckles. In the ultrasound images used in this project, because most clutter and artifacts generally have a lower intensity compared with dominant features of interest, i.e., the boundaries between the heart muscle and chamber blood pools, a threshold is applied to the ultrasound images to obtain a binary mask image, which is employed to extract the filtered image, so only the most apparent anatomical features of interest are depicted and most artifacts are removed. The threshold value of 100 units within the 256 level scale is used in our system. Moreover, the field of view in the ultrasound images is restricted to highlight the cardiac structures to reduce the amount of information being processed during the registration procedure, therefore, the registration speed can be improved.
For the purpose of evaluating the registration accuracy, five anatomical landmarks were identified in both ultrasound and MR images, and their misalignment in the two registered images was calculated. The registration transformations were then applied to these landmarks to calculate the target registration error (TRE) of ∼1.8±0.9 mm. The intraprocedural transformation matrices were also applied to the segmented cardiac chambers of interest acquired in Subsection III-A, i.e., LAA, myo, and RAV for image alignment. The registered 3D MR images with segmented cardiac chambers and ultrasound images were processed by algorithms described in [21] to generate 2D montage images and then loaded to the vertex and fragment shaders in graphics memory, to render the registered multi-modality cardiac image volume in a web browser.
For each voxel in the volume V [i, j], we calculate its normal using trilinear interpolation, obtaining their corresponding normal volumes N [i, j], which are also stored as 3D arrays with names nolMR j , nolUS j , nolLAA j , nolMYO j and nolRAV j , respectively. The intensity value of each voxel in the volumes V [i, j] and N [i, j] is normalized to a range [0, 255], and post-color attenuated classification algorithm [47] is employed to create a 2D lookup table, which maps each value between 0 and 255 to R, G, B color component and opacity α, where φ ∈ [0, 255] and α ∈ [0, 1], φ = R, G, or B. These lookup tables are represented as 2D arrays T [i] for optical mapping of MR j , US j LAA j , myo j , and RAV j respectively with i = 0, 1, · · · , 4.
In our JavaScript based software platform, WebGL functions createTexture and bindTexture are used to create and bind five 3D textures using TEXTURE_3D with initial data V [i, 0], and create the corresponding five 2D textures using TEXTURE_2D with data T [i] (i = 0, 1, · · · , 4). All these textures are loaded into the GPU fragment shader in a web VOLUME 11, 2023 browser using gl.getUniformLocation to upload data values to 3D textures vol i and nol i as well as 2D textures col i respectively. Therefore, we have 5 texture sets i , where i = 0, 1, 2, 3, and 4 that are used to represent the 3D volume data textures and corresponding normal textures of MR, US, LAA, MYO and RAV respectively, as shown in Fig. 1.

D. IMPLICIT SURFACE GENERATION
For the purpose of modeling virtual septal occlusion tools used to repair the septal defect, a raymarching algorithm was designed to generate the implicit surface function f (x) (x ∈ R 3 ) in 3D space. To create an image on the virtual display window between the implicit surface and the camera, a ray r(t) = p c + r d × t (t ∈ R 3 , r d is the ray direction) is cast from the camera p c = (0, 0, c) through every pixel (fragCoord.xy) on the image plane into the scene to find the intersection point between r(t) and the implicit surface f (x), which is demonstrated in Fig. 2. If the search is successful, the color and shading of the corresponding pixel is computed, otherwise, the pixel is set to the background color.
To define an implicit surface in the scene, we assume the implicitly describes a point set P ⊂ R 3 and is strictly negative over the interior points P, therefore, it returns zero on the boundary ∂P, which forms the implicit surface of f (x) and can be represented as f −1 (0). The entire 3D scene is defined as a signed distance function (SDF), which takes in a point p i (i = 0, 1, · · · , N − 1) as input and returns the shortest distanced i from that pointp i to the surface f −1 (0) of any object in the scene. For a pointp i (i = 0, 1, · · · , N − 1) inside the rendering scene, the surface f −1 (0) is given by a distance estimator, which returns the distancẽ d i to it from the pointp i . By this means, for every point p i , we can find the distance to the closest surface in the scene.
The SDF is employed to find the intersection pointp of the casting ray r(t) and implicit surface f −1 (0). At each ray marching step k (k ≤ M − 1, where k starts from 0 and M is the preset maximum steps), if the return distanced k ≤ ε (k ≤ M − 1), the casting ray r(t) hits the implicit surface f −1 (0), then the calculated color with shading information is added to the pixel on the image plane from which the ray is cast. This pixel is then used to generate the final pixel color of the rendered image on the display window. If k > M − 1, (i.e. there are no implicit surfaces with the range along the marching ray), then the distanced k from camera to the interest point is set to -1, and the corresponding color of the pixel from which the ray is cast is set the background color. Otherwise, ifd k > ε and k < M − 1, march to the next step k + 1 along the ray r(t) untild k ≤ ε and k < M − 1 or k ≥ M − 1, and the corresponding calculated or background color is set to the pixel on the image plane from which the ray is cast.

E. SEPTAL OCCLUDER PATCH MODELING
The Amplatzer Septal Occluder (ASO) is the standard of care tool for minimally invasive atrial septal defect (ASD) closure. As illustrated in Fig. 3, the ASO is a double-disc occluder comprised of nitinol mesh and polyester material. To model a virtual ASO, the signed distance function (SDF) is employed to create implicit surfaces, which returns the minimum possible distance from pointp to the surface it describes. When the sample pointp is inside the implicit surface, a negative value is returned by SDF. The absolute distance from the sample pointp to the surface f −1 (0) is used in the process of implicit surface searching and calculation. Basic mathematical implicit surface functions are first defined, including torus, cone, round box, capped and rounded cylinders, and prism. Then, various implicit surface blending operations such as smooth union, subtraction and intersection are designed to merge multiple simple implicit surfaces to model comprehensive geometric surfaces such as the ASO tool.  between 0.01 and 0.04. The modeled ASO can detach the delivery cable under user control, which is demonstrated in Fig. 4 (c). The ASO is designed to securely appose the septal wall between left and right atria and create a platform for tissue in-growth after implantation. To accommodate different aperture sizes in the septum, our system permits the users to interactively control the patch size, as described in Fig. 5. The parameter φ ∈ [0.0, 1.5] is interactively controlled in the user interface, where φ = 0.0, 0.32, 0.5, 0.65, 0.86, and 1.08 are used to create the corresponding virtual ASO patches in (a) -(f) respectively.

F. DYNAMIC CARDIAC DATA RENDERING AND FUSION
Inside a web browser, vertex and fragment shaders supported by WebGL2 are used for raycasting calculation and data rendering inside the graphics memory. The variables listed in Table 1 are used in the description of Algorithm 1. The texture set tex i , i = 0, 1, · · · , 4, represents 3D texture generated from dataset of magnetic resonance (MR), ultrasound (US), left atrium and aorta (LAA), myocardium of left ventricle (MYO), and right atrium and ventricle (RVA) respectively.
We assume r = r(t) = ray(x(t),y(t),z(t)) is a viewing ray cast from the screen into texture space, where s(t k ) = s( r(t k )) and n(t k ) = n( r n (t k )) are the intensity value and normal at the sampling point r(t k ) on the ray r(t), which is acquired by sampling texture vol i and nol i using texture coordinate at r(t k ). At each point r(t k ), the 2D texture generated from lookup table colMap i maps the intensity value s( r(t k )) to color C i (t k ) and opacity α i (t k ), which is acquired from sampling texture col i using the texture coordinate at r(t k ), where i = 1, 2, · · · , 4 represents 2D texture generated from dataset MR, US, LAA, MYO, and RVA respectively.
Eq. (1) represents the alpha blending used in computing the accumulated color and opacity, which is based on volume rendering integral.
The pre-multiplied α i (t k )C i (t k ) represents the total light emitted from the point, and r(t k ), T i (t k ) = 1 − α(t k ) describes the transparency of the point, where α i (t k ) ∈ [0, 1] is its opacity, and blψ i ∈ [0, 1] is a blending factor, which can be interactively adjusted. Here i = 0, 1, · · · , 4 represents the ray-casting calculation for MR, US, LAA, MYO and RAV respectively. During this process, for every ray r(t) cast, the computation based on Eq. (1) must be conducted at each sampling point r(t k ). To add sharing in our rendered cardiac volume, we set the light direction to ltDir, the ambient light color to ambLt, and the diffuse light color to difLt. Then, at each sampling step r(t k ), the improved Phong optical model is used to update the calculated color C i (t k ). VOLUME 11, 2023 For each casting ray r(t) described by Eq. 1, the calculated value I i includes both RGB color I c i and opacity I α i . We set a threshold µ = 0.95, when 4 i=0 I α i ≥ µ, the calculation is stopped on the casting ray, and brightness adjustment factors br i are added to color, i.e., I c i = I c i × br i . The final color of the fused heart volume generated by the casting ray r(t) is 4 i=0 I i . The raycasting computations are executed in parallel on the GPU, so for each pixel in the display window, a ray is cast into texture space, and the raycasting calculation is conducted simultaneously on all the casting rays, enabling the final image to be generated in real time.
During the data visualization process, the patient's ECG signal is digitized and used to adjust heart rate through updating the 3D textures in WebGL function texImage3D from 19] in one cardiac cycle, and from V [i, 19] to V [i, 0] in the following cardiac cycle. This process is then repeated. Users can interactively manipulate the lookup table editor to adjust color and opacity mapping through updating T [i] in WebGL's texImage2D function. Therefore, during the rendering process, the texture values can be dynamically updated in a GPU fragment shader and thus the rendered heart beats synchronously with the acquired ECG signals.
For each casting ray r(t) described by Eq. 1, the calculated value I i includes both RGB color I c i and opacity I α i . We set a threshold µ = 0.95, when 4 i=0 I α i ≥ µ, the calculation is stopped on the casting ray, and brightness adjustment factors br i are added to color, i.e., I c i = I c i × br i . The final color of the fused heart volume generated by the casting ray r(t) is 4 i=0 I i . The raycasting computations are executed in parallel on the GPU, so for each pixel in the display window, a ray is cast into texture space, and the raycasting calculation is conducted simultaneously on all the casting rays, enabling the final image to be generated in real time.
During the data visualization process, the patient's ECG signal is digitized and used to adjust heart rate through updating the 3D textures in WebGL function texImage3D from 19] in one cardiac cycle, and from V [i, 19] to V [i, 0] in the following cardiac cycle. This process is then repeated. Users can interactively manipulate the lookup table editor to adjust color and opacity mapping through updating T [i] in WebGL's texImage2D function. Therefore, during the rendering process, the texture values can be dynamically updated in a GPU fragment shader and thus the rendered heart beats synchronously with the acquired ECG signals.

Algorithm 1 Dual-modality data visualization and enhancement with integration of a virtual instrument
.rgb×(brMR−1.0)+val [1].rgb×(brUS−1.0) 58: C f ← C f +val [2].rgb×(brLAA−1.0)+val [3].rgb×(brMYO− 1.0) 59: C f ← C f + val [4].rgb × (brRAV − 1.0) 60: if brTran > 1.0 then 61: The items (a) -(i) explain Algorithm 1, which uses variables volFlg and toolFlg to control the display of the image data, virtual tools, or both interactively. Transform and scale camera and 3D texture positions to get the casting ray direction r d , which is used to make the casting ray r and the function calIntersect( r, ) is used to find the intersection points d 0 and d 1 of the casting ray r with the texture unit cube , which is illustrated in lines 9 -10. If toolFlg is true, use camera position p c and casting ray direction r d to calculate the ray-marching color and opacity C s and α s , then find the distance d s from the camera to the implicit surface in lines 11 -13. (d) As demonstrated by lines 14 -19, when d 0 > d1, i.e., there is no intersection between casting ray and texture volume, if toolFlg is false, i.e., no implicit surface rendered, set C f and α f to the initial values and stop sampling the current ray. Otherwise, use the product of ray-marching color C s and brightness adjustment factor brSur as the final output color C f . (e) When d 0 ≤ d1, there are two cases to consider for data and implicit surface rendering. -The first case (lines 20 -52) is when the rendering volume, i.e., volFlg is true. Calculate the start and end points p s and p e of the casting ray. If toolFlg is true and the distance from camera p c to the implicit surface is less than the distance from camera to the casting ray start point p s , calculate the final color C f : lines 23 -25. As shown by lines 26 -48, at each sampling point p i (i = 0, 1, · · · , Num -1), if toolFlg is true and the distance from camera p c to the implicit surface is less than the distance from camera to point p i , calculate the final color C f through blending C f and C s using α f and stop the current casting ray. Next, calculate the mapped color col[j] using texture sampling of the 2D color mapping texture colMap j , where j = 0, 1, · · · , 4 corresponding to MR, US, LAA, MYO, and RVA respectively, and then acquire the corresponding normals nol[j] using 3D texture sampling. As explained by code in lines 49 -51, calculate the lighting value at point p i for each of the five data modalities, and compute the accumulated color and opacity of each data modality and final color and opacity. If α f is greater than the preset threshold θ, stop the current casting ray, otherwise, continue to loop to the next sampling point. When the casting ray is stopped, check whether toolFlg is true and d s is greater than the distance from camera to the end of the casting point p e . If it is true, blend the implicit color C s with the final color C f on the current casting ray.

H. VISUAL SYNCHRONIZATION AND INFORMATION SHARING
This software is developed on a web-based framework, which is based on a client/server architecture and bidirectional connections are established between server and all connected client computers. The Node.js server and WebSocket protocol ensures persistent connection and information sharing. Fig. 2 outlines the system architecture of the server side programming and the two-way interactive communications using WiFi gateway wireless access points or ethernet for wired networks. The server IP address and port 192.168.0.21:4000 is the IP and port of our server in the study used for client computer connections.
Using an event-driven scheme, our software platform can synchronize web-based data visualization, virtual instruments, and message transformation, allowing users to share the same dynamically updated streamed data. Algorithm 2 describes server side programming and operations, where FIGURE 6. Demonstration of the software platform's information streaming and data storage, in which users can input their comments regarding atrial septal defect diagnosis and treatment planning that can be shared among all the connected users, i.e., seven users' platforms are displayed. All the messages are stored in a MySQL database and can be extracted and displayed in web browsers. line 1 shows the Node.js web application framework express, used to provide a set of features for web and mobile applications to monitor all the events at the designated server IP and port address. The event listeners are added to all the interactive panels, buttons, menus, and data rendering windows of our software platform, and when any of the listeners receive events or messages from user input or from a connected server, the software updates the data rendering, patch manipulations, message display, and other operations according to the commends contained in the events or messages. As shown in lines 2 -3, the Socket.IO library runs on a Node.js server to build real-time, bidirectional communications between the server and connected clients. The created server object io is employed to listen to all events and receive all messages. At the same time, the function io.sockets.on broadcasts the received events and messages to connected clients in real time using the function socket.broadcast.emit, which controls all the events and message data from 1 to N as demonstrated by in lines 5 -11, where N is the total number of events and messages.
We have also developed a message streaming scheme for data processing and information sharing, which is outlined in Fig. 6. All the input information and messages from the graphics user interface are automatically streamed into a database and synchronized amongst all the connected clients. Seven user platforms are listed 1 , 2 , · · · , 7 , user input messages such as name, id, email address, phone number, and diagnostic comments are shared among all the connected client computers in real time, and stored in a MySQL database running on an Apache HTTP server, which shares the same hardware platform with the Node.js server. All the saved data can be extracted and displayed in a web browser when user clicks the ''Display User Feedback'' button from any connected computers.
The algorithm for data storage and streaming described in Fig. 6 is explained inthe Algorithm 2 panel above. As demonstrated by line 4, mysql library is employed to create an object, which is then used with express object app to conduct data post and get operations. As shown by lines 16 -19, app.post is used to acquire all user input messages from the client computer's user interface and save them to a database table. The mysql library creates a connection between the Node.js server and all the connected clients based on host and user: lines 20 -22. At the same time, as described in lines 23 -26, all the saved data can be extracted through the mysql.createConnection function and displayed in a web browser using jason on client computers, which connect via the Apache web server using the WebSocket protocol. The Apache web server operates on a Node.js framework that is supported by the Socket.IO library.
The received events and operation are forwarded to the Node.js server through function socket.emit using bidirectional WebSocket protocol in real time, and the server broadcasts the received messages to all the connected client computers using the Socket.IO library running on the server side, as presented in lines 4 -6. Therefore, all the system and data rendering operations are synchronized by the Socket.IO library running on the Node.js server as well as all the client computers connected with WebSocket protocol, in real time.

IV. RESULTS AND DISCUSSION
An Intel 9900KF CPU, 64GB DDR4 memory, and equipped with four Nvidia graphics cards was used in the study to evaluate the system performance. As an illustrative example, we tested the dual-modality 3D rendered beating heart with chamber enhancement and virtual ASD repair instrument integration using the Chrome, Edge, and SeaMonkey in web browsers. The Node.js server was hosted on a HP Spectre x360 Laptop connected to the client computers using a wireless network that has ∼100 MB download and ∼20 MB upload speed. Using this platform, users can interactively manipulate the 4D image brightness, blending effect, virtual tool position and display transparency, and volume clipping.

A. DISPLAY RESULT WITH VIRTUAL OCCLUDER PATCH
As demonstrated in Fig. 7, to outline the structures of interest, the end-diastolic (ED) image from THE 4D dataset is chosen as a reference for segmentation. Three chambers (a) LAA, (b)myo, (c)RAV along with (d) the combination of all three, are chosen for segmentation and visualization. VOLUME 11, 2023   Each is transformed into the 19 cardiac volumes using the matrices acquired from nonlinear registration as described in Subsection III-B. All these outlined chambers are merged with the entire cardiac data rendering pipeline.  (ai) shows the cardiac MR display, (bi) the virtual SOP merged with the MR image for planning the location of the patch to effect the repair, while in (ci), the transparency of the displayed heart is increased by adjusting the virtual tool factor TransSurf from 0.98 to 1.35, and keeping the factor TransSurf value constant in the following images, so the virtual SOP can be seen more clearly than (bi). In (di), the occluder patch size is enlarged to fit a larger defect, and the user can adjust the patch size interactively. The delivery cable can be detached to simulate the final repair result of closing the upper chamber septal defect, as shown in (ei).
Due to the single color used in Fig. 8, the chamber structures cannot be outlined clearly, so it is difficult to correctly place the virtual patch during planning the treatment. Therefore, as demonstrated in the right side of Fig. 9, the segmented chambers are registered and merged with the 4D rendered images, and the real-time 3D ultrasound is also included to provide additional imaging information. The left side of Fig. 9 describes the primum and secundum ASD configurations as well as the procedure using the occluder patch to effect the repair, along with the removal of the delivery cable to show the final treatment result. The right side this figure shows the result of using the virtual occluder patch to repair the defect, with the chambers of interest being enhanced with unique colors to show the boundaries, and 3D ultrasound is merged to provide real-time information. The SOP is placed between the left and right atria, and the cardiologists can interactively adjust virtual occluder location and the patch size to fit the heart wall hole. When the patch position inside the heart is satisfied, the delivery cable or bar can be removed to simulate the final surgical result, as show in the right image of (ei).
Various image exploration and septal patch operation results are illustrated in Fig. 10, in which the virtual patch tool factor TransSurf is interactively adjusted between 1.25 and 1.5 to enhance patch visibility. The bottom row images (a2) -(e2) are enlargements of the corresponding images of the top row in the same column (a1) -(e1). Fig. 10 (a) is a cardiac image rendered in grey with full opacity, and (b) shows only the chambers, both of which have a virtual SOP placed in an orientation that is different from those demonstrated in Fig. 8 and Fig. 9. Semi-transparency is used in the displayed MR heart of (ci) and (ei), allowing the user to visualize enhanced chambers,complementary ultrasound image data and the SOP clearly with the MR heart as a reference. Moreover, (ei) shows the final ASD treatment planning result by removing the delivery cable from the virtual patch, so the cardiologists can detect the placed virtual occluder patch as well as enhanced heart chambers in each of the 20 volumes of the cardiac data set.
The developed software platform can be used in various environment as long as suitable graphics hardware TABLE 2. The System rendering environment comprises an Intel CPU 9900k and Nvidia graphics hardware: graphics processing unit (GPU), Firefox and Chrome web browsers; and software platform performance analysis. This latter analysis includes measuring web-based static and dynamic rendering speed in frames per second (FPS), time used for data loading in minutes from Node.js server to client computer, as well as time used for system synchronization between server and connected computers in milliseconds. We employ a wireless network that has ∼100 MB download and ∼20 MB upload speed.

FIGURE 11.
Bar chart of the dual-modality cardiac data rendering with chamber enhancement on four different Nvidia graphics cards and three web browsers using the evaluation results listed in Table 2. The visualization methods include cardiac static and dynamic rendering without and with virtual tool integration. support, high-speed Internet, and web browser are available. When the multi-modality medical data have been loaded to the system, the system can deliver real-time synchronized dynamic data visualization with feature of interest enhancement and virtual instrument integration and medical information can be streamed among all the connected users. The introduced system can process various medical images, including computed tomography (CT), MR, and ultrasound. We plan to add functions for processing and analyzing CT scan heterogeneity, which can be quantified using texture analysis to extract spatial information from medical CT images. Table 2 shows the rendering performance of the system operating on the 4D cardiac MR images registered with complimentary 3D ultrasound images and with chambers of interest enhanced. Moreover, the virtual SOP is modeled and integrated into the image rendering pipeline. We have also tested image exploration synchronization and data the loading speed from a Node.js server to client computers connected with a wireless network and WebSocket protocol.

B. PERFORMANCE EVALUATION
All the listed performance numbers in Table 2 are mean rendering speeds of 30 tests in frames per second (fps) and corresponding standard deviation (STD) for static and dynamic rendering with chambers of interest enhanced with and without virtual SOP integration. For static rendering, the graphics hardware is the major factor that determines the rendering speed. As shown in Fig. 11 (a), the system performance for static data rendering without virtual patch integration can achieve ∼141 fps when using an RTX 2080 card in the SeaMonkey web browser and ∼117 fps when using Edge and Chrome browsers with an average STD ∼8.2 fps. The lowest rendering speed of the system is greater than ∼55 fps when using the GTX 980 card on a Chrome browser, and all the STDs are less than 10 fps, which means that it can achieve real-time performance with relatively low speed variations without updating the hardware or web browser.
We also demonstrate that web browser can have an impact on the performance, which is demonstrated in Fig. 11 (b). For instance, when compared with Chrome or Edge browsers, SeaMonkey can increase rendering speed by up to ∼24 fps. When the virtual instrument is integrated into the data visualization pipeline, we note that the average performance increases from ∼50 fps for GTX 980 to ∼80 fps for RTX 2080, and the Edge browser gives the best performance when compared with Chrome or SeaMonkey. For virtual instrument integration, when using each of the four testing graphics hardware systems, we can obtain rendering performance that is faster than ∼57 fps on all the three web browsers.
When visualizing the entire 4D dual-modality dataset with the chambers of interest enhanced, we note a performance degradation, which can change from an average ∼125 fps to ∼7 fps. However, the graphics hardware employed has little impact. As illustrated in Fig. 11 (c), the average rendering speeds using the four GPUs are within the range from ∼6.7 fps to ∼6.9 fps with the average STD ∼0.16 fps, which means that the system can maintain a uniform rendering speed. When integrating the virtual instrument into the rendered data, we obtain an average speed ∼6.5 fps with STD ∼0.16 fps for the four graphics systems, a performance degradation of only ∼4.4%.
As demonstrated in Table 2, when using a wireless network connection with ∼100 MB download and ∼20 MB upload speed, for data loading from Node.js server into client computers, the average time needed to load all the 20 by 4 data sets is ∼110 seconds when using Edge or SeaMonkey, with a slower loading speed when using SeaMonkey with time variation less than 3%, which needs up to ∼200 seconds. In the developed system, the ∼2 minutes high loading time is only for the initial medical data loading, when the data have been loaded, they remain in the CPU main memory of the client computers and can be used for the following operations and rendering. The system only sends signals to synchronize the visualization of the heart beating. For the beating heart rendering synchronization between client computers connected with WebSocket protocol, the time delay is usually less than ∼5 milliseconds for all four systems and three web browsers used in the experiment, so users can get real-time visual feedback without perceptible delay. We believe the reason for the relatively high standard deviation of the loading times is the Internet speed fluctuation.
We conclude that the main performance restriction of our system for 4D data rendering, is texture loading from CPU main memory to GPU graphics memory and binding them with the 3D and 2D textures in GPU fragment shader, with the impact of the graphics card type being minimal. In Table 2, we compared the rendering speed using different generation GPUs, from which we can see that when using the newest generation GPU such as Nvidia 2080 in the experiment, the presented system can deliver a rendering speed that is faster than 110 fps, and the speed is ∼7 fps for dynamic rendering (tissue feature enhanced beating heart) with virtual tool integration. We believe the system rendering speed can be greatly improved when using the newest GPU such as Nvidia 4090. Moreover, the synchronization delay is less than ∼5 milliseconds, so all the connected users cannot detect any delay in beating heart synchronization and exploration over the Internet. Therefore, we can conclude that our system can provide sufficient performance to be employed in a collaborative multi-user environment for procedures such as cardiac intervention planning. Fig. 6 illustrated that users can stream diagnostic messages over Internet, share the same dynamic cardiac images and interactively manipulate the virtual instrument using web browsers.

V. CONCLUSION AND FUTURE WORK
In this paper, we presented a web-based architectural model that enables interactive multi-user visualization and manipulation of dynamic medical image data in real time, and demonstrated its capabilities using the planning of an atrial septal defect repair procedure as an example. New algorithms, computational methods, implicit surface models, and a web-based collaborative software architecture were developed to display a series of MR and registered 3D ultrasound images of a beating heart, in which incorporated colorenhanced regions of interest and an interactive instrument -in this case a septal patch occlusion device. The software system runs on a Node.js framework and enables synchronous visualization across multiple Internet-connected observers without appreciable delay.
Taking advantage of mainstream graphics hardware and modern Internet protocols, our system can display static cardiac data in ∼140 frames per second (fps) with less than 8 fps standard deviation (STD), and render dynamic chamber enhanced cardiac images with virtual tools in a speed that is faster than the heart rate. During the planning procedure, cardiologists can stream diagnostic messages amongst all Internet connected collaborators, share the same high-quality cardiac views and instrument manipulations with less than ∼5 milliseconds delay. VOLUME 11, 2023 This web-based architecture can stream synchronized images of the beating heart, which provides cardiologists and radiologists with 4D information of heart inner anatomies and its functions in real time, enabling them to collaboratively explore cardiac chamber structures and study its dynamic behavior without geographical limitations, and thus can be used as a valuable platform for collaboratively diagnosing atrial septal defect and planing an optimized treatment strategy over Internet.
The presented algorithms and software platform are for new technology development and medical verification, which have been not integrated with existing Electronic Health Record (EHR) systems. In the future work, when this research project is implemented in a real-world clinical scenario, we will consider the Protected Health Information (PHI) through setting advanced security measures, such as dynamic password and dual layer verification for protecting the access of the sensitive data and information regarding patients' identities, testing results, medication or prescription histories, phone records, and billing information. Role-based security levels will be used to ensure only those with clearance can see PHI. Furthermore, we will use encrypted methods when sending PHI electronically over Internet.
Deep learning based registration algorithms will be designed and used for MR and ultrasound image registration in our future work to improve the efficiency of processing a large amount of medical data. We also plan to intelligently integrate multi-source data generated from the Internet of Medical Things (IoMT) environment for first aid assistance to patients with chronic cardiovascular deceases.