4D Ultrasound-Based Knee Joint Atlas for Robotic Knee Arthroscopy: A Feasibility Study

In this work, we proved for the first time the feasibility of using high-refresh-rate 3D ultrasound (US) also known as 4D US imaging to create a volumetric atlas of the knee anterior compartment for an autonomous robotic platform for knee arthroscopy. A dataset of 42 4D US sequences (including 94 US volumes) and 25 MRI volumes was collected from seven volunteers, in several leg positions simulating the surgical scenario of knee arthroscopy. MRI-US volume pairs were manually registered, and the knee structures of interest identified on the US volumes. The resulting atlas comprised the femur, tibia and patella surfaces, patellar tendon, femoral cartilage, the anterior parts of the menisci and the ACL, for knee angles between 0 and 90 degrees flexion. The inter-operator reproducibility of the registrations was calculated as the norm of the difference in the translation and the rotation values selected by two experienced orthopaedic surgeons and resulted to be on average of 4.42 mm ± 1.89 mm SD and 7.77 degrees ± 2.80 degrees SD, respectively. A new metric was introduced to measure the overlap of the US volume located at the position selected from the first and the second experts and the agreement resulted to be on average of 87% ± 3 SD. The US scanning protocol adopted could be considered compatible with the arthroscopy procedure, as proved through six cadaver studies. These preliminary results show that 4D US is an excellent candidate for automatic image-based guidance in knee arthroscopy.

To reduce these shortcomings, our group has recently proposed and is currently investigating a robotic platform for knee arthroscopy [4]- [8]. This system implements volumetric ultrasound imaging (US) to automatically localize the structures of interest and to provide guidance for the arthroscope and the surgical tools during the intervention [4]. US is cost-effective, non-invasive, portable and most importantly it is presently the only modality compatible with the surgical theatre that can provide ''real-time'' (high-refresh-rate) volumetric images, usually referred to as 4D US [9], [10]. The term ''4D US sequence'' will be used throughout this paper for a collection of US volumes acquired sequentially at high-refresh-rate.
In the current robotic MIS scenario, many applications make use of 2D US for surgical guidance, whereas the 4D US modality remains largely unexplored, despite the numerous advantages over the 2D US mode [10]. 4D US can provide a comprehensive view of the surgical site, enabling the monitoring of the tissue response due to the contact with surgical tool/s or physiological factors. This is particularly important to safely enhance the level of automation in MIS, for which the limited field of view provided by the endoscope represents one of the major limitations.
The first step to assess the feasibility of using 4D US in robotic knee arthroscopy is to evaluate if it allows identifying intra-operatively the targets and the structures at risk in the surgical site. To the best of our knowledge, no literature is available on either 3D or 4D US imaging of the intra-articular anatomy of the knee. The available literature reports on the clinical use of 2D US imaging for diagnostic applications or percutaneous needle injections [11]- [16]. The US examination is focused on each of the knee structures independently. Depending on the structure of interest, it requires the US probe to be positioned with a specific orientation on the area of interest of the knee joint, flexed at a fixed angle (Fig. 1). The menisci are commonly visualized from the medial/lateral knee sides. The femoral cartilage and the anterior cruciate ligament (ACL) are imaged from the anterior knee compartment; while the posterior cruciate ligament (PCL) from the posterior knee compartment. This choice depends on the structure position relative to the bones since the bony layer proximal to the US probe delimits the region that can be imaged [17]. In fact, the patellar cartilage is the only knee structure typically inspected during knee arthroscopy that cannot be visualized with US because it is located on the underside of the patella [12]. This effect plays also an important role in the selection of the knee flexion angles while acquiring the scans. Since the bones in the knee joint rotate and translate relative to each other, the selected knee flexion angle should guarantee that the structure of interest is not shielded by the bones. For example, the femoral cartilage is assessed with the leg in full extension. In this position the patella lies superior to the femur, uncovering the cartilage on the femoral condyles and the femoral groove.
In the surgical scenario of knee arthroscopy, additional requirements and constraints should be considered. 4D US imaging should enable the simultaneous identification of the tissues within the surgical site under static and dynamic conditions (e.g. knee flexion), without interfering with the presence of the surgical tools. The purpose of this article was to develop an US scanning protocol compatible with the surgical procedure and anatomically map 4D ultrasound sequences of increasing complexity to a dataset of MRIs taken at varying degrees of knee flexion. This process resulted in a 4D US knee joint atlas for robotic knee arthroscopy.

Cadaver Study:
Cadaver studies were performed to develop the protocol for the acquisition of US scans in this study and to investigate its compatibility with the arthroscopy surgical scenario. US data was gathered from 6 cadaver knee models (4 left and 2 right knees). The Queensland University of Technology Ethics Committee (No. 1400000856) granted the ethics approval for the data collection.

A. US REFERENCE PROBE POSITION AND SCANNING CONVENTION
The US scans were acquired using a VL13-5 4D US probe and a Philips EPIQ7 US system (Philips Medical Systems, Andover, MA, United States). During the US scans, the probe was positioned on the patellar tendon, with the scanning plane parallel to the tibial shaft ( Fig. 2 a), to possibly allow for the simultaneous visualization of the relevant knee structures, without hampering the motion of the surgical tools. This US probe position will be referred to as the ''reference scanning position'' throughout the manuscript.
An example of US volumetric image of the anterior knee aspect is shown in Fig. 2 b. During image acquisition, for visualization purposes, the three conventional anatomical planes (sagittal, transverse and coronal planes) within the volume are shown on the workstation screen. Since bones are rigid and easily identifiable on US images, the scanning convention adopted was to visualize the region from the inferior tip of the patella to the superior surface of the tibia in the sagittal plane and both the femoral condyles in the transverse plane (Fig. 2 b).

B. US SEQUENCES ACQUISITION DURING ARTHROSCOPY
Before the acquisition of the US scans, the knee joint was injected with saline solution as performed during the standard arthroscopy procedure [1]. An arthroscope and a hook-like surgical probe were inserted respectively in the lateral and medial parapatellar portals. To scan the knee in the range of flexion angles in which the leg is typically positioned during surgery 0, 30, 60, 90 degrees were selected as reference flexion angles, considering as 0 degrees flexion the neutral leg position with the femur and the tibia aligned (Fig. 3 a). For each reference flexion angle, the leg was positioned on a knee cushion customized for the flexion angle considered (Fig 3. b) and 4D US scans were acquired with the US probe in the reference scanning position, while the surgeon guided the surgical probe on different areas of interest (medial\lateral meniscus, femoral cartilage and ACL) through the arthroscope view. These operations were performed in different sessions by three experienced surgeons (F.S., Y.T. and R.C), while a clinician was in charge of holding the US probe in position (Fig. 4 a). Acoustic coupling between the ultrasound probe and the knee surface was ensured using a standard water-based gel.
Two additional US probe positions at the medial and lateral knee sides were also tested to possibly enable  real-time guidance whenever the tool moved towards the most medial-lateral aspects of the knee (Fig. 4 b and c).

C. US SYSTEM PARAMETERS
The US system settings were initially selected by an experienced sonographer: 13 MHz probe frequency; 4 cm penetration depth; far-field focus; dynamic range of 60 dB; emission power of -0.5 dB and medium persistence; XRES image processing and SonoCT compound imaging. Since the ultrasound signal cannot traverse bony surfaces, the surface of contact between the femur and the tibia defines the maximum depth from the skin for which meaningful anatomical information can be acquired. For larger flexion angles (and/or for larger knee cavities), this surface of contact is deeper from the skin and thus the penetration depth needed to be adjusted accordingly. Penetration depths between 3.5 cm to 6 cm were selected, corresponding to voxel dimensions of 0.14mm × 0.16mm × 0.16mm (272 × 510× 256 voxels) and 0.13mm × 0.12mm × 0.24mm (288 × 510× 256 voxels), respectively. To further optimize knee structures visualization, dynamic range and focus were also adapted to the specific knee and knee flexion angle. Dynamic range values between 48 dB and 60 dB were selected. The refresh-rate for a full US volume acquisition was 1 Hz.
Volunteer Study: To create an US-based atlas of the knee, 4D US sequences and MRI scans were acquired. In brief, the data was collected from both left and right knees of 7 volunteers (5 men and 2 women) with a mean age of 32 years (range: 24 -45 years) and mean BMI of 24.3 kg/m2 (range: 20.1 -29.2 kg/m2). The full study protocol is outlined in the following sections A and B. The MRIs acquired were also used for the medical examination of the volunteers included in this study (Table 1). The Queensland University of Technology Ethics Committee (No. 1700001110) granted the ethics approval for the US and MRI data collection. Signed informed consent was obtained from all the volunteers participating in the study.

D. PROTOCOL FOR 4D US SEQUENCES ACQUISITION
42 4D US sequences were acquired simulating several possible surgical scenarios, with the leg fixed at the reference knee flexion angles (0, 30, 60, 90 degrees knee flexion) or where the leg or the US probe were subjected to motion. Leg motion was limited to knee flexion between 0 and 90 degrees. US probe shifts were performed only along the craniocaudal direction since the US probe would be constrained in the medial/lateral direction due to the presence of surgical tools at both sides ( Fig. 2 and Fig. 4). Three different scanning procedures have been performed acquiring: 1. ''4D US static sequences'' with the leg fixed at reference knee flexion angles. The US probe was positioned in the reference scanning position (section II Cadaver study A). For knee flexion angles different from 0, the US scans were recorded with the leg placed on the cushion designed for the corresponding knee flexion angle. Each 4D US sequence was recorded for about 10 seconds, resulting in 10 volumes. 2. ''4D US dynamic sequences with leg motion'' while the leg was extending from 90 to 0 degrees knee flexion. The scan started with the leg at 90 degrees knee flexion, positioned on the cushion customized for this flexion angle and the US probe in the reference scanning position. During the scan, a force was applied on the thigh to keep it in contact with the cushion while moving the lower leg in the anterior direction to reach full extension, at the same time holding the US probe in the reference scanning position. The whole extension movement was performed on average in 5.85 seconds (range: 5 to 8 seconds). 3. ''4D US dynamic sequences with US probe motion'' with the leg fixed at 0, 60 and 90 degrees flexion. The US probe was translated from the reference scanning position along the caudal direction, keeping the probe orientation parallel to the tibial shaft. For flexion angles different from 0 degrees, the leg was positioned on the corresponding leg cushion. The US recording was ended when the femoral cartilage was not visible anymore along the workstation sagittal plane. This movement was recorded on average in 4.25 seconds (range: 3 to 8 seconds). As opposed to the cadaver study, the volunteers' knees were imaged while submerged in water. This solution has been adopted to fill possible discontinuities between the lateral edges of the probe and the knee surface, causing the corresponding volume parts to not be imaged and reducing the overall image quality. The US probe/system and US system parameters specified in section II Cadaver study C were used for this part of the study.

E. PROTOCOL FOR MRI IMAGE ACQUISITION
The volunteers' knees were also examined on a 3T MRI system (Siemens Magnetom 3T Prisma, Erlangen, Germany) with 3D SPACE sequences in PD-weighting using dedicated knee coils. The MRI images were acquired with the knee flexed at the reference angles (0, 30, 60, 90 degrees knee flexion). The voxel spacing was isotropic for each volume and varied between 0.5 and 0.9 mm.
During the MRI acquisition at 0 degrees flexion, the volunteer lay on the patient table in the supine position. For the other flexion angles, the leg was set at the reference angles using the designed cushions (as for the US acquisition) that were fixed to the leg using straps. Due to the limited space left for the MRI coil, the volunteer was positioned in the left/right lateral recumbent position for the corresponding leg to be scanned.

F. KNEE ATLAS FOR 4D US STATIC SEQUENCES
This part of the study aimed at the identification and characterization, whenever possible, of the knee structures in the joint directly involved in the surgical procedure (femoral cartilage, menisci, PCL and ACL) and relevant anatomical reference structures of the joint (e.g. patella, femur, tibia and patella tendon) on the US volumes of the 4D US static sequences (section II Volunteer study A).
17 US volumes were selected from 17 4D US static sequences (one volume per sequence) collected from 5 volunteers and were manually registered by Y.T. with the MRIs acquired for the corresponding knee flexion angles using Imfusion (ImFusion, München, Germany). The superimposition of US and MRI volumes was visually inspected, and the knee structures localized on the US volumes sagittal plane, being the highest resolution plane. Table 2 reports the MRI-US pairs analysed in this part of the study. The MRI-US registration was also used to define the actual region of the knee captured by the US volumes. The region of the knee included in the US volumes along the anterior-posterior direction was determined to calculate the maximum depth from the skin to which the tissues could be visualized. In the craniocaudal direction, the overlap between the US and the MRI volumes was visually inspected to ensure that the US volumes enclosed the knee space between the femur, the tibia and the patella. To determine the knee region imaged along the medial-lateral direction, the femoral epicondyles were used as a reference to calculate the maximum extension of the knee joint, i.e. the total width of the joint. The distance between them was calculated for each volunteer on the MRI axial plane using a calliper available in Imfusion. The maximum distance between the femoral condyles covered by the US volume was also computed and the percentage of the total width covered calculated.

1) MRI-US REGISTRATION PROCEDURE
The registration performed was a rigid rotation-translation since the US and the MRI volumes were scanned for the same knee flexion angles. Imfusion allowed us to import and to visualize the two modalities in a common coordinate system. The user was then enabled to select for the three orthogonal planes the translation values (in mm) and the rotation values (in degrees) to be applied to each volume. The view panel allowed to visualize the two volumes simultaneously along three orthogonal planes and in 3D rendering mode while the rotation and translation values were modified by the user. The procedure followed for each MRI-US volume pair can be summarized in three steps: 1. The MRI was rotated within the common coordinate system such that the three planes in the view panel would match the three anatomical planes. The coronal plane was selected to intersect the medial and the lateral femur epicondyle (highlighted with yellow arrows in Fig. 5 a). The axial plane was aligned with the lateral and medial parts of the trochlear groove. The sagittal plane was consequently imposed to be orthogonal to the other two planes (Fig. 5 a). 2. For the US volume, the three planes in the panel view corresponded approximately to the three anatomical planes and coincided with those of the MRI in the view panel (Fig. 5 b). An initial alignment between the two modalities was obtained by translating the US volume in the region inferior to the patella, visualizing the position of the US volume with respect to the leg rendering produced for the MRI. By modifying the US transformation matrix, the US volume was then rotated around the sagittal direction such that the inclination of the patella tendon would match approximately along the sagittal plane in both modalities (Fig. 5 c). 3. The transformation matrix of the US was then adjusted to find the best fit between the two modalities along the three anatomical planes by matching the femur and the tibia bony surfaces in the MRI-US pair (Fig. 5 d).
This resulted in the whole US volume being matched with the corresponding MRI. The overlap of the patella surface was considered less relevant for the MRI-US volumes match, as the probe pressure exerted on the patellar tendon during the US acquisition could have caused small shifts of the patella.
In case a good correspondence between the US and MRI volumes was not found, 3 different registrations were performed, considering the overlap of the two modalities based on the tibia and the femur individually. The same procedure described above was applied to initially align the two modalities to perform these registrations.

2) REPRODUCIBILITY OF THE MRI-US REGISTRATION
F.S. manually registered 8 out of the 17 MRI-US volumes pairs, obtained from 3 volunteers. F.S. performed the registrations starting from the initial MRI alignment with the anatomical planes provided by Y.T. and then following the procedure described in the previous section independently and blind to the registration performed by Y.T.. Since the MRI volume was kept fixed after the initial alignment, the registration accuracy was assessed for each MRI-US pair by comparing the US volume positions obtained by the two surgeons.
Two metrics have been utilized: • The field of view overlap (O)(1), which is a new metric defined specifically for this purpose measuring the overlap between the US volume (V ) located at the position selected from the first surgeon (V 1 ) and the same US volume located at the position selected by the second surgeon (V 2 ) . All the voxels contained in the two volumes were considered as 1s and all the voxels outside the volumes as 0s.
With · being the dot product; j = 1,. . . (r 1 × c 1 × d 1 + r 2 × c 2 × d 2 -r 12 × c 12 × d 12 ), where r, c and d are the rows, the columns and the depth of the volume indicated by the subscript (with the subscript 12 indicating the elements belonging to both volumes) and |V | is the total number of voxels contained in the US volume.
Since the 3D US volumes shape is a cylindrical sector with flat apex, when sliced along a 2D plane, 2D images of different sizes would result. For this reason, when extracted from the US system padding with black pixels is automatically applied to obtain a rectangular parallelepiped volume. When computing the field of view overlap of the volumes where black pixel padding was applied, areas of the volume containing no US information where included. For this reason, this metric was also computed considering only the central slices of the volume where no padding was present.
• The norm of the difference vector of the translations (d) and rotations (r) values selected by each surgeon [18] is calculated by (2)(3)(4).
Where x, y, z and α, β, γ are the three translation and rotation values, respectively, selected by the surgeons 1 or 2 as indicated by the subscript.

G. KNEE ATLAS FOR 4D US DYNAMIC SEQUENCES
This part of the study focused on the localization of knee structures in the central knee compartment (ACL, the femoral cartilage, the anterolateral horn of the meniscus, the tibia and the Parson's knob, a bony knob on the tibia, used to identify the tibia insertion of the ACL [19]) on the 4D US dynamic sequences (section II Volunteer study A). These structures were selected as they are particularly relevant for two common arthroscopy procedures: ACL reconstruction and femoral cartilage restoration [1], [20]. 77 US volumes were extracted from 25 4D US dynamic sequences collected from 7 volunteers. These volumes were manually registered by Y.T. with the MRIs at corresponding knee flexion angles as described in section II Volunteer study C. The registered MRI-US pairs were visually inspected by the expert and the knee structures identified on the US volume's sagittal plane as in section II Volunteer study C. US volumes where the tibia surface could not be identified (8 US volumes out of 77) were discarded from the identification of the structures attached to this bone. In most of the cases, this occurred for the first volume (in chronological order) of the 4D US dynamic sequences with US probe motion, where the scan was too superior to include the tibia in the field of view.
For the 4D dynamic scan with leg motion, the knee angles of the US volumes at the start and the end of the sequence were known (90 and 0 degrees flexion angles, respectively), whereas it was not possible to infer the precise flexion angle for the US volumes in between. For this reason, these US volumes were registered with MRIs at a similar knee flexion angle (either at 30 or 60 degrees of flexion).
The rate of detection of the structures of interest was then computed for the different types of scans acquired. Three possible cases were identified: positive (R p ) negative (R n ) and uncertain detection (R u ) (Eq. 5). In the positive and the negative cases, the structure of interest was well recognizable or not recognizable, respectively. In the uncertain case, the structure was possibly present on the US volume slices, but not well defined. Thus, it could not be confidently detected.
where V p , V n and V u are volumes where the detection rates were positive, negative and uncertain, respectively and V tot the total number of volumes in the study.

A. US PROBE COMPATIBILITY WITH KNEE ARTHROSCOPY
The cadaver experiment proved that the US probe could be positioned on the patellar tendon without affecting significantly the motion of the surgical tools inside the patellar portals, for each knee flexion angle between 0 and 90 degrees. Good US probe contact (probe almost fully in contact with the knee surface) was achieved by expanding the knee capsule with saline solution. This reduced significantly the presence of air gaps at the extreme sides of the patella and allowed us to acquire US volumes almost ''free'' from acoustic coupling discontinuities.
The additional probe positions on the medial and lateral knee sides did not interfere with the surgical tool manoeuvrability either.

B. KNEE ATLAS FOR 4D US STATIC SEQUENCES
The femoral cartilage layer, the bony surfaces of the femoral condyles, patella and tibia, the patellar tendon, the anterior cruciate ligament, the anterior horns and the anterior parts of the menisci were identified on the US volumes analysed. Fig. 6 shows two corresponding sagittal slices of MRI and US of a volunteer knee flexed at 60 degrees ( fig. parts a and  b) and the 3D segmentation obtained by manually contouring the structures in the acquired US volume ( fig. part c).
On US images, healthy femoral cartilage appeared as a black hypoechoic band confined by the hyperechoic boundaries (Fig. 7). Different aspects of the femoral cartilage were visualized depending on the knee flexion angle. As the leg flexes, the femur slides in the cranial and the posterior directions. At 30 degrees flexion, part of the central aspect of the femoral cartilage (the femoral groove) was located beyond the patella and thus it could not be imaged. This effect was enhanced due to the relative motion between the femur and patella when the leg was flexed to 60 and 90 degrees. The femoral cartilage boundaries were more defined when the US beam was perpendicular to the femoral cartilage surface (Fig. 7 b). As the cartilage curvature increased the femoral cartilage boundaries gradually lost definition on the US image (white dashed arrows in Fig. 7 b). This effect was enhanced by the more distal location of the areas of  increased curvature where the physical resolution of the image is reduced.
The boundaries of the ACL and the lateral meniscus were typically less defined compared to the femoral cartilage and covered a smaller area on the US slices within a volume, and thus in some cases were more challenging to identify. The ACL appeared as a broad hypoechoic band, with the proximal interface visible as a thin hyperechoic layer (Fig. 8). The ACL inclination angle varies depending on the knee angle: it lies almost parallel to tibial plateau when the knee extends, and it comes to stand obliquely to the tibia plateau as the knee bends. For smaller flexion angles the ACL attachment to the tibia was typically visible as a hyperechoic line (Fig. 8 a). The rest of the ligament could only be imaged with the knee bent deeply (60, 90 degrees angle) (Fig. 8 b). Fig. 8 a highlights also the patellar tendon and the soft tissues below the patellar tendon (the Hoffa's fat pad and the synovial tissue) on MRI and US volumes. The interfaces of these tissues typically coincided in the two modalities. In some cases, the patellar tendon was slightly shifted posteriorly (about 1 mm) on the US volume, due to the compression applied by the  US probe during the image acquisition. The US volumes provided finer details compared to MRI about the texture and the composition of these tissues.
Lateral and medial menisci were homogeneous hypoechoic or grey spots locating beside the femoral cartilage, characterized in some cases by a hyperechoic signal at the proximal interface. The insertion of the anterolateral horn of the meniscus could be identified on all the volumes analysed since it is adjacent to the ACL tibia attachment (Fig. 9 a). The more central aspect of the lateral meniscus could also be localized (Fig. 9 b). The medial meniscus could not be identified in all the volumes analysed, as a small and thin portion of it was included in the US volumes.
12 out of 17 US volumes (70% of the volumes) could be fully matched with the corresponding MRI volumes. For the remaining cases, the registration had to be performed considering the individual bones in the two modalities. Apart from offering comprehensive anatomy information relative to the knee joint, the MRI volumes were also used to quantify the knee joint part covered by the US volumes. The US volumes covered up to 6 cm in depth from the skin and enclosed the intra-articular space in the craniocaudal direction. Along the medial-lateral direction, the US scans could partially cover both femoral condyles (the most medial-lateral aspects of the femur were not imaged) for each volunteer and knee flexion angle. Considering the distance between the femoral epicondyles identified in the MRI as the total width of the knee space, the US volumes covered on average 65.80 % ± 4.66 SD of the joint, where the mean distance between the epicondyles was on average 8.06 cm ± 0.48 SD and the US volume could cover a mean distance of 5.28 cm ± 0.21 SD. In addition to the most lateral and medial aspects of the femur, the PCL, the attachment of the ACL to the femur, the most medial and lateral sides of menisci were not in the US volume field of view.

1) REPRODUCIBILITY OF THE MRI-US REGISTRATION
The mean field of view overlap calculated for the 8 MRI-US pair was of 87% ± 3 SD (range:82 -91 %) considering the US volume with padding and of 86% ± 3 SD (range: 79 -90 %) when only the full-size slices at the centre of the US volume were selected.
The average norm of the translation and the rotation difference vector was of 4.42 mm ± 1.89 SD (range: 1.67 -7.90 mm) and 7.77 degrees ± 2.80 SD (range: 4.05 -11.99 degrees), respectively. Fig. 10 shows the norm of the rotation difference obtained for each MRI-US pair as a function of the norm of the translation difference and the corresponding field of view overlap obtained.

C. KNEE ATLAS FOR 4D US DYNAMIC SEQUENCES
The femoral cartilage was detected on all the US volumes of the 4D US dynamic sequences. For the ACL and the anterolateral meniscus, the detection rate for all cases analysed is reported in Table 3. The highest detection rate for the ACL was obtained as expected for the volumes acquired for the larger flexion angles in the range. A similar trend was found for the lateral meniscus. Most of the negative and uncertain detection rates were found for the scans where the probe was translated on the knee. This can be explained by the additional challenge faced to perform the MRI-US registration, as due to the probe translation most of these volumes contained only a small part of the femur and thus only the tibia was available as a bony landmark to superimpose the two modalities. A prominent Parsons' knob could be identified on the MRI volumes of 5 volunteers out of 7. It could be identified in 22% of the US volumes, while it was uncertain in 30% and not detected in 48% of the volumes. A clear distinction in the detection rate based on the scanning angle could not be found for the Parsons' knob. For each volunteer, all the volumes in the 4D US sequence acquired while the leg was extended had either a very clear, unclear or not detected knob, regardless of the knee angle. Fig. 11 shows for one volunteer the US slices of a volume where the ACL, the lateral meniscus and the Parson's knob were identified. No substantial difference was recorded in terms of image quality between the dynamic and the static 4D US sequences acquired, indicating that if the movements performed are not too rapid dynamic scanning does not represent a limitation.

IV. DISCUSSION
In this study, we report on the first attempt of using 4D US for an autonomous or robotic-assisted application. The preliminary results show that 4D US holds the potential to provide guidance for a knee arthroscopy robotic system, currently being investigated by our research group. 3D/4D US imaging was used to create a volumetric knee atlas of the anterior knee compartment (femur, tibia and patella surfaces, patellar tendon, femoral cartilage, the anterior parts of the menisci and the ACL), for different knee angles between 0 degrees and 90 degrees knee flexion (section III B). Several possible realistic knee arthroscopy scenarios were simulated under both static and dynamic conditions and proved to be compatible with all the arthroscopy procedures tested during cadaver experiments (section III A). During knee arthroscopy, the femoral cartilage is examined while the leg is flexed through the whole range of motion, while a 90 degrees flexion angle is utilized for the ACL [1], [21]. We proved that under these conditions the femoral cartilage and the ACL could always be identified using 4D US (section III C). This result is particularly relevant for the femoral cartilage as, throughout arthroscopy procedures, this structure is also the most commonly unintentionally damaged [2].
It should be noted that the movements of the probe or the leg performed while the scans were acquired were relatively slow and thus the image quality of these volumes was not significantly affected by motion. Limited volumetric refresh-rate is one of the main present limitations of the 4D US technology and the main disadvantage of 4D US over the 2D US mode. Typically, the refresh-rate of a mechanically swept 4D US probe (e.g. the one used in this work) is limited to about 1 volume per second. This is one of the reasons why the real-time arthroscope view would be still essential during the surgical procedure. It is important to remark, though, that phased array US probes can collect up to 6 volumes per second. Should such probes be available with the required frequency, in a setting where the leg and the tools could be manipulated by robotic arms, the speed at which the movements are performed could be accurately controlled to match the available refresh-rate.
While in this study we focused on the central knee compartment, in a typical arthroscopy procedure the whole knee joint is inspected. The medial and lateral knee compartments are probed with the leg flexed at a small angle (10-30 degrees) and valgus force placed on the joint and in the figure-four position, respectively [1]. Our results show that the extreme sides of the knee were not in the field of view of the US volumes acquired. However, through the cadaver studies performed, we proved that additional US probe positions on the medial and lateral knee sides are feasible (section III A) and should be further explored considering the corresponding leg positions adopted during surgery.
The US sequences examined in this work were collected from the volunteers' knees while submerged in water, which ensured optimal acoustic coupling between the knee and the US probe surface. While this might look like a limitation of this work, we showed in the cadaver studies that during surgery the acoustic coupling issues are significantly mitigated. As part of this research project, we are investigating the development and use of specific coupling devices to optimize image acquisition accuracy and reliability for the final clinical application. Moreover, at this preliminary stage, the surgical procedure simulations on cadaver models and data collection were performed by surgeons and clinicians. For the final application, the procedures will be performed by a robotic platform (autonomously or in assistance to surgeons) and the US probe will be held and manipulated by a robotic arm, so refinements in the scanning procedures might be necessary.
The main limitation of this study was that it was not possible to acquire the US and the MRI volumes in the exact same conditions. This was caused by the relatively long acquisition time for the MRI volumes that made it difficult to keep the leg in position and possibly also by the slightly different leg positions used for the US and MRI acquisitions (i.e. leg flexed while the volunteer was sitting vs leg flexed in the lateral recumbent position, respectively). The inter-operator differences found in the image registrations might be partially due to the suboptimal match between the US and MRI volumes acquired (section III B 1)). This effect may have introduced a degree of subjectivity in finding the best compromise to match the bony surfaces in the two modalities.
Future steps to enable 3D/4D US guidance for robotic knee surgery also involve the automatic identification and tracking of the structures of interest [5], [6], [8]. The anatomical information extracted from the US volumes would then be complemented with the information from the MRI scan, which is typically acquired before the surgical procedure. The MRI information can be potentially useful to localize the sections of the femoral cartilage which are not well defined on the US volumes and the ACL and the lateral meniscus in the most challenging cases. Through the detection of common features/structures in the two modalities, MRI could be even used to identify structures which are not visible on the US scans, such as the patellar cartilage, the PCL and the ACL attachment to the femur. Automatic registration techniques detecting common features in the two modalities should be explored to avoid inter/intra-observer differences in the registration results and the image interpretation. Manual registration of the two modalities is, in fact, particularly challenging due to partial field of view provided by US imaging. Moreover, it is also a time-expensive process, requiring the experts to be trained to interpret these types of volumes.
To date, 4D US has not been clinically used for either diagnostics or surgical applications. The volumetric and the dynamic capabilities of this mode could be exploited to obtain a more comprehensive view of the body region and to monitor the region of interest in time. When combined with automatic image acquisition and automated image processing, 4D US could fully reach its potential and possibly become a widespread solution for autonomous applications both in surgery and diagnostics. ANJALI T. JAIPRAKASH received the B.S. degree in biotechnology from Bangalore University, India, in 2005, and the M.S. degree in biotechnology and business and the Ph.D. degree from the Queensland University of Technology, Australia, in 2007 and 2014, respectively. She is currently working as an Advance Queensland Research Fellow at the Queensland University of Technology. Her work merges different disciplines, such as medicine, engineering, and design, to develop medical devices that translate robotic vision into affordable systems that can be used to improve healthcare outcomes. She has experience in the fields of medical robotics, medical devices, and orthopaedics. a Postdoctoral Research Fellow with the Institute of Health and Biomedical Innovation, QUT. Since 2019, she has also been the Deputy Director of the ARC Training Centre for Multiscale 3D Imaging, Modelling, and Manufacturing, QUT. Her research interests include the interdisciplinary area of biomedical engineering and medical physics applying multiscale 3D imaging, and modeling to medical and non-medical problems. She is an Active Member of the IEEE QLD Section Committee. Since 2020, she has also been a member of the R10 Professional Activities Committee. She was the Chair (from 2016 to 2018) and has been the Vice-Chair (since 2019) of the IEEE Women in Engineering Affinity Group QLD Section Committee.
AJAY K. PANDEY is currently a Senior Lecturer of robotics and autonomous systems with the School of Electrical Engineering and Robotics, and a Domain Leader, within manufacturing with advanced materials, with the Institute of Future Environments. Prior to his current appointment, he held prestigious fellowships, including the QUT-Vice Chancellor's Senior Research Fellowship and the Australian Renewable Energy Agency (ARENA) Research Fellowship at The University of Queensland (UQ). His research interests include the interdisciplinary mix of photonics, chemical physics, molecular electronics, computer science, and robotics. He has published widely around issues surrounding energy, environment, and health.
ROSS CRAWFORD is currently holds the position of the chair of orthopaedic research and the director of the Medical Engineering Research Facility, Queensland University of Technology, Australia. With more than 200 publications, collaborations with Industry and Hospitals, he is an internationally recognized expert in orthopaedics and robotic surgery.
DAVIDE FONTANAROSA worked in one of the top institutions for radiation therapy (MAAS-TRO Clinic, The Netherlands) and in one of the largest industrial research laboratories in the world, Philips Research, as a Senior Scientist. He moved to the Queensland University of Technology, Brisbane, QLD, Australia, to take up a position as a senior lecturer, where he is currently doing research in several fields related to ultrasound, imaging techniques, and radiation therapy. He is a Physicist with a solid background in ultrasound imaging and medical physics. VOLUME 8, 2020