A Postoperative Free Flap Monitoring System: Circulatory Compromise Detection Based on Visible-Light Image

Microvascular free flap surgery has been a reliable reconstruction method, but some flaps may have circulatory compromise within five to seven days of surgery. Besides, many auxiliary devices are expensive and complex to operate. Therefore, most of the medical personnel still use the manual observation. To decrease heavy workload on medical personnel, the monitoring system was developed to analyze the color variation of free flap by a visible-light camera with advantages such as fast operation, low cost, noninvasive behavior, and contactless and nonradiative operation would be also needed for patients. The proposed system was mainly comprised of a longitudinal image registration without markers and a circulatory compromise detection, which were aiming to reduce the deformation from patients’ motion and eliminate the influence of illumination variation from the surrounding, respectively. In the clinical trial, the performance assessment of image registration reveals that the average and the standard variation of dice similarity coefficient (DSC) can reach 0.959 and 0.011, respectively. The relationship among the situation of blood perfusion and the results of circulatory compromise detection at flap was summarized and shown that the detection system not only determined the moment of vascular occlusion but also the type of occlusion. In addition, vascular congestion was even detected much earlier than manual observation. Therefore, our monitoring system is expected to relieve the workload on medical personnel and could be a reliable tool for the surgeon to hold onto the chance of repairing the free flap with vascular obstruction.


I. INTRODUCTION
Free flap surgery of re-anastomosed microsurgery has become a proven technique for the reconstruction of defects in the head and neck region during recent decades. As free flap surgery is not only versatile by transplanting a variety of tissue types, which are skin, muscle, and bone, from other parts of body to sizeable tissue deficits, but restores both functions and aesthetics. Accordingly, free flap surgery has The associate editor coordinating the review of this manuscript and approving it for publication was Rajeswari Sundararajan . some advantages in terms of the wound, such as excellent functional capacitation, a more aesthetic appearance, limited side effects [1], [2]. However, microsurgery is required for this operation because of the creation of microvascular anastomoses between donor and recipient vessels, and the flap is supplied blood by the pedicle until the neovascularization occurrence. Hence, this surgery should be conducted carefully and requires a large amount of time.
The success rate of the free flap surgery is related to factors such as the age and injury severity of the patient and whether the patient has diabetes mellitus, hypertension, and VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ cardiac arrhythmias [3]- [8]. Although the success rate of the free flap surgery is higher than other surgeries, a few cases have exhibited circulatory compromise after the surgery. Previous studies have indicated that 5%-25% of free flap surgeries must be remedied because of circulatory compromise [9]- [11], and the probability of vein compromise is higher than artery compromise [12]. Kroll et al. proposed the same result by analyzing 990 surgery cases. Moreover, if a circulatory compromise is detected in time, the repair rate is 99% [8]. In 2007, Chen et al. analyzed 1142 cases of free flap surgery. Among them, 113 cases were remedied because of circulatory compromise. Moreover, cases with 95.6% circulatory compromise were detected within 72 h after surgery and cases with 82.3% circulatory compromise were detected within 24 h. If the circulatory compromise was detected within 120 h after surgery, the repair rate was 85% [13]. If circulatory compromise cannot be detected in time, necrosis may be caused to the transferred flap, thus leading to tissue loss [14]- [16]. Therefore, postoperative free flap checking is the most important step for decreasing the danger of circulatory compromise and improving the repair rate. Postoperative variation in the free flap is usually examined by manual observation in the clinic. The flap is evaluated according to aspects such as its color and temperature and its capillary refill, turgor, and pin-prick test results. These clinical manifestations of arterial occlusion and venous congestion are summarized in Table 1 [17]. The observation results are influenced by the training of the medical personnel and the accuracy of the experiment conducted. Moreover, the protocol of postoperative free flap observation specifies that an observation should be conducted once per hour for 24 h after the surgery and every 4 h for 72 h from the second day. Such observation requirements lay a heavy workload on medical personnel. Furthermore, a surgeon sometimes misses the chance of repairing the free flap with vascular obstruction because some of the initial clinical manifestations are not obvious and thus cannot be found. Therefore, if some flap monitoring techniques are made available to medical personnel, the observations can be more efficient and accurate. Moreover, vascular occlusion can be detected earlier.
Many techniques have been applied for detecting vessel occlusions to improve the free flap repair rate, such as a method involving a hand-held Doppler probe [18], surface temperature estimation [19], a method involving an implantable Doppler ultrasound probe [20], pulse oximetry [21], laser Doppler flowmetry [22], and near infrared spectroscopy [23], [24]. However, these techniques are expensive, are inaccurate, have a high technology background requirement, and have an invasive injury risks to patients. Therefore, most of the medical personnel still use the manual observation method for vessel occlusion detection rather than using the aforementioned auxiliary devices. A flap monitoring technique with advantages such as fast operation, low cost, noninvasive behavior, and contactless and nonradiative operation would be useful for medical personnel. Among the aforementioned methods, infrared detection appears to be the better choice because previous studies have reported that a variation in the surface temperatures of the flap and surrounding normal skin could indicate meaningful blood circulation problems in the flap [19], [25], [26] and because the method includes almost all the aforementioned advantages.
To demonstrate the potential of clinical monitoring by using infrared cameras, 12 swine pedicle myocutaneous flaps were harvested and monitored vascular thrombosis in our previous study [27]. When the estimated error of flap surface temperature less than 0.86 • C, the sensitivity and specificity of the results were 90% and 81%, respectively. However, it is still inadequate for clinical monitoring by using infrared cameras, because boundaries of the free flap in infrared images are too blur to use them for analyzing the variation in free flap, and attaching markers on patients' skin for a long time is impractical. Moreover, an infrared camera does not have the autofocusing feature. Thus, it may lose focus once the object performs a large motion or rotation. Consequently, infrared detection has limitations because of these problems. In this study, a new monitoring system is presented to improve our infrared system and provide additional information for analyzing the occlusions of free flaps.
Visible-light image has considerable potential for use in postoperative free flap monitoring because of its high resolution, high contrast, and comprehensive information. In 2014, Kiranantawat et al. developed a smartphone application that analyzes digital photos according to the color difference and achieved high sensitivity (94%) and specificity (98%) in detecting vascular compromise [28]. The visible-light imaging method has been demonstrated to be robust for detecting vessel occlusion. However, the critical challenges for extending the application in clinic are summarized as follows: 1) The color of free flap is the important factor to distinguish whether vascular thrombosis happened or not, but visible light is easily influenced by illumination change, which is shown in Figure 1. 2) Markers cannot be attached on patients' skin for a long time. Hence, the boundary of free flap is undefined, and the manual determination is not only labor-intensive and time-consuming but also likely to lead to interobserver differences. 3) Patients may be with a large motion or rotation during monitoring. Hence, longitudinal image registration is used to overcome the problem of deformation between two images. However, free flap image registration, similar to registration of most medical image types, is classified as a nonrigid problem because of the softness and flexibility of the human skin. Nonrigid registration is more difficult than rigid registration because underlying nonrigid transformations are often imperceptible, complex, and difficult to model. Therefore, the new visible-light monitoring system proposed in this study would focus on the accuracy evaluation and overcome those aforementioned problems for extending its application to clinical purposes. Moreover, in order to simplify these problems, stereo vision would not be considered in this system, and patients changing their position are still in the monitoring area.
To perform the longitudinal monitoring system of free flaps for clinical application, it is expected to have the following abilities: 1) Object tracking: To calculate the trajectory of patients' motion for assisting image registration. 2) Longitudinal image registration: To identify the boundary of free flap, and the sequential visible-light images generated during monitoring are aligned appropriately in the spatiotemporal domain for conducting pixel-wise analysis. 3) Vascular occlusion detection: The detection system is expected to alert medical personnel in advance and remove the influence of factors such as illumination variation. With those aforementioned abilities, the proposed system is expected to eliminate the error of illumination variation in intensive care unit (ICU) and reduce the deformation from patents' motion. Besides, the vessel occlusion may be detected earlier because of pixel-wise analysis. In our study, a visible-light camera and an infrared camera were attached to the metal arm together. This setup was used to capture different types of images simultaneously and provides the ability of analysis from two kinds of images in the future. Therefore, only the visible-light results are discussed in this paper.
Selecting the correct features is the most important aspect in object tracking. Optical flow is one of the most famous object tracking methods that has two branches: constantbrightness-constraint (CBC) and nonbrightness-constraint (NBC) optical flowmetry. The CBC optical flow has a constraint on the brightness constancy, whereas the NBC optical flow can be operated under varying lighting conditions. Moreover, famous optical flow methods such as the Kanade-Lucas-Tomasi (KLT) [35], [36] and Horn-Schunck (HS) [37] methods have a constraint that objects should be rigid [38]. In our study, the intervals of sequential images were short and constant. Thus, the brightness and deformation of images were approximately constant during a particular interval. In other words, the constraints of the aforementioned methods could be ignored.
To analyze the changes in a free flap with time, image registration is usually considered as the primary step before analysis. Image registration methods are categorized into rigid registration and nonrigid registration methods. Nonrigid registration algorithms are known for their use of bioimages because the human skin and tissues are soft and nonrigid [39]. However, nonrigid registration remains a challenge in computer vision because of its complex nonlinear transformation models. In general, nonrigid registration includes several steps, such as feature extraction, matching, transformation, and optimization [40]. In computer vision, a simple point set registration approach that directly associates the points was proposed by Scott and Longuet-Higgins [41]. However, the performance of the method is poor for nonrigid objects. Another famous method for point set registration is iterative closest point (ICP) [42]. This method iteratively calculates the least square results of every closest corresponding point between two point sets to minimize the distance between each pair. However, ICP is unsuitable for our study because providing two point sets adequately close to each other is impractical, especially because free flaps are nonrigid. In contrast to ICP, coherent point drift (CPD) is a probabilistic method. In CPD, the first point set is assumed to fit Gaussian mixture models (GMMs), and the Gaussian centroids of these models are used as the initialized points of the second set [43]. Once the two point sets are optimally matched, the correspondence is the maximum GMM posterior probability. Compared with other methods, CPD is more robust, especially in terms of nonlinear deformation and noise. However, patients may change the position of their head and thus seriously deform the flap. Moreover, there occur some interruptions from the error of tracking or the cover area of the gauze after medical personnel change the dressing. The aforementioned issues are severe challenges of image registration in our study.
For circulatory compromise detection, the color difference is the principle factor to determine the situation of a free flap in our study. According to the standard of observation methods, the color of the flap is pale when arterial occlusion occurs and changes to dusky when veins experience congestion. According to the rules, our detection method is expected to depend on the RGB intensity. However, our method is easily affected by the variation in illumination from the surrounding. In image processing, the issue of white balance occurs. Thus, the intensities of specific colors would be adjusted for rendering the images correctly. The visible-light images should be normalized or corrected before being analyzed. Popular white balance methods include the Gray World Algorithm and the maximal RGB value method. Hsu et al. proposed a white balance technique for scenes with two specific light types [44], and Gijsenij et al. proposed a color constancy algorithm for multiple light sources [45]. However, these methods may have interobserver errors because of the specified light types or the selection of sampling methods. These errors are more serious when they occur in time-course images. To analyze a large amount of correlated variables, factor analysis (FA) is the most popular method for data reduction. The aim of this method is to reduce the dimensionality with a minimum loss of information and to determine potential factors that have a low number of unobserved latent variables. After simplifying the description of these correlated variables, accurate results of circulatory compromise detection can be obtained.
The paper is organized as follows. Section II describes the architecture of the postoperative free flap monitoring system, including hardware and the algorithms of object tracking, image registration and circulatory compromise detection. Section III summarizes the experimental studies, which demonstrate that our postoperative free flap monitoring System achieves exact accuracy. Finally, discussions and conclusions are drawn in Section IV and Section V respectively.

II. MATERIAL AND METHOD
The proposed visible-light monitoring system is comprised of a photograph hardware module, an image processing module, and a circulatory compromise detection module. The pipeline, displayed in Figure 2, is described as following: 1) The hardware part: Before monitoring, the visiblelight camera would be adjusted to an initial position and aimed at the patient's free flap. Then, the images are captured and transferred to computer for image processing.
2) The image processing part: The motion trajectory of patient would be traced and face regions of sequential images during monitoring are recognized to reduce the computational complexity. Then, the feature points extracted from free flap are used to identify the region of free flap, and some of feature points are selected as corresponding points for our proposed image registration. After optimizing the registration results, the deformation images are obtained.
3) The circulatory compromise detection part: In order to overcome the influence of factors such as illumination variation, the registered images are analyzed by residual factor analysis to obtain the common factor and the specific factor. Then, the vascular occlusion of free flap is detected by evaluating those factors' change.
Moreover, the modules of image process and circulatory compromise detection (marked using red boxes) are focused on in this paper.

A. HARDWARE AND EXPERIMENT SETUP
To evaluate the accuracy of proposed system and the potential of clinical application, the experiments were verified using swine and human flaps. The animal study was reviewed and approved by the Institutional Animal Care and Use Committee of Taipei    A visible-light camera (α6000; SONY, Tokyo, Japan; 6000 * 4000 pixels) and an infrared camera (Spectrum 9000MB; United Integrated Services Company, Taiwan; 320 * 240 pixels, 7∼14µm) were operated simultaneously, and the arm can be manually adjusted by medical personnel. But, only the visible-light results are discussed in this paper. In animal study, swine (weight, 10-15 kg) were anesthetized generally by using intravenous isoflurane and dissected their pectoralis major myocutaneous flaps to tie up the pedicle vessels by rubber loops like blood flow controllers, which are shown in Figure 3 (a) and Figure 3 (b). After that, the flap was sutured back to its original position and the pedicle vessels were temporarily clamped to simulate vascular occlusion. Furthermore, the detection of occlusion type is also discussed in this study. Therefore, artery occlusion and vein occlusion were designed for this experiment respectively, which are shown in Figure 3 (c) and Figure 3 (d).
In the clinical trial, these participants were oral cancer patients who just had free flap surgery for the reconstruction of defects in the head and neck region. Then, they were moved to ICU and monitored by our system more than 24 hours. During monitoring, the medical personnel would use the manual observation method for vessel occlusion detection, and they may change patients' position and the cover area of the gauze.

B. OBJECT TRACKING
The optical flow method was used in this study to track a free flap. This method is used to calculate the apparent motion between two image frames taken at times t and t + t. Each pixel between the two image frames is defined as follows: If the pixel only moves within a small area, the brightness of the pixel is approximately the same. The Taylor series presented in equation (1) can be written as follows: H.O.T represents the higher-order terms, which can be ignored in our study because the motion of patients under anesthesia was limited. Then, the best optical flow estimation method involves minimizing the error between two image frames. Therefore, the equation is solved under the assumption that the difference between two image frames is zero. The equation is rewritten as follows: which results in I x = ∂I ∂x , I y = ∂I ∂y , and I t = ∂I ∂t are partial differential equations of x, y, and t , respectively; ∇I = (I x , I y ) is the brightness gradient; and v = (v x , v y ) is the velocity of a pixel (x, y) at time t. The result of object tracking obtained after calculating the velocity of the pixel by using the optical flow method is displayed in Figure 4. The volunteer moved marginally to simulate the movement of patients when they slept on their bed, and the movement was captured in two image frames by using a visible-light camera. The two images were overlapped, and the feature points around the free flap were connected using yellow lines. The motion of the volunteer was tracked successfully. However, the free flap was soft and nonrigid. Therefore, the area of the free flap was adjusted after object tracking.

C. IMAGE REGISTRATION
The scale-invariant feature transform (SIFT) provides good results of image registration despite issues related to the scale, rotation, illumination, and viewpoint [31]- [33]. However, image registration could not be suitably performed in this research by only using the SIFT. Sometimes, the registration VOLUME 10, 2022 results are influenced by the complex background and large movement or rotation of patients. This causes a decrease in the number of matching feature points or incorrectly matched feature points, which would lead to the accuracy decrease of the registration results. Hence, some algorithms were combined to obtain a suitable and accurate performance. Our image registration system ( Figure 5) performs six processes-face region recognition, SIFT and random sample consensus (RANSAC), template matching, CPD, thin plate spline (TPS), and gradient vector flow (GVF) snake deformation.

1) FACE REGION RECOGNITION
In a general case, facial recognition is used to extract features such as eyes and the nose to define the area of the face. However, conducting facial recognition is difficult when patients close their eyes or when an oxygen mask is on their face. Hence, the features of color and shape of the face were used in our study.
HSV (Hue, Saturation, Value) is an alternative representation of the RGB color model, and the transfer function is given as follows [46]: According to the experiment results, when 0.01 ≤ H ≤ 0.072, 0.2 ≤ S ≤ 0.68, and V ≥ 0.2, the most similar free flap colors were observed, which are shown in Figure 6 (b), Figure 6 (c), and Figure 6 (d) respectively. Moreover, other parts of the body, such as the neck, shoulder, and arm, were eliminated from the analysis area. Therefore, we excluded the connected components whose major-minor ratio of the ellipse shape were not between 1 and 2.5. The result of face recognition is illustrated in Figure 6. In contrast to Figure 6 (a), Figure 6 (e) presents that face without gauze, other parts of body and background are eliminated by HSV.

2) SIFT + RANSAC
The feature points of the SIFT were detected by conducting two steps, namely extraction and filtration. In the first step, the image was convolved using Gaussian filters at different scales and the differences of these consecutive Gaussianblurred images were then determined. The Difference of Gaussians (DoG) of the image between different scales (k i σ and k j σ ) is given as follows: where G (x, y, kσ ) is the Gaussian function at the scale kσ and I (x, y) represents the original image. Once the DoG images were obtained, the feature points were identified as the local minimum or maximum of the DoG images within different scales. The feature points were generated by comparing each pixel in the DoG images with its eight neighbors at the same scale and its nine corresponding neighboring pixels in each of the neighboring scales. A feature point can be represented at its scale by a magnitude m(x, y) and direction θ(x, y) as follows, (10) and (11), as shown at the bottom of the next page. In the aforementioned method, too many feature points were produced and some were unreliable. Therefore, the second step involved filtering out the points that exhibited a low contrast or were poorly localized along an edge. In this study, the threshold of SIFT was 0.005 for patients and 0.03 for the swine because the swine had larger free flaps than the patients, with a higher number of feature points.
After extracting the feature points and matching the feature points of two images by using the SIFT, some incorrectly matched points existed. These points seriously influenced the registration result. To eliminate the mismatched points, the RANSAC algorithm was used for estimating a transformation matrix (homography matrix) that suppressed the transformation errors of all matched points to the minimal value. First, four pairs of matched points were picked randomly to solve the transformation matrix as follows: where (x, y) and (x , y ) are the coordinates of the pairs. The best transformation matrix is established when the cost function is minimal.
The transformation model was developed using the estimated matrix to evaluate whether the matching error was less than the threshold. Therefore, all the matched points that complied with the model were considered inliers and the points that failed to comply with the model were discarded and considered outliers. Figure 7 illustrates the result obtained before and after using RANSAC to eliminate the mismatched SIFT feature points. Figure 7 (a) presents the original image with some mismatched pairs. When RANSAC was used to remove the mismatched pairs, all the resulting corresponding pairs were correct, as shown in Figure 7 (b).

3) TEMPLATE MATCHING
A free flap forms a small part of the face. Therefore, the corresponding pairs are sometimes less than ten when patients have large movements. This causes the image registration results to be incorrect. Thus, to compensate for the large shift, template matching was used to track and estimate the best position of the free flap between two images. First, the free flap region was divided into five blocks. Moreover, the regions were rotated by 45 • clockwise and counterclockwise, as shown in Figure 8. There were 15 blocks in total. Second, the template matching results were evaluated using normalized cross correlation (NCC), which has the following equation: where I is the compared image and T is the image of the block. After the most similar regions of those blocks were found, the group with maximal NCC was clustered using the mean shift method [47], which iteratively calculates the position of the center of mass to be used as the center of the next circle until the termination conditions are satisfied. Therefore, the template points set between two images could be easily extracted from the five blocks to increase the quantity of the feature points.

4) CPD
After using the SIFT and template matching, the feature points would be matched to obtain corresponding pairs through CPD. Hence, two point sets are given. The template point set Y = (y 1 , . . . , y M ) T (expressed as a M × 2 matrix) can be aligned with the reference point set X = (x 1 , . . . , x N ) T (expressed as a N × 2 matrix). The template point set Y is supposed to be the centroid of the Gaussian mixture model, and the reference point set X serves as the data point that must be fitted. If the transfer function X = T (Y , θ) exists, then the energy function can be written as follows: where λ is a weighting constant and φ(Y ) is a smooth function of motion. The aim is to minimize the energy function through estimation maximization to determine the parameters m(x, y) = (L (x + 1, y) − L (x − 1, y)) 2 + (L (x, y + 1) − L (x, y − 1)) 2 , (10) θ(x, y) = a tan 2 (L (x, y + 1) − L (x, y − 1) , L (x + 1, y) − L (x − 1, y)) .
(11) VOLUME 10, 2022 θ and σ . Then, the corresponding pairs between the two point sets can be found.

5) TPS
Our deformation model based on the TPS algorithm is expected to use these corresponding points to deform the n th image to increase its similarity to the reference image. The TPS model uses interpolation between these corresponding points based on the cubic spline function, and the deformed coordinates f (x , y ) can be represented as follows: where a l , a x , a y are the parameters of linear transformation, (x, y) is the coordinate of the reference image, n is the number of corresponding points, j is the j th corresponding points, w j is the nonaffine parameter, and U (r j ) denotes the shape function. The term U (r j ) represents the intensity of the force between the corresponding points and can be represented as follows: where r j is the distance between (x, y) and the j th corresponding point. The relationship between the corresponding points of the reference image and n th image is obtained through SIFT or CPD. The relationship can be used to calculate the unknown parameters a l , a x , a y and w j .

6) GVF SNAKE DEFORMATION
For making the deformed free flap edge suitably fit the reference one, the snake model was used to optimize the registration results [48]. However, the snake model has poor convergence to boundary concavities. Thus, the GVF snake model was used to improve the problem. The constraint energy of the traditional snake model was replaced by the GVF [49]. The GVF snake equation is represented as follows: where α and β are the weighting parameters that control the snake's tension and rigidity, respectively, and x (s) and x (s) denote the second and fourth derivatives, respectively, of x(s) with respect to s, which is the function of the curve. In the aforementioned equation, w(x, y) is defined as the GVF field and the energy function should be minimized.
where µ is a regularization parameter; u x , u y , v x , and v y are derivatives of the vector field along the x-and y-axes; and f (x, y) is an edge map derived from the original image. Therefore, the second term dominates the integrand when |∇f | is large. The second term is minimized by setting w = ∇f . The optimal result of fitting the free flap edge is shown in Figure 9. The free flap boundary in Figure 9 (b) was derived from the image deformed by CPD. The reference image did not fit well. After adjusting the boundary through GVF snake deformation, the boundary of the free flap was near the reference boundary.

D. CIRCULATORY COMPROMISE DETECTION
To prevent the influence of illumination change in the ICU, the factor of illumination variation would be eliminated by conducting residual FA before determining the color of the free flap. The flowchart of circulatory compromise detection is displayed in Figure 10. After the longitudinal free flap images are registered, areas around the free flap are selected as control groups. Figure 11 (a) presents that the yellow boxes are the selected control groups and the red box is the region of flap. Then, to increase the accuracy of analysis, the region of the flap is split into 16 blocks, and every block is compared with the control groups respectively, which is shown in Figure 11 (b). After normalizing the average color of every block and control groups, the normalized data is analyzed by residual FA. The equation of residual FA at time i is defined as follows: where SF i and CF i are specific factors and common factors, respectively, and X i and λ i are the normalized data and factor loadings, respectively. Moreover, CF i can be represented the variation in illumination from the surrounding. Thus, the   color variation of free flap is more unvarnished after CF i is eliminated. Therefore, SF i is the important factor to distinguish whether vascular thrombosis happened or not.

A. ERROR EVALUTION FOR IMAGE REGISTRATION
In the animal study, the swine was under anesthesia and 150 sequential images were captured (one per minute) after free flap surgery. The first image was the deformed image, and all the others were the reference images. These 150 images were well aligned in the spatiotemporal domain, and the parts of registration results are shown in Figure 12.
The red region represents the deformed free flap image at time t, which fits the reference image well. The green region represents the deformed free flap image at time t − 1, which can be compared with the red region and observed whether the swine had a large movement or not. Moreover, the results were evaluated using the DSC, as depicted in Figure 13. As the effect of anesthesia wore off, the swine had a large movement between 26 and 40 and between 106 and 122. These time intervals are marked by dashed red lines in Figure 13, and the results of image registration are shown in Figure 12 (b) and Figure 12 (c). The DSC results were gradually decreased because the free flap edge close to leg was deformed and squeezed more seriously to render the image registration error increased, which were marked by yellow arrows. However, the DSC (average: 0.961, standard variation: 0.01) was still reasonable for image registration.
In the clinical trial, the patient was monitored using our system after free flap surgery, and the evaluated results are shown from the time 1 to 60. The 60 sequential images were captured (one per minute) and well aligned in the spatiotemporal domain. The parts of registration results are shown in VOLUME 10, 2022   In this case, the patient presented a large motion and rotation, especially from 11 to 20 and from 51 to 60. Thus, the DSC results was influenced directly, as displayed in Figure 15. However, the DSC results under the severe influence were still higher than 0.93. Moreover, the image registration for the patient was more difficult than that for the swine. The DSC results of the patient (average: 0.959, standard variation: 0.011) still had high accuracy and were extremely close to the swine.

B. SIMULATED EXPERIMENTS FOR CIRCULATORY COMPROMISE DETECTION
To verify the validity of our circulatory compromise detection system, two experiments were designed to simulate the vascular thrombosis of the free flap with illumination variation  before clinical trial. Separate experiments were conducted for volunteer and animal study.
To simulate the situation of vessel occlusion, the black region as the volunteer's free flap was painted soft red at the beginning, and the color of free flap was gradually darkened at time 64, 91 and 121, which is shown in Figure 16 (b). Besides, the illumination was randomly changed during the experiment. The analysis results of residual FA are depicted in Figure 16 (a). The original data, the common factor and the specific factor are divided into three rows, respectively, and they are decomposed into the RGB data. The first row which represents the original data was influenced by illumination change and thus severely unstable and hard to analyze directly. The second row which represents the common factor was extracted by residual FA, and it indicated the variation in illumination from the surrounding. The last row is the specific factor which was without the influence of illumination change. In contrast to the original data, the specific factor was easily detected whether vascular thrombosis happened, which indicated that the color of the free flap was dusky at time 64, 91 and 121 (red arrow). Moreover, the intensity of green and blue decreased more than red, which was determined that the type of occlusion could be venous congestion because of the color of the free flap from soft red to dark red.
In animal study, the swine would rest for a while after the free flap surgery. Then, the free flap artery of the swine was tied up by a rubber band to simulate arterial occlusion from time 33 to 93. Moreover, the illumination varied during the interval time 64-82 to evaluate whether residual FA could be used to extract the common factor. The analysis results of the residual factor presented in Figure 17. The original image was influenced by illumination variation and obviously unstable during time 64-82. When the common factor was extracted successfully, the specific factor was analyzed without the  influence of illumination change. The red intensity of specific factor decreased when the artery was tied up, and the time interval of vascular occlusion was marked by the red dashed line. Moreover, the specific factor of the red data decreased more obviously than that for the green and blue data from the beginning of arterial occlusion, which implies that the color of the free flap gradually changed from ruddy to pale. Therefore, the detection system not only determined the moment of vascular occlusion but also the type of occlusion.

C. LONGITUDINAL ANALYSIS OF THE FREE FLAP IMAGE IN THE CLINICAL TRIAL
In the clinical trial, patient 1 had a free flap surgery and was monitored using our system from December 12, 2016. The nurse verified that venous congestion occurred at 15:02 on December 13. The residual FA results were selected from 11:00 to 15:02 on December 13. As displayed in Figure 18 (a), the common factor of illumination variation was successfully removed from the original data. The specific factor indicated that the color of the free flap was a little dusky at time 100 (red arrow), which indicated venous congestion. Venous congestion occurred considerably earlier than the time noted by the nurse (brown arrow). To verify the reliability of the detection system, patient 2, who did not have vascular occlusion, was monitored using our system. The residual FA results were selected from 2:59 on January 12, 2018, to 16:36 on January 13, 2018, as displayed in Figure 18 (b). The original data was influenced by a marginal illumination variation and slightly went up by physiological phenomenon. After the common factor was removed, the specific factor of the RGB data was found to be flat and stable. This implies that vascular occlusion did not occur. Therefore, our detection system is accurate and robust.

A. SEVERE CHALLENGES OF IMAGE REGISTRATION
In some severe cases, corresponding pairs between two images are less than ten owing to the change of patients' position or the cover area of the gauze. As shown in Figure 19, the patient captured ten images at different time had large movement or rotation, and the part of free flap in every image was covered by the gauze and the ventilator, which had the less corresponding pairs leading to the lower quality of image registration results. Thus, the deformed image (on the top) was used to track and estimate the best position of the free flap on the reference image by conducting matching template before image registration. As the results shown, the six images were successful to find the position of free flap (marked by the red region), even the patient had large movement or rotation. In the other cases, the free flap in the first two images was squeezed and excessively covered by the gauze, and the ventilator may be considered a part of free flap in the last image, which were leading to the poor matching results.
However, it still provided the rough free flap region for image registration, which would lead to the better image registration results.

B. A STEREO VISION MONITORING SYSTEM
To further increase the accuracy of image registration under severe situation, a stereo vision system may be a nice solution in the next step. To establish a three-dimensional model by using only one camera, Saxena and Liu used the multiscale Markov random field method and proposed a deep convolutional neural field model for estimating depths by using single monocular images [29], [30]. However, their method requires considerable prior knowledge and training data for calculation, and the results are insufficiently accurate because of the lack of sufficient depth information. He et al. proposed an approach in which one monocular image is used to recover the depth information of objects by applying the SIFT. They captured the first image and then moved the camera at a distance to capture the second image. These images were used to extract the corresponding points through the SIFT. Moreover, the extraction process can be reduced from two dimensions to one dimension by using epipolar geometry [34]. In the aforementioned method, a robotic arm is necessary for the medical personnel to adjust the position of camera automatically owing to the high technology background requirement. Therefore, the automatic stereo vision system is going to be a part of our study in the future.

C. DETERMINATION OF THE OCCLUSION TYPE
To evaluate the accuracy of the determination of occlusion type, 10 swine pedicle myocutaneous flaps were harvested and monitored vascular thrombosis by our system. The comparison of the residual FA results and occlusion type was summarized in Table 2. The specific factor was decomposed into the RGB data, and the last column was occlusion type which the swine had. As the results shown, venous occlusion was determined easily, which had large decrease in all RGB data. In contrast to venous occlusion, the variation in RGB intensity changed more slightly when arterial occlusion happened. However, the intensity of green and blue increased in most of arterial cases, that is obviously different from venous cases. Even though the intensity of RGB also decreased in a few arterial cases, it still could be distinguished owing to the range of variation.

V. CONCLUSION
The postoperative free flap monitoring system was based on one visible-light camera to detect circulatory compromise. It has many advantages such as fast operation, low cost, noninvasive behavior, contactless and nonradiative operation, and most detection techniques in clinic are difficult to have all of them. To conduct pixel-wise analysis and prevent the manual determination of the boundary of free flap, a longitudinal image registration without markers was proposed in this study, and the results were evaluated using the DSC. In the animal study, the 150 sequential images of the swine under anesthesia were well aligned after free flap surgery, and the average of DSC was 0.961 and the standard variation of DSC was 0.01. In the clinical trial, the 60 sequential images of the patient monitored by our system after free flap surgery were well aligned, and the average of DSC was 0.959 and the standard variation of DSC was 0.011. In some severe cases, patients had large movement and rotation, and the part of free flap even covered by the gauze, the proposed image registration still had reliable results. Moreover, to verify the validity of our circulatory compromise detection system, experiments were divided into simulation experiment and clinical trial. In simulation experiments results, the influence of illumination variation from the surrounding was eliminated successfully, and the analysis of specific factor in RGB data was indicated that the detection system not only determined the moment of vascular occlusion but also the type of occlusion. In clinical trial, vascular congestion was even detected much earlier than manual observation. Therefore, our monitoring system is expected to relieve the workload on medical personnel and could be a reliable tool for the surgeon to hold onto the chance of repairing the free flap with vascular obstruction.
HSIN-JU YU received the B.S. degree in applied mathematics from the National Chung Hsing University, Taichung, Taiwan, in 2016, and the M.S. degree in biomedical engineering from the National Taiwan University, Taiwan, in 2018. Her research interests include bioinformatics, data mining, and computer vision.
LI-WEI CHEN received the B.S. degree in biomedical imaging and radiological science from China Medical University, Taichung, Taiwan, in 2015, and the M.S. degree in biomedical engineering from the National Taiwan University, Taiwan, in 2017, where he is currently pursuing the Ph.D. degree with the Institute of Biomedical Engineering. His research interests include image segmentation, pattern recognition, and machine learning.
AI-SU YANG received the B.S. degree in medical imaging and radiological sciences from Chung Shan Medical University, Taichung, Taiwan, in 2016, and the M.S. degree in biomedical engineering from the National Taiwan University, Taiwan, in 2018, where she is currently pursuing the Ph.D. degree with the Institute of Biomedical Engineering. Her research interests include image segmentation, bioinformatics, and machine learning.
CHERNG-KANG PERNG received the B.S. degree from the School of Medicine, National Yang Ming Chiao Tung University, and the Ph.D. degree from the Department of Biomedical Engineering, National Yang Ming Chiao Tung University. He is currently the Division Chief of Plastic and Reconstruction Surgery, Taipei Veterans General Hospital. He is also an Assistant Professor with the Department of Medicine, National Yang Ming Chiao Tung University. His research interests include head and neck reconstruction, breast reconstruction, lymphedema reconstruction, lim reconstruction, microsurgery, cosmetic surgery, and laser treatment filler injection. VOLUME 10, 2022