A Comprehensive Survey Analysis for Present Solutions of Medical Image Fusion and Future Directions

The track of medical imaging has witnessed several advancements in the last years. Several medical imaging modalities have appeared in the last decades including X-ray, Computed Tomography (CT), Magnetic Resonance (MR), Positron Emission Tomography (PET), Single-Photon Emission Computed Tomography (SPECT) and ultrasound imaging. Generally, medical images are used for the diagnosis purpose. Each type of acquired images has some merits and limitations. To maximize medical images utilization for the purpose of diagnosis, medical imaging fusion trend has appeared as a hot research field. Different medical imaging modalities are fused to obtain new images with complementary information. This paper presents a survey study of medical imaging modalities and their characteristics. In addition, different medical image fusion approaches and their appropriate quality metrics are presented. The main aim of this comprehensive survey analysis is to contribute in the advancement of medical image approaches that can help for better diagnosis of different diseases.


I. INTRODUCTION
At the moment, the precipitous advancement in high technology and contemporary instruments result in making the medical imaging the extremely essential diagnostic tool for diseases and treatment [1]- [4]. For the purpose of medical diagnostics, various multimodal medical imaging techniques like MRI, CT, PET, SPECT are employed. Each modality is utilized in a particular application and performs a specific function. The SPECT and PET show the functional and metabolic activity, while the MRI, CT, and Ultrasound provide the anatomical structures of organs. For additional useful images with higher corresponding data and higher image details, medical image fusion is the perfect answer [5]- [9].
The associate editor coordinating the review of this manuscript and approving it for publication was Rajeswari Sundararajan. Therefore, medical image fusion multimodalities became a challenging research field, recently.
Lately, the optimization process has found concern from scientists to improve the execution of numerous signal processing algorithms. The optimization procedures offer the optimum standards for a scheme to accomplish the greatest feasible accomplishment. Global optimization schemes are a formidable methodology that can deliver better explanations for many challenges.
Various global stochastic optimization schemes have been performed effectively in medical digital image fusion such as GWO (gray wolf optimization) that greatly improves the performance of the fusion technique [74]. The CFO (central force optimization) technique which is depending on the gravity law and it has many benefits like the simplicity of implementation, straightforward mathematics involved, high convergence speed, and short processing time [81]- [84]. The PSO (particle swarm optimization) scheme is depending on swarm intelligence. The main advantages of PSO are very simple calculations, adopting the real number code, no overlapping and mutation calculation, the fast search, and a memory for updating the velocity [40]. The MCFO (modified CFO) algorithm incorporates the great merits of both PSO and CFO optimization techniques to have acceleration timevarying coefficients, a memory capability, and additional velocity into the position probe updated equation.
In fact, the local contrast enhancement techniques have a strong effect for improving the clarity and details information of images. Different techniques are used for medical images contrast enhancement like histogram equalization, intensity adjustment, histogram matching, and adaptive histogram equalization [81], [84].
The main target of such a review study is to give a collective survey of image fusion techniques and their applicability that could be helpful. A number of investigations have addressed the topic of medicinal imaging fusion from various viewpoints. In [55], a medical image fusion algorithm has been presented. This algorithm is based on a multi-wavelet basis and regional variance to prove that fusion procedure based on multi-wavelets with better performance than a single waveletbased fusion procedure. The fusion process in this case is done through the merge rule, which may or may not depend on the application, and whether or not the sub-band is merged.
In [57], the authors presented a study of a multi-modality medical images fusion algorithm based on DT-CWT that enhances the multi-sensor fusion procedure using DT-CWT instead of DWT implemented with the window-based fusion rule. This achieved much better performance than the pixelbased fusion rule. Also, the authors in [20] have presented an algorithm for fusing multi-focus and multi-modal images using DT-CWT decomposition before segmentation for perfect tumor identification.
In [63] the authors proposed a new scheme for a multifocus fusion procedure that is based on uniform discrete curvelet transform. This can manage the problems of conventional multi-scale analysis image fusion of rising data redundancy ratio and poor performance to achieve higher image quality.
Recently, many researchers have been interested in combining PCA fusion algorithm with other transform fusion techniques such as hybrid fusion approach using PCA and wavelet transform [66], PCA and contourlet transform [67], . . . , etc.
In [5], [6], the authors introduced a review study on medicinal imaging fusion approaches used in the health system to provide an overview of the fusion methods utilized in the applications of medical services, like curvature transformation, waveform, contour transformation, standing waveform transformation, and frame transformation.
In [81], the authors presented an optimized medical image fusion system using the Non Sub-Sampled Shearlet Transform (NSST) and MCFO for setting the optimal gain fusion parameter estimations. The MCFO optimization is employed to set the optimum gain parameter scores for the fusion process. The proposed algorithm implements a further enhancement step using adaptive and combined adaptive histogram equalization, and histogram matching to overcome contrast limitations and increase details in the fusion results. Finally, the proposed algorithms are evaluated using different quality metrics. The examined tests demonstrated that the proposed MCFO-based NSST has achieved an improved performance in terms of improving image visualization, enhancing local contrast, and increasing PSNR.
In [82], the authors introduced an optimized Discrete wavelet transform (DWT) fusion system using various wavelet families like Ddb1, Haar, db1, discrete meyer, and Coiflet 1. The proposed fusion system utilizes the MCFO scheme to provide the optimum fusion gain parameter scores. The proposed algorithm performance using various wavelet families is investigated and examined with standard quality measures for proving their superiority and validity with respect to DWT and PCA algorithms. The adaptive histogram equalization has been used for improving the proposed algorithm and providing higher image quality. The histogram matching and the adaptive histogram equalization are the best solution for local contrast enhancement, increasing PSNR, and improving image visualization.
In [83], the authors presented an optimized medical image fusion system that employs the Non Sub-Sampled Contourlet Transform (NSCT) and MCFO for setting the optimal gain fusion parameter estimations. The MCFO optimization is employed to set the optimum gain parameter scores for the fusion process. The proposed algorithm implements a further enhancement step using various local contrast enhancement approaches. Also, the proposed algorithms are evaluated using different quality metrics. The examined results demonstrated that the proposed MCFO-based NSCT has achieved an excellent performance with considerably high scores of edge intensity, average gradient, local contrast enhancement. VOLUME 9, 2021 This results in improving image visualization, enhancing local contrast, and increasing PSNR.
In [84], the authors presented an optimized medical image fusion approach using the DT-CWT, MCFO, and matched histogram. For achieving an optimum performance of DT-CWT, three approaches have been tested. The proposed optimized (OPT) fusion algorithms with optimum gain parameters improved image quality at a very short processing time and made a good benefit from matched histogram and adaptively histogram equalization for achieving an excellent performance with high image quality, the highest details information, and the best visualization for accurate diagnosis. The Max-OPT fusion technique with matched histogram and adaptive histogram equalization is proposed for extra improvement in quality factor, PSNR, local contrast, and image visualization. The Max-Max fusion technique has the highest quality factor but lower values of the other quality metrics. So, the adaptive histogram equalization is used to improve its performance and provide higher image quality. So, it can be applied in real time applications. The matched histogram and the adaptive histogram equalization are the best solution for local contrast enhancement, increasing PSNR, and improving image visualization.
The importance of studying medical image fusion algorithms lies in the objective analysis of the weaknesses and strengths of the techniques currently used and finding the appropriate solutions that achieve fusion of medical images with high quality and accuracy, which helps specialists in an accurate and precise diagnosis of the disease and then prescribe the appropriate treatment. Besides, clarifying the vital importance of multi-modality fusion process for the same patient in an accurate diagnosis of diseases and develop the medical radiology devices to enable them to fuse multi-modality images that can be used for computer-aided diagnosis applications and increasing the clinical applicability of medical images. Therefore, this research provides a comprehensive study of the latest methods implemented in medical image fusion.
This article is concerned with utilization of medical image fusion to obtain a fused image with as much complementary information, highest clarity, and best visualization as possible helping for accurate diagnosis and optimal therapy. The article objectives can be summed up as follows: 1) Review the preliminaries of medical imaging fusion.
2) Cover the medical imaging modalities with comparisons and includes the major applications of medical image fusion. 3) Introduce medical imaging fusion quality measures. 4) Compare the performance of some of the most currently used medical image fusion methods with comparisons. 5) Discusses medical image fusion and denoising techniques.
6) Explore the recent new trends and directions in medical imaging fusion. This research work is arranged as follows. Section II includes the preliminaries of medical image fusion. Section III covers the medical imaging modalities with comparisons and includes the major applications of medical image fusion. Section IV discusses image fusion quality measures. Section V explores the performance of some of the most currently used medical image fusion methods with comparisons. Section VI Section VI discusses medical image fusion and denoising techniques. Section VII includes some new trends in the medical image fusion subject. Section VIII gives the conclusions and directions for future works.

II. PRILIMINARIES OF MEDICAL IMAGING FUSION
Image fusion may be considered as a merging tool for relevant knowledge from a series of medical images into one enlightening and comprehensive image. More accurately, fusion is the incorporation of evidence from a set of recorded medical images not including the establishment of misrepresentation [2]- [7].

A. DEFINITION OF FUSION
There are so many applications of image fusion that require high spatial and spectral resolutions within a single image besides achieving higher image quality. This can be provided using a fusion procedure having several advantages [8]- [11]. A comparative study of some main benefits, difficulties, and applications of the fusion process can be observed through this review to describe the importance of the topic of interest and its implementations. Many good features distinguish the fusion process in the medical field such as: 1) Accurately provides the image location of lesions and significantly reduce the surgical risk. 2) Extraction all the advantageous info from source images into a specific image. 3) Fusion of images can improve reliability and capability by complementary information. 4) It is convenient for the detection and classification of numerous diseases. 5) Fusion process reduces the data storage required and time for transmission. 6) It can provide accurate information for a precise localization and size estimation for tumors.
On the contrary, other difficulties may face the fusion process like: Input images must be accurately registered before the fusion process, feature information extraction is required, during the fusion process, noise can affect the fused image, some color artifacts can be produced due to the transformation used in the fusion techniques, the dissimilar illumination problem of the fused image, processing of data is slow when images are fused, and finally, more than a source image is required for the fusion procedure. The main application for implementing medical image fusion process including: Diseases detection, classification, and segmentation, Computer-aided-diagnosis, Helping the specialists in Diagnosis abnormalities in medical images, clinical application, diagnosis and treatment of diseases.

B. TYPES OF IMAGE FUSION
The image fusion process may be employed in three different manners which can be classified as; at a low level, average, and high-level; or pixel element, feature, and determination levels [12]- [15]. Pixel level fusing the individual pixel values only. Feature level fusing the segmented regions of input images considering their properties. Decision level fusing the segmented regions of input images considering their initial object detection and classification.
Lately, scientists have proved that it is further significant to merge items or areas rather than picture elements. Different advantages can be found in the region-based algorithm over pixel-based algorithm as it is a smaller amount of hypersensitive to the commotion, has brighter differentiate, and a smaller amount of influenced by misregistration, but at the rate of complication. This was proved by John Lewis and Robert Callaghan in their research in 2007 [12].

C. FUSION CATEGORIES
According to the nature of the images to be fused, fusion can be categorized as follows [13]- [22]: Multi-view fusion: The images to be fused are of the identical sense modality and carried at the equivalent moment in time, but under dissimilar conditions and the main goal of the fusion process in this category is to have all the complementary information under the different conditions in the fused image.
Multi-temporal fusion: The images to be fused are of the same modality too, but they were taken at different times. In this case, the fusion process is performed by subtracting two or more images, and the main purpose of the fusion, in this case, is to detect changes in the scene at different times.
Multi-focus fusion: The images to be fused are divided into regions and the fusion is applied to have a fused image.
Multi-modal fusion: The images to be fused are of different modalities and the main goal in this category is to have a fused image that contains information as much as possible from the different modalities without losses in the overall meaning of the image.

D. MULTI-MODAL FUSION FOR MEDICAL IMAGES
The medical image fusion process, especially the multimodal medical image fusion process, aims to enhance the imaging quality by reducing the dismissal to improve the medical pertinency of the medical images in medical diagnosis and medical problems evaluation [23].
The fusion algorithms are input dependent. Therefore, building a fusion algorithm depends on three main aspects that must be taken into account the imaging modality used, organs to be imaged, and the algorithm of fusion implemented.
The imaging modalities area focuses on imaging modalities related to medical image fusion, their identification, and improvements.
Fusion algorithms focus on the design and enhancement of different algorithms for the fusion of medical images and their assessment [24].
Organs studies focus on the application of the fusion process in medical images of human organs of concern such as brains, breast, lung . . . , etc. [25].

III. MEDICAL IMAGING MODALITIES
Numerous medical imaging modalities exist with each having distinctive characteristics that provide various sources of information that facilitate the study of organs, diagnosis of diseases, follow-up treatments of patients, and further processing procedures such as the fusion process.
These modalities can be classified into five types: microscopy, 3D reconstruction, visible photography light, radiology, and printed signals (waves). For diagnostics purposes and treatments, radiology imaging modality is the most frequent assistant imaging modality such as CT, MRI PET, SPECT, and ultrasound [26]- [38]. A brief discussion about these modalities is introduced in Table 1 [52]- [70].
A remarkable note from the introduction to these modalities is that medical image quality characteristics have some common limitations of radiological imaging. They are spatial characteristics of the imaged body part that describes its relative size, shape, and position within the body. Also, artifacts, noise, blurring, and contrast sensitivity of the scanned image formed are of concern. These factors can affect scanned images, unequally. For qualitative and informative medical images, hybrid imaging sounds to be a good choice.
In general, the main objective of medicinal imaging fusion is to provide a full description of the anatomical structure of organs and follow up the function, behavior, and interaction of cells inside these organs. Hence, medical imaging can provide images having the required information for diagnosing diseases and can be used for numerous medical applications. Medical image fusion is also applied to several medical fields that require higher resolution images for different parts of the human body such as brain, lung, liver, heart, . . . , etc. [28], [39]- [50]. Table 2 introduces the applications of image fusion between different imaging modalities in the medical field on the human body that mix between mutually functional and anatomical imaging for additional informative and qualitative images [71]- [84].

IV. FUSION QUALITY EVALUATION METRICS
Many features of medical images affect the performance of medical image fusion that can be measured and evaluated based on standard quality metrics. These metrics include average gradient, entropy, edge intensity, standard deviation, local contrast, PSNR, and the Xydeas and Petrovic [35].

A. AVERAGE GRADIENT
It is utilized to express the details or texture variations number within the image. It may be expressed for an image f as: (1) where M and N are the image dimensions.

B. LOCAL CONTRAST
It can be utilized as an image quality index and for view clarity. It may be expressed as: where µ target and µ background define the gray-level average of local region of interest, and the image background average. A high local C value means more image clarity.

C. STANDARD DEVIATION
It can be utilized for determining how much data variation is from the average value. The image is termed as being of better quality when its standard deviation (STD) scores is high. The STD can be expressed using Eq. (3) as:

D. EDGE INTENSITY (EDGE I)
High image edge intensity indicates a high image quality. The image Edge intensity S can be expressed using the Sobel operator as: where and Image entropy measures of the information amount inside in the image. If the image pixel levels probability density is determined, then the image information amount can be expressed using the entropy E as: where L is the gray levels number of the image.

F. PEAK SIGNAL-TO-NOISE RATIO
The Peak Signal-to-Noise Ratio (PSNR) measure depends on the Root Mean Square Error (RMSE) among the interest image and the reference one. It can be expressed as: where f max defines the maximum image pixel score.

G. XYDEAS AND PETROVIC METRIC
The Xydeas and Petrovic (Q ab / f ) measures the transferred edge information amount from the original image to the fused image. This metric can be expressed as: where Q af (i,j) , and Q bf (i,j) represent the edge information scores, and W af (i,j) , W bf (i,j) represent their respected weights.

V. PERFORMANCE EVALUATION OF SOME EXISTING MEDICAL IMAGE FUSION TECHNIQUES
In this section, an investigation of the performance of some existing medical image fusion techniques has been introduced [57], [60]- [62], [70], [71], [81]- [84]. These algorithms applied on several CT and MR image datasets and evaluated using various quality metrics. Samples of the evaluated results are presented in Tables 3 and 4.

VI. MEDICAL IMAGE FUSION AND DENOISING TECHNIQUES
In addition to the fusion process for image enhancement, denoising techniques can also increase image quality by suppressing noise from images. Images could have different types of noise that affect image visualization like Poisson noise, salt and pepper noise, Gaussian noise, speckle noise, Brownian noise, film grain noise, shot noise, quantization noise, and an-isotropic noise [68]. In medical images, these noise types could affect medical diagnosis badly resulting in an inaccurate diagnosis. This motivates researchers for using denoising techniques for better visualization and higher image quality. Brief classifications of denoising techniques are presented in Fig. 1.
These techniques are used for retrieving the original image from the noisy image and provide enhanced image quality. The type of noise in the image determines the required filter used. Many researchers have discussed denoising problems and some of them are introduced in Table 5.

VII. MEDICAL IMAGE FUSION NEW TRENDS
Throughout this survey, several basic definitions for medical image fusion have been declared as it presented some existing survey studies in the scope of medical image fusion.
It also presents many kinds of research that are concerned with medial image fusion techniques and their applications in different medical fields, and others that are interested in developing, evolving, and enhancing these techniques. The objective is to handle different problems for better image visualization and higher quality that are required in various medical implementations. Despite that, medical image fusion is a promising research area that is full of benefits for different applications and investigations that may be explored in the future. Hence, this study has identified some relevant issues and novel trends for the medical image fusion domain that deserve further investigations such as: 1) Hybrid image fusion approaches that combine two transformations before application of fusion rules. This makes good use of the characteristics of both transforms for better enhancement of fused image visualization, higher image quality, and suitable processing time. Hybrid Techniques SVM-Shearlet, Wiener filter-Shearlet, Curvelet-Contourlet, DWT-Shearlet can be suggested for better performance. Many works have been introduced for this objective such as a fusion technique for Lungs Tumor images introduced by Naveenadevi R and Nirmala S of a title ''Fusion of CT-PET Lungs Tumor images using Dual-Tree Complex Wavelet Transform'' [70]. They present a solution for further image processing such as target identification by combining multi-image information in one scene. The proposed fusion technique is based on Dual-Tree Complex Wavelet transforms for image decomposition and the Non-Subsampled Contourlet transform rule is employed for obtaining the high and the low-frequency coefficients of the fused image. Another work was introduced by Sivakumar and Helenprabha entitled ''Hybrid medical image fusion using wavelet and curvelet transform with multiresolution processing'' in 2017 [71]. They proposed an algorithm for enhancing the fused image quality by combining wavelet and curvelet transform techniques after the decomposition stage using a sub-band coding algorithm that performs the Multi-Resolution Analysis (MRA). This technique improves PSNR by 5 dB better than curvelet fusion and by 10 dB better than wavelet transform fusion algorithm. Also, the root mean square error has been decreased.
2) Implementation of the optimization techniques where the parameters of the fusion algorithm should have the optimum values to achieve the best performance and perfect image visualization. This topic is a future trend and a few kinds of research introduced work in this area. Ebenezer Daniel and J. Anitha introduce the topic of ''Optimum spectrum mask based medical image fusion using Gray Wolf Optimization'' [72]. This research is interested in providing the best scale value selection for improving the quality of fusion. The proposed OSMF (optimum spectrum mask fusion) is based on a conventional Gray Wolf Optimization (GWO) scheme. This spectrum mask technique provides the swiftest and dynamic scale selection for spatial domain and transform domain fusion algorithm. Another research using Genetic Algorithms (GAs) to obtain the more optimized fused image produced from discrete wavelet transform (DWT) fusion scheme [73].
3) Color artifacts that result from transformation-based algorithms or obtained from imaging mode are an important matter that should take much more concern. In B-mode imaging, parallel beamforming is used to increase image quality, but this introduces artifacts in the images. These artifacts are reduced in the research of a title ''Reducing color flow artifacts caused by parallel beamforming'' by Hergum et al. [74]. The artifact reduction is done by interpolation of the autocorrelation estimates obtained from overlapping receive beams.
4) The problem of uneven illumination of the fused images as the implemented fusion rules could result in darker images or brighter images to enhance image details. Hence, solutions for optimum fusion rules that do not disturb the illumination of the fused image should be developed. Hiroshi Tsutsui and S. Yoshikawa proposed an algorithm for reducing halo artifacts [75]. They designed a cost function that is based on illumination characteristics. The parameters of the cost function are adaptively adjusted to reduce these halo artifacts and maintaining image contrast. 5) Noise effect is also a matter of interest that has been neglected in many kinds of research. There are so many types of noise that could affect the fused images and denoising procedures are required for clear visualization [76]- [80].
6) Practical implementation of fusion algorithms is still for soft applications only where two input images are applied only to an algorithm after capturing from different sensors or for different times. Researchers should pay more attention to developing medical devices that can produce different modalities in the same device which could be inputted to an implemented fusion algorithm to obtain an exclusive fused image of higher quality. These suggestions could be very useful and provide more informative images at a low cost for the patient as more than one image can be taken through the same device and at the same time. The final obtained images could provide excellent visualization for an accurate diagnosis. 7) Evaluation metrics that are used to evaluate the performance of fusion algorithms through complexity, processing time, throughput, PSNR, image entropy, edge intensity, similarity index, mutual information, and other evaluation metrics should be developed to offer a correct approximation of performance of fusion scheme and fused image characteristics that enhance visualization and increase image quality.

VIII. CONCLUSIONS AND FUTURE WORKS
This paper covered a very hot area of research, which is the integration of different imaging modalities for better diagnosis. Different imaging modalities have been covered in detail by highlighting their advantages and disadvantages. Different approaches have been considered and compared for this objective. The general description and mathematical formulation of these approaches have been presented. In addition, a general discussion has been presented on how to develop new directions for multi-modality medical image fusion. The work in this paper may contribute in the development of automated medical image diagnosis systems. Future Directions of medical imaging fusion may include: 1) Application of the optimization schemes on different factors of fusion algorithms that provides the optimal standards and realize the greatest accomplishment and ideal image vision; 2) Studying the effect of other optimization techniques on the performance of fusion techniques such as GWO, GAs, and PSO . . . etc. to find the best technique that achieves the best performance; 3) Investigating other multi-scale transforms for medical image fusion such as Ridgelet, K-SVD, Tetrolet, and Ripplet . . . etc.; 4) Applying efficient denoising techniques on images before fusion for higher details information, better clarity and image visualization; and 5) Practical implementation of fusion techniques using Field Programmable Gate Array (FPGA). Teaching Staff Member with the Department of Electronics and Electrical Communications, Faculty of Electronic Engineering, Menoufia University. He is a coauthor of about 200 papers in international conference proceedings and journals, and four textbooks. His current research interests include image enhancement, image restoration, image interpolation, super-resolution reconstruction of images, data hiding, multimedia communications, medical image processing, optical signal processing, and digital communications. He was a recipient of the Most Cited Paper Award from the Digital Signal Processing journal in 2008.
GAMAL G. N. GEWEID is currently working with the Department of Engineering and Technology, Worldwide College of Aeronautics, Worldwide Campus, Embry-Riddle Aeronautical University, Daytona Beach, FL, USA. He has a solid background in physics, mathematics, biology, radiology and imaging sciences, and strong programming skills in MATLAB, Python, and C++ programming languages. He has a record of research publications and four patents, all in the field of biomedical engineering. His research work aims in the long term to develop realistic three-dimensional anatomically based electrical models of healthy and diseased neurons based on nanotechnology, in addition to developing novel high-resolution and fast imaging methods for neurological diseases, establishing risk stratification algorithms, and developing new tools for detecting diseases in early stages, optimal diagnostic intervention tools for rehabilitation, and electrophysiology for pathological electrophysiological abnormalities identification. He has experience and interests in using deep-learning, machine learning, image processing, signal processing, radiology and imaging sciences, multimodality medical imaging (ultrasound, PET/CT, and MRI), CT image reconstruction, mechatronics, medical robotics, biomedical instrumentation, computer vision, pattern, recognition, nano-robots in medicine, software reliability and modeling, biosensors, imaging and information technology in biomedicine, excellent analytical, courses development, and problem-solving skills.