Image-Based Artefact Removal in Laser Scanning Microscopy

Recent developments in laser scanning microscopy have greatly extended its applicability in cancer imaging beyond the visualization of complex biology, and opened up the possibility of quantitative analysis of inherently dynamic biological processes. However, the physics of image acquisition intrinsically means that image quality is subject to a tradeoff between a number of imaging parameters, including resolution, signal-to-noise ratio, and acquisition speed. We address the problem of geometric distortion, in particular, jaggedness artefacts that are caused by the variable motion of the microscope laser, by using a combination of image processing techniques. Image restoration methods have already shown great potential for post-acquisition image analysis. The performance of our proposed image restoration technique was first quantitatively evaluated using phantom data with different textures, and then qualitatively assessed using in vivo biological imaging data. In both cases, the presented method, comprising a combination of image registration and filtering, is demonstrated to have substantial improvement over state-of-the-art microscopy acquisition methods.

Abstract-Recent developments in laser scanning microscopy have greatly extended its applicability in cancer imaging beyond the visualization of complex biology, and opened up the possibility of quantitative analysis of inherently dynamic biological processes. However, the physics of image acquisition intrinsically means that image quality is subject to a tradeoff between a number of imaging parameters, including resolution, signal-to-noise ratio, and acquisition speed. We address the problem of geometric distortion, in particular, jaggedness artefacts that are caused by the variable motion of the microscope laser, by using a combination of image processing techniques. Image restoration methods have already shown great potential for post-acquisition image analysis. The performance of our proposed image restoration technique was first quantitatively evaluated using phantom data with different textures, and then qualitatively assessed using in vivo biological imaging data. In both cases, the presented method, comprising a combination of image registration and filtering, is demonstrated to have substantial improvement over state-of-the-art microscopy acquisition methods.
Index Terms-Image restoration, image processing, laser scanning microscopy.

I. BACKGROUND
A S A result of recent technical developments in intravital microscopy (IVM) it is now possible to observe dynamic biological processes such as tumour growth with adequate brightness and contrast, over an extended range of penetration depths, and at high temporal and spatial resolutions [1]. Equally importantly, advances in laser scanning microscopy (LSM) and fluorescent probes enable not only visualisation of such processes but also quantitative measurement. This holds the promise of quantitative biology to which this paper forms a contribution. Increased spatial resolution, together with tiling of the field of view, enables the scanning of significantly larger areas of a specimen, and in our case to analyse the tumour microenvironment. However, in practice, the acquisition of large images (typically, 8192 × 8192 = 64 megapixels with the newest generation of LSM) compromises image quality, which in turn challenges quantitative analysis. Image quality in the majority of imaging modalities including optical microscopy is a tradeoff between several factors, of which the most important are: image resolution, signal-to-noise ratio (SNR), and acquisition speed. The two main artefacts that degrade image quality are: motion of the specimen during acquisition, due to cardiac, respiratory, or other muscle contraction; and variation in spatial image consistency due to laser motion. While a number of different frameworks for specimen motion induced artefacts are already widely studied, e.g., [2]- [7], the distortions caused by the laser speed have not been addressed in the literature [8], [9]. When an image is acquired, the speed of the laser varies, causing artefacts such as jaggedness and substantially reducing the often poor SNR. Manufacturers are aware of such artefacts and offer methods to compensate for it. Such methods remain, however, both proprietary and limited, especially when the microscope is pushed to its limits in preclinical research where both fast acquisition speed and high spatial resolution are required for quantitative analysis. This is the case in our work in modelling and measuring tumour growth. Moreover, naturally occurring artefacts specific to optical LSM acquisition are further exaggerated during high-speed acquisition, and significantly reduce the ability to image tumour growth at the cellular resolution with a SNR sufficient for quantitative analysis. Overall, these effects limit the opportunity for new discoveries in in vivo cancer biology. This is the problem we address in this paper to enable fast restoration of high-resolution, sufficient SNR, and artefactfree intravital LSM for cancer biology and imaging research. The high temporal and spatial resolution in LSM imaging is also a prerequisite for monitoring various dynamic biological phenomena including neuronal or glial activation in the brain [10]- [13]. Similarly, at a subcellular level, high-speed acquisition enables observation of calcium signalling events such as calcium sparks, calcium blips or calcium puffs [14], [15].
We present a framework for the removal of jaggedness artefacts in LSM images based on a combination of two efficient image processing techniques: deformable (non-linear) image registration to estimate local displacements; and image filtering to increase the final quality of the image. We show that our framework enables restoration of distortion-corrected images from acquired distorted images. We stress that our framework neither requires prior knowledge of the microscope settings (or control system) nor information from jaggedness-free images.
The manuscript is organized as follows. First, we describe the source of jaggedness artefacts in LSM (Section II-A). We describe in detail how to incorporate efficient deformable image registration to compensate for the local distortions caused by the unknown, variable motion of the microscope laser (Section II-B). We then introduce a fast, post-image-registration weighted local filtering method applied to reduce the level of noise and therefore to increase the quality of images (Section II-C). In Section III, we describe the experimental platform used to evaluate our framework, and in Section IV we present the experimental results. The manuscript concludes in Section V.

A. Problem Statement
In Laser Scanning Microscopy (LSM), the imaging data are acquired sequentially pixel-by-pixel along a predefined path to cover the imaging field of view or the specimen. The predefined laser acquisition trajectory can either be unidirectional or bidirectional (see the top row in Fig. 1).
In unidirectional line acquisition, the laser scans a specimen along a pre-determined line; upon reaching the end of this line, the laser moves back to the beginning of the next line. Bidirectional line acquisition decreases the overall acquisition time by moving from the end of the scanned line directly to the end of next line and from the end to the beginning of this line. Although bidirectional line scanning is faster than unidirectional line acquisition, it poses a greater challenge since it is difficult to maintain spatial consistency of imaged data as the neighbouring lines are scanned in opposite directions. More precisely, the spatial image inconsistency is caused by the varying speed of the forward and backward acquisitions, which are normally not symmetric functions since the (unknown) laser control system involves an acceleration that is typically different from the subsequent deceleration. Such spatial image inconsistency is particularly apparent when the laser acceleration (or deceleration) approaches the maximum provided by the microscope specification, which is based in turn on unknown motors and associated control algorithms. Generally, the precise parameters of the control system for the transport of the laser in either unidirectional or bidirectional line acquisition mode are not published by microscope manufacturers. Nevertheless, in engineering practice, such trajectories are typically realised using a bang-coast-bang controller, comprising an initial constant acceleration, followed by a constant velocity (coast), followed by a constant deceleration. Assuming such a control regime, the difference between the forward and backward bidirectional line acquisition options equates to the difference between the acceleration and deceleration. These are quite different physical processes with different values. Therefore, the spatial image inconsistency severely deteriorates the quality of acquired images by introducing geometric distortions such as jaggedness artefact, and thus significantly limits in practice the use of fast, high-resolution microscopy imaging techniques for the fast bidirectional mode. A typical example of the jaggedness artefact in a bidirectional line acquisition is presented in Fig. 1. As is readily apparent, the geometry of the specimen is severely distorted by the jaggedness artefacts, leading also to the reduced contrast and poorer SNR in the acquired images. Such image quality degradation becomes a major limiting factor for quantitative image analysis at higher spatial resolutions. For example, this would significantly impact on segmentation and measures of the size of an object of interest, for example the neovasculature developed by a tumour [16], [17]. More importantly, such effects would significantly impact on the measurement of change, since the error in a difference is the sum of the errors at the two time points.
The primary reason to use the bidirectional image acquisition in our application, monitoring in vivo tumour growth, is the speed of acquisition that is almost twice as fast as the unidirectional acquisition. The bidirectional acquisition reduces the total imaging time by half, and therefore allows to scan the whole sample covering the tumour microenvironment within the time limit specified in the imaging licence and ethical approval. All animal studies were performed in accordance with the Animals Scientific Procedures Act of 1986 (UK) and Committee on the Ethics of Animal Experiments of the University of Oxford under the Project License (PPL: PCDCAFDE0).

B. Efficient Estimation of Local Displacement
In order to restore quality of an image, we first aim to recover geometrical image consistency of the specimen via compensation for local displacements caused by the variable speed of laser scanning of the specimen using the bidirectional line acquisition. To this end, we estimate a local displacement u between the sequentially acquired lines i n that constitute a whole image I, where n = 1 . . . N denotes a line in the image. For example, during the first phase line i 1 is acquired (the forward acquisition), and on the return the laser scans line i 2 (the backward acquisition). For each pair of lines (scanned forwards and backwards) estimation of the local displacement u can be defined as the optimization of a generic cost function (u) as follows: where sim denotes a dissimilarity term between the lines i, reg denotes a regularization (smoothness) term of the local displacement u, and α is a weighting parameter. Since we assume that each backward acquired line i n should be similar to the two nearest forward acquired lines, i n −1 and i n +1 , we use the sum of the squared differences (SSD) as the dissimilarity measure sim as follows: where x denotes spatial position along the line i. The dissimilarity measure can include all available channels to drive the registration process. Further, we assume that the estimated displacement u causing the jaggedness artefact is a locally smooth function (that has continuous derivatives). For this reason, we choose a local diffusion model to regularise the displacement: The optimization of the cost function given by Eq. (1) is done using the iterative efficient second-order minimization Gauss-Newton scheme, and multi-resolution optimisation is used based on a Gaussian image pyramid for the presented registration to improve the overall performance [18].

C. Fast Post-Acquisition Denoising
The quality of data obtained from the fast Laser Scanning Microscopy (LSM) is limited in practice as it is inevitably a trade-off between several factors: the overall acquisition speed; spatial and temporal image resolution; and SNR. Increasing the speed of LSM acquisition can, in general, be done either by increasing the speed of the laser movement, or by using bidirectional scanning protocol (as explained in the previous section). Increasing the speed of the laser reduces the number of photons collected during scanning, implying poorer SNR. To increase the SNR without reducing the laser speed, a line averaging mode has been proposed that repeats the line scan to collect more excitation light from the specimen. However, repeating the line acquisition, in turn, increases the overall image acquisition time proportionally to the number of repeated scans (and additionally can increase photobleaching). Furthermore, if the line averaging is performed without prior correction of the jaggedness artefact, it results in the blur that is often seen in practice, and severely restricts the final spatial resolution (see Fig. 4(c)). Since fast high-resolution imaging acquisition is required to facilitate observation of complex, dynamic, biological processes, the quality of single line acquisition is thus severely degraded resulting in poor image contrast and low SNR.
In order to provide data with the highest possible spatial and temporal resolution for advanced quantitative image analysis, we use single line acquisition with an additional image denoising algorithm. Instead of using sequentially repeated images and averaging them, we use locally weighted filtering for a single bidirectional acquisition (after performing the jaggedness correction described in the previous section) to increase the SNR, and thus improve the final quality of the restored image. Following previously published applications of edge-preserving image filters for dynamic optical imaging [19], we use guided image self-filtering (GIF) [20]. The GIF employs a locally weighted averaging filter, which is computationally attractive since its computational cost is independent of the filter size (contrary to bilateral image filtering [21]). The GIF is defined as follows: where O is a filtering output, I is an input image, G is the guidance image, ω k is a local window centred at pixel y where μ z and σ z are the mean and variance of image G in ω z , respectively, |ω| is the number of pixels in ω, and η is a regularization parameter supplied by the user. The filtering algorithm that we use exploits information provided in the input image I to increase the SNR in the output image O. The presented method has the added advantage that it does not involve repeated line acquisition, so it does not increase the overall acquisition time.

D. Summary of Proposed Methodology
We proposed a method for removal of the jaggedness artefact, which combines two crucial image processing elements: image registration and image filtering. First, for the input image -that is, the image acquired using fast bidirectional line acquisitionthe adapted image registration efficiently estimates the local displacement field for each acquisition line, compensating for the shifts caused by the varying speed of the laser. Second, the output registration image is filtered using a fast, locally-weighted, self-guided image filtering algorithm to further increase the SNR without reducing the original image resolution.

A. Phantom Data
To assess the assumptions underlying our image restoration framework, we first prepared and scanned an image phantom with different texture patterns (Zeiss APO Calibration objective). The phantom was scanned several times with different microscope settings: unidirectional and bidirectional modes, as well as without and with two and four times repeated line acquisitions. All phantom scans were also repeated for three different speed setups: 5, 7 and 10 (the maximum possible on our LSM). A total of 27 separate images were acquired. Fig. 4(a) and Fig. 4(b) show the phantom scanned in unidirectional (ground truth) and bidirectional line acquisition using the highest possible line acquisition speed. The microscope system settings for the phantom study is listed in Table I.

B. Image Quality Assessment
Inevitably, we lack ground truth in the form of the microscope laser control system. For this reason, we evaluate performance using image quality measurements. We used three commonly known criteria to assess image quality: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and mean-squared error (MSE). The default parameters for the image quality assessment were used for the SSIM [22]. The measurement of image quality is based on a phantom image using single unidirectional line acquisition mode (denoted un1 in Table I) as a reference for all criteria as it represents the fastest possible imaging acquisition without introducing geometrical distortions.

C. Experiments
The experimental framework is as follows: First, the image registration (Section II-B) and image filtering (Section II-C) algorithms were implemented using custom-designed routines in Matlab. Specifically, for image registration, linear interpolation with a Neumann boundary condition is used. To improve the convergence rate, we employed a two level multi-resolution scheme with downsampling factors of [2 1]. The maximum number of iterations was set to 15 at each level. The weighting parameter α for image registration was set to 1.0, based on the dense parameter search obtained for the phantom data (see Fig. 2.) These were used for all of the experiments presented in this manuscript. For the filtering procedure, we employ a dense search of the best filter parameters (Degree of Smoothing η, and Size of Neighbourhood |ω| from Eq. (5)) to achieve the highest quality of the final images. The results of the dense parameter search obtained for the phantom data are presented in Fig. 3.
Raw images from microscope manufactures format were transferred to Matlab using the Bio-Formats toolbox, a standalone Java library for life science image formats.

A. Phantom Data
In this section, we present the results of both the quantitative and qualitative evaluation using the phantom data. The quantitative results of image restoration on the phantom data are presented in Table II for the microscope Zeiss LSM 880. As shown in Table II, our method (denoted by ou1 and ou1F) gave marked improvements in the image quality criteria as compared to both bidirectional line acquisition without (nb) and with the vendor's proprietary shift correction (cb). The results show that the method is capable of restoring images with a quality (for the SSIM measure) similar to those acquired using the two unidirectional line acquisitions with averaging (un2). Our method produces lower MSE and PSNR than un2, however, note that the unidirectional line acquisition with averaging requires at least four times more time to acquire data than the bidirectional line acquisition used as an input for our method. For increased clarity of presentation, a typical zoom view of the phantom data restored using different approaches is shown in Fig. 4. To compare full-size restored images, we refer the reader to the supplementary materials provided on-line with the manuscript. Furthermore, the quantitative results from Table II are also illustrated in Fig. 5 for the SSIM. These are consistent with the visual inspection of the restored images shown in Fig. 4 and the quality criteria shown in Fig. 5.
In summary, our method applied to raw data acquired using the bidirectional line acquisition demonstrates encouraging performance, especially when compared to the much slower unidirectional line acquisition. Our method yields a geometrically plausible image of the field of view (when compared to the unidirectional line acquisition), without additional equipment for monitoring and calibrating the current laser position, and furthermore it shows superior results in terms of the length of acquisition: un2 requires roughly 4 times longer than using single bidirectional acquisition with our restoration method.

B. Estimation of Local Displacement
In this section, we present results of estimation of displacement caused by the varying speed of the laser during bidirectional line acquisition. In our experiments, the displacements were estimated for two microscope systems: Zeiss LSM 780 and Zeiss LSM 880 using the phantom data presented in Section III-A. The result of estimation displacements for three different laser speed setups are visualised in Fig. 6.
The results show that the displacement caused by the laser speed depends on the current laser position (x) in scan line. Therefore, this displacement cannot be corrected by either manual trial-and-error process or automatic estimation of single translation parameter to shift the backward line acquisition to be aligned correctly with the forward line acquisition. The results also suggest that the level of displacement depends on the particular microscope speed setup. This is particularly noticeable for Zeiss LSM 880 (see Fig. 6), where the increase of acquisition speed simultaneously increases displacements (and causes significantly greater jaggedness artefacts). Moreover, the results show that, as anticipated in the Introduction (see Section II-A), the displacement function is not symmetric, which relates to the nature of the jaggedness artefact. Finally, the results also show that our method can be applied to different laser scanning microscopy systems, increasing its application in biology, and enabling results on different microscopy systems to be compared quantitatively.

C. Example of Image Analysis: Corners Detection
In this section, we present results of performing "corner detection" (local second order image variation) which is often a component of complex image analysis tasks, particularly used to identify interesting candidate locations in images, such as branch and end points of vasculature. As a typical corner detector algorithm we chose the Features from Accelerated Segment Test (FAST) algorithm [23]. We perform corner detection on the phantom data using unidirectional line acquisition (un1), bidirectional line acquisition without correction (nb1), with displacement field correction only (ou1), and with our displacement correction and locally weighted filtering (ou1F). The results are shown in Fig. 7. The results show that even the basic operation of corner detection cannot be accomplished on the image obtained using bidirectional line acquisition without performing our method for jaggedness removal. This is not surprising as the corner detection computes second order derivatives, which are severely impacted by the jaggedness artefact. For quantitative evaluation, we compared the number of the detected corners using the unidirectional line acquisition (un1) against the competing methods, and the results are shown in Fig. 8. The results of corners detection on the images restored using our method with displacement field correction only (ou1), are similar to those obtained using the unidirectional line acquisition (un1). Additionally, the results obtained using our method with displacement correction and locally weighted filtering (ou1F) are comparable to those obtained using the two unidirectional line acquisitions with averaging (un2). In summary, our method applied to raw data acquired using the bidirectional line acquisition demonstrates satisfactory performance for the corner detection, especially when compared to the much slower unidirectional line acquisition.

D. Plant Sample
In this section, we present results of restoring an image of a plant sample Convallaria majalis, which is a common slide for microscopy training due to its autofluorescence properties. Fig. 9 shows one example of acquisition and the results of applying our method. Visual comparison between images acquired using unidirectional and bidirectional line acquisition with our method show a good geometrical similarity across the imaging plane, proving the robustness of our method. The quantitative results of image restoration on the plant data are presented in Table III. As shown in Table III, our method (denoted by ou1 and ou1F) gave marked improvements in the image quality criteria as compared to bidirectional line acquisition (nb1).

E. In Vivo Data
Finally, we tested our method in an in vivo cancer imaging application. Fig. 10(a) and Fig. 10(b) each show an example of Fig. 6. Displacement estimated using the presented framework for two microscope systems. The function displacement caused by the variable laser speed is clearly not a symmetric function. Fig. 7. Result of performing a corner detection algorithm on the phantom data. Corners detected on images using (a) unidirectional line acquisition (un1) (red circles), (b) unidirectional (un4) with four time repeated line acquisition, and (c) bidirectional line (green crosses) without correction (nb1), (d) bidirectional with vendor's correction (cb1), (e) bidirectional with displacement field correction only (ou1), and (f) bidirectional with our displacement correction and locally weighted filtering (ou1F). Even a simple analysis of image acquired bidirectional acquisition without correction is challenging, while the results obtained for the image restored using our method are comparable to the reference unidirectional acquisition.    vascular imaging for tumour growth analysis. The results of applying our method to in vivo data are visualised in Fig. 10(c) and Fig. 10(d). Using our method, automatic restoration of acquired images was achieved, based only on information available in the acquired images.
In summary, the results show that our method can reduce artefacts, without any prior information about microscope setup or the need for a prospective study with a phantom to obtain a ground truth. Visual comparison between images acquired using unidirectional and bidirectional line acquisition restored with our method show a good geometrical similarity across imaging plane, demonstrating the robustness of our method.

V. DISCUSSION AND CONCLUSION
We have presented a dedicated image-based method to remove significantly reduce artefacts, in particular jaggedness, from raw images generated by laser scanning microscopy. The framework enables recovery of geometrical distortions caused by the varying speed of the laser, and it can be applied to different laser scanning microscopy protocols, enabling direct comparison of results across platforms. From the perspective of biology, our method enables fast high-resolution acquisition of microscopy data that can monitor complex processes such as the presented example of tumour growth, without sacrificing spatial or temporal resolution, or signal-to-noise ratio. Furthermore, it also could render formerly unusable data open to quantitative and correlative imaging analysis and would enable analysis of data that otherwise would have had to be discarded by biologists.
The presented method is also generalisable because it does not require the microscope specific parameters, or any artefact-free images to build a prior model. Instead, the framework is purely driven by the information estimated from retrospective processing of the already acquired images. We have presented results for 2D image restoration, however restoration of a 3D stack can be straightforwardly achieved by either modification of the cost function (given in Eq. (2)) to include neighbouring horizontal planes, or running the presented framework for each plane independently. While the quality of data obtained from the fast Laser Scanning Microscopy is a trade-off between the acquisition speed, spatial and temporal image resolution, and SNR, our method enjoys a number of obvious advantages. It is capable of restoring images just from fast bidirectional line acquisition, furthermore with a SNR comparable to the SNR in images acquired using slow, unidirectional line acquisition with the line averaging (see results in Table II and Table III). The visual inspection of image restoration outcomes on a phantom, plant sample and real biological tumour data further supports the obtained quantitative results. The presented framework is easy to implement for any experiment using the laser scanning microscopy, and it can also be applied to retrospectively acquired jaggedness-distorted data sets (even if the microscope setup is not known).
The presented method can be further extended in a number of ways. From a preclinical perspective, combining the presented method with method removing motion of specimen during acquisition would be an interesting approach to explore in future research. Such a joint microscopy image restoration method could potentially lead to further quality increase of restored images.

ACKNOWLEDGMENT
The method described in this paper has been the basis for a patent filing no.: GB1807598.6.