3D Surface Reconstruction Method for Microsamples Based on Polarization Modulation Microscopy

A novel microscopic Shape-From-Polarization (SFP) 3D shape reconstruction method is proposed, in which degree of polarization is calculated using Polarized Bidirectional Reflectance Distribution Function (PBRDF) to overcome azimuthal ambiguity for both specular and diffuse polarization microscopic scenes. Based on the calculated degree of polarization, a polarization modulation microscopy oriented variational model is established to estimate surface normal vector and rebuild 3D shape of the sample with “fuzzy guidance” based on atomic force microscopy (AFM). Experiments on different microscopic samples indicate the proposed method can restore the stereoscopic morphology distribution with precision in micrometer.


I. INTRODUCTION
T HE detection and visualization of microstructure have numerous applications in the imaging, spectroscopy, material science, biology, and healthcare fields. 3D surface topography is considered to be one of the most important ways to study microparticle samples because intuitive and multiperspective observations are helpful for studying biological microcosm.
The acquisition of 2D images by microscopic imaging is a prerequisite for achieving 3D reconstruction. There have been quite a few outstanding approaches proposed in the past few years. Fluorescence micrographing [1] can achieve singlemolecule imaging sensitivity, which requires photosensitive or light-convertible fluorometers. Differential interference contrast (DIC) [2] takes advantage of differences in the light refraction by different samples, which make them visible. The equipment used for DIC is quite expensive and specimen carriers, such as culture vessels, made of plastic may not be suitable. To overcome the resolution degradation and increase the universality of the current technology, polarization-based microscopic imaging systems, such as polarization parametric indirect microscopic imaging Manuscript [3] were designed, which allows images to be obtained with subwavelength resolution to yeild multi-polarized sequences by modulating point spread function with varying polarization statuses. Different from 3D microscopy, for stimulated emission depletion (STED) [4] and stochastic optical reconstruction microscopy (STORM) [5] technology, only monocular multipolarized images, are required as input in polarization-based 3D reconstruction method. These methods are based on the idea of establishing a physical relationship between the surface normal of the target object and the polarization modulation curve.
Shape from Polarization (SFP) was proposed by Rahmann [6] for shape recovery of textureless specular surfaces. Atkinson and Hancock exploited surface reflections polarization using images of smooth objects to recover surface normals and, hence, height [7]. However, it was found that rebuilding 3D morphology from only polarization information often involves fuzzy estimation of the azimuth angle. Consequently, Kadambi pointed that polarization information may only provide clues to auxiliary calculations [8]. In order to improve reconstruction accuracy, Zhao peng proposed polarimetric multi-view stereo imaging, which combines per-pixel photometric information from polarization with epipolar constraints from multiple views [9]. Xuan and Fei proposed that the reconstruction results can be modified combined if spectral information is also used [10]. In the field of microscopic imaging, it is inappropriate to use the above improved method without modification. Indeed, observation of microscopic samples involve complex micro-optical phenomena, and they require higher imaging accuracy. To the best of our knowledge, there are still few studies on polarization reconstruction in the field of microscopy.
In this paper, we propose a 3D surface reconstruction method of microscopic samples based on polarization modulation microscopy (PMM). A new degree of polarization (DOP) model based on polarized bidirectional reflectance distribution function (PBRDF) [11] are deduced to fit the complex reflection situations in microscopic images and establish PMM-oriented variational regularization model. Compared with results of traditional SFP method and atomic force microscopy (AFM), the proposed method can effectively keep polarization feature and avoid phase ambiguity and height relativity so restore the three-dimensional morphology distribution of samples.
The rest of this paper is organized as follows. We introduce the proposed method in Section II. The experimental results are presented in Section III. Finally, the conclusions of this study are drawn in Section IV. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

A. Theoretical Basis
Based on polarization modulation theory, the polarization state of the incident light can be modulated to obtain a sequence of polarization images of each sample under a single viewing angle. The modulated image and the angle of the polarizer satisfy the sinusoidal response as: (1) where φ pol is the rotation angle of the polarizer, and I max and I min refer to the maximum and minimum light intensity densely, respectively. The initial phase, ϕ, here is considered to be the result of encoding the azimuth angle with the phase ambiguity of π. The degree of polarization ρ can be described as: Polarization information can reveal the normal vector information of a reconstructed surface by calculating the azimuth angle and the zenith angle. The azimuth angle, ϕ, is encoded as the phase of the received signal. In general, the azimuthal angle is a numerical value corresponding to the first wave peak. Calculation of the zenith angle is relatively complicated for relating to the selected polarization model. There are generally two models used to calculate the zenith angle, specular reflection and diffuse reflection, which are respectively described as: and where n denotes the refractive index and θ the zenith angle. |n * | 2 = n 2 (1 + κ 2 ), and κ is the attenuation index of the material, which allows the zenith angle to be found. Assuming the above parameters are known, the zenith angle can be estimated either in closed-form, or by numerical optimization. Then, the surface normal vector can be expressed as: where p = tan θ cos ϕ, q = tan θ sin ϕ.
It is important to note that there are some SFP methods that aim to obtain surface normals in three dimensions, integration is needed if you want the surface function z = f (x, y).

B. Improved SFP Based on PBRDF Function
Considering that the calculation of the degree of polarization of microscopic samples does not follow a simple reflection model, we describe it with the concept of bidirectional reflection  distribution function, which defines how light is reflected at an opaque surface.
As shown in Fig. 1, the incident light dω i , irradiates the sample after passing through a polarizer with a determined polarization direction. The incident angle between dω i and the z axis is described as θ i ; the microfacet is marked as dA. Ideally, the angle of emergence, θ r , is determined by the reflected light dω r , which is numerically equal to the angle between dω r and the z axis. Due to the input and output light not being in the same plane, we record the angle between their projection line in the xoy plane and the x-axis as φ i and φ r . We assume that the output is composed of polarized specular reflection and non-polarized volume scattering, and the primary specular reflection affects the characteristics of reflected radiation. According to Fresnel's law of reflection, the degree of polarization of reflected radiation can be described by the following formula: where The DOP simulated by PBRDF is only related to the reflection observation direction when incident angle is fixed. Inspired by statistically modeling in Microfacet-based BRDF works [12], we propose that, the reflection observation angle θ r is perturbed centering specular reflection angle θ i in incidence plane, resulting from perturbation of the normal [13] of the microfacet subsequently affecting outgoing direction. Synthesizing the perturbation effects scalarly, reflection observation angle can be described by a statistical concept: where The greater the deviation angle means the smaller the energy component. The perturbed light intensity factor at angle (θ + Δθ) can be approximated by a Gaussian distribution with mean zero, that is p( . σ equals to π 6 as a consequence of conservation of energy (PauTa or 3σ criterion). The reflection observation angle approach to ideal value with the increase of m. Fig. 2 shows simulated error within one degree can be ensured when m is greater than 32. Accordingly, substitute θ r into the (8) to obtain the PRDF-based DOP model: Fig. 3 shows three curves, which represent how DOP value changes with angle under diffuse reflection, specular reflection and our proposed PBRDF situation. Generally speaking, traditional reflection models may not always yield a proper solution due to distinct extinction effect of a fixed polarizer on various samples or fitting error of trigonometric function. In almost all the samples we tested, the images of microscopic samples range marginally under the rotating polarizer which also confirms the superiority of the model from the side. Eq. (5) describes the process of obtaining the surface normal vector from the zenith angle and the azimuth angle. An important problem faced by SFP is that even if the normal is restored correctly, the depth obtained by integration is often discontinuous. The Frankot&Chellappa (FC) algorithm [14] can always realize enforcing integrability, which is described by: The enhanced integral can only ensure the continuity of the surface shape, while reconstruction accuracy is still obsessed by by relative depth and surface distortion. In order to avoid above problem, we construct a variational optimization function in following section C.

C. PMM-Oriented Variational Model
Considering that a registered depth map, z 0 , plays a guiding role in some 3D sensing, we may thus take the following prior term: where Ω 0 ⊂ R 2 is the target region for which prior information is available.
C. Dummy Head3: a) Reference depth z 0 : The reference image records the corresponding 3D information, which must be provided by a more accurate measurement method. In the atomic force microscopy [15] under the tapping mode, the probe maintains a fixed frequency vibration on the z-axis, and contacts the sample when the vibration reaches the bottom, which is suitable for observing the "fragile" sample without damaging its surface.However, directly using the registered AFM as the reference depth will lead to over fitting. In microscopic imaging field, AFM is not the only way to obtain three-dimensional information of samples. The z-axis low resolution and the troublesome registration are common problems when measuring the 3D information of the sample, even with the laser confocal microscope. To demonstrate the feasibility of our approach, we choose the following three methods to generate "fuzzy guidance": 1) noisy input, 2) quarter quadrant reserved input and 3) quarter area reserved input. Noisy input means adding Gaussian noise to the depth map acquired by AFM. Quarter quadrant retention refers to the retention of the groundtruth on one quadrant, and the other parts are replaced by the normalized results of shape from shading; as shown in Fig. 4, the difference between c and b is that the regional position of the groundtruth is random. These generation methods simulate low resolution   reference input, partial area correction and dislocation respectively. The results in Figs. 8, 9 and Table I below are based on scheme b, and other schemes are specifically discussed at the end. b) Regularization constraint: Inspired by the shape from shading problem, the relationship between the reconstructed 3D surface and image irradiance can be formulated as a first-order nonlinear partial differential equation [16]. The light source can be considered normally incident in polarization modulation microsystems [17], so the projection constraint can be expressed as: where ∇u(x) = [p(x), q(x)] T , and the incident parallel lighting whose direction is characterized by ω = [ω 1 , ω 2 , ω 3 ]. When we process the image sequence obtained by the PMM system, the partial differential equation is simplified to the Eikonal equation [18]: The regularization term that characterizes the projection constraint can be designed as: where I refers to the unpolarized image by averaging the four polarized images to achieve depolarization and z refers to the reconstructed surface function. Since we are reconstructing nano samples, the height of the surface corresponding to the highlighted area needs to be controlled by an additional constraint.
Here, we chose the surface function S ∇z [19], which is defined as: The regularization term that characterizes the surface area constraint can be designed as:  where f refers to focal length. If we assume measurement errors in microscopic imaging model can be added in quadrature, and represented as Gaussian noise, then the maximum likelihood solution is attained by estimating the depth map, z, which minimizes the following function. The augmented Lagrangian functiona is defined as follows: Algorithm 1: Split-Bregman Algorithm Based on PMM. Input: Surface function z, regularized parameters λ, ξ, η Output: The k-th round result of z k a.Initialization : The Split Bregman method [20] is an iterative algorithm that can be used to solve certain convex optimization problems involving regularization.

D. Proposed PMM-Based 3D Surface Recontruction Method
A graphical abstraction of proposed method is shown on Fig. 5. Foremost, polarizing image sequences under different polarization angles of prepared microsamples are acquired by PMM system. Based on the theory of polarization modulation, the initial phase of the trigonometric function encodes the azimuth information of the 3D surface normal to the corresponding position and zenith angle can be calculated densely by imporved PBRDF model. At the same time, 3D morphology of the corresponding areas are measured by AFM. Subsequently, "fuzzy guidance" can be generated by adding noise or randomly reserving a part of AFM image, tring to represent the awful or blurry reference depth. Finally, the 3D reconstructed surface is iteratively solved by Split Bergman method by optimizing a variational optimization model with the help of improved SFP and "fuzzy guidance," which solve the problem of phase ambiguity and relative depth.

A. Experimental Setup
Imaging system: An Olympus BX53 M was selected as the PMM system. The objective lens was an mplfln100xbd with a 0.9 numerical aperture, and the eyepiece was a whn10x. Using a mercury lamp (about 532 nm) as the light source, linearly polarized light with an equidistant polarization angle of 45 degrees was obtained by rotating the polarizer. As shown in Fig. 6, Bruker dimension XR fastScan, which can realize high-speed scanning and imaging of nanosamples, was chosen as AFM system to provide 3D morphology distribution. The size of the imaging  area was selected in the 10 μm range, and the scanning speed is about 1 Hz. The imaging magnification of the PMM system was about 1000, and the obtained image size was 1922 × 2560; the actual size of the area obtained by AFM is 10μm ×10μ m, and the image size was 512 × 512.
Experimental materials: In consideration of the resolution of the optical microscope, materials of about 1 micron are selected, such as polystyrene microspheres and MIL-53 (Ni), as the representative of periodic image and non-periodic reconstruction, respectively.

B. Results and Discussions
Data alignment: In our work, data alignment is done manually. The first step is to roughly observe the sample with an optical microscope, select a representative area, photograph the area, and roughly mark the position on the glass slide fixed with the mica sheet. The second step is to find the corresponding area under AFM according to the marks in the first step. The third step is data alignment. Sometimes the polarization image needs to be rotated to fit the AFM result and the resolution of the same region are inconsistent due to different acquisition principles. These tasks can be completed through simple data processing such as image rotation, downsampling, etc. As shown in Fig. 7, the polarization characteristic images in jpg format are acquired by PMM system while the original AFM data need to be saved in spm format, which can be read out by python. To distinguish 2D polarized images from depth images, we choose to display the latter group in Parula-style pseudo color without any other modifications on polarization images.
3D topography analysis: Fig. 8 shows the reconstructed results from the SFP, the proposed method and AFM. SFP method can capture more polarization characteristics than other methods, but it can not overcome the problems of surface distortion and relative numerical value. Good reconstruction results can still be achieved even if with only a quarter real region. A reasonable explanation is that the multi-polarization information can be used to speculate the 3D shape by itself, and the fuzzy guidance only serves as a height corrector. In order to facilitate the calculation of the evaluation index, the values are all normalized here.
To further illustrate the reconstruction details, we choose 2D profile curves to illustrate reconstructed results as shown in Fig. 9. It is not difficult to find that the results of SFP are misaligned and the values are relative. From the perspective of surface distribution, our result can be more fit the genuine height, credit to the improved DOP model and fuzzy guidance. The shape predictions in some blurred and incomplete regions conform to our assumptions, which contributes from the excellent design of regularization terms.
Reconstruction quality evaluation: In order to verify the effectiveness of the algorithm, we choose the mean absolute error (MAE), structural similarity (SSIM) and earth mover's distances (EMD) as evaluation indicators. Considering that it is hard to provide more information such as spectral multi view geometry, only linear SFP was elected as a competitor. MAE represents the difference between the reconstruction result and the groundtruth, SSIM reflects the similarity between them, and EMD takes into account the amount of calculation required to change from the reconstruction result to the groundtruth. Our method has achieved state of art in these three evaluation indexes. Only the unique MAE of MIL-53 is slightly lower than that of linear SFP.
Discussion on "fuzzy guidance": It is unreasonable to directly apply the manual registered data obtained by AFM to the variational model, which will lead to over fitting. To illustrate effectiveness of proposed method when the reference information is noisy or incomplete, we have tested on polystyrene sphere sample under three types of fuzzy guidance. As is shown in Fig. 10, it is pleasantly surprised to find that there are only slight differences in the details of reconstruction and overall shape and evaluation parameters have not changed greatly. Quadrant reservation has better reconstruction effect than random block reservation because of the continuity of the region. We draw a similar conclusion in the test results of non-periodic samples, silver nanowires.
The discrepancy between priori and results urges for more physical and geometric knowledge to simulate of optical phenomena on micro level surfaces. In addition, this paper just starts to explore "optimal 3D reference" and there still remain lots of to define. More types of microscopic samples will be observed to unravel undetected optical principles, and polarization may be a key technology to unlock the 3D micro world.

IV. CONCLUSION
This paper proposes a 3D surface reconstruction method for mircosamples utilizing the polarization parameters in microscopic system. Firstly, the PBRDF model can simulate the degree of polarization model of mircosamples so correct the zenith angle error. Furthermore, we proposed a 3D variational reconstruction model for high quality 3D reconstruction, which includes a image irradiance model term and a projection constraint. To solve the optimization problem in the proposed model, we propose an efficient solution based on Split Bregman. We conduct experiments on different kinds of microsamples to verify the efficiency and robustness of the proposed method. Compared with the registered depth map by AFM, the experimental results demonstrate that the proposed method can generate accurate 3D reconstruction results with polarization characteristic information indicating its broad application prospect in optical microscopy. In order to improve our research, we will focus on the reconstruction of mircosmaples at different scales and on the development of polarization-based super-resolution theory in the future. Fluorescence polarization modulation related super-resolution imaging will be our future research direction.