Practical Calibration Method for Aerial Mapping Camera Based on Multiple Pinhole Collimator

Calibration of aerial mapping camera has an important influence on the applications of earth observation. However, the traditional aerial mapping cameras calibrations depend on large-scale calibration target or collimated light, moreover, it is difficult to build the large-scale calibration targets and the collimated light method requires an accurate turntable and a high-precision goniometer, which is a kind of expensive instrument. To solve this problem, this paper proposes a novel high-precision calibration method which is not restricted by the rigorous conditions, such as small aperture, unevenly energy distribution and expensive equipment. Specifically, a collimator and an elaborately designed multiple pinhole mask are firstly used to generate the collimated light of a large aperture with known directions to simulate the calibration targets at infinity. Then, the camera takes pictures for the aperture of the multiple pinhole collimator at multiple angles to ensure that the image points cover the entire detector. Thirdly, the final calibrated results are obtained by solving the data acquired from multiple angles. Finally, the proposed method is verified by Monte-Carlo simulation and real experimental data, whose results indicate that our method can reach the same accuracy performance as the existing methods at lower cost and faster speed, and thus is practical for engineering application.


I. INTRODUCTION
The aerial mapping camera is usually mounted on the aircraft for imaging long-distance ground, and can achieve the advantages of high precision, fine resolution and long focal length. Therefore, it is widely used in earth obser-vation applications, such as aerial mapping, photogram-metry, resource survey and military, etc. In practice, the mapping camera must be calibrated before usage and the calibration accuracy has significant influence on the ortho-rectification, digital elevation model (DEM) generation [1]- [3], surface change detection and measurement accuracy [4].
Camera calibration is one of the most active research areas [5]- [7]. In the past decades, lots of methods have been proposed to calibrate the camera. The classical laboratory calibration method can be divided into two types: the calib-ration grid method [8]- [10] and the collimated light method [11], [12]. In the scheme of calibration grid method, The associate editor coordinating the review of this manuscript and approving it for publication was Wei Liu. the camera images the precisely measured calibration grids at different angles and the calibration parameters are obtained by solving the nonlinear equations according to the coordinates of the calibration grids and the image points. This scheme does not need to measure the angle of the camera relative to the calibration grids, and the data processing is high precision and easy to be implemented. However, when calibrating a long focal length camera, the calibration grids must have a large-scale due to the long imaging distance which does not cause defocusing. In the scheme of the collimated light method, a collimator is employed to generate a single beam of collimated light and adjusts the angle of the parallel light relative to the camera through a precision turntable, in which the angle is measured by a high-precision goniometer. The calibration parameters are calculated according to the angle value of the parallel light and its image coordinate. Although this method is suitable for long focal length cameras, the experiment condition is very critical and expensive.
Recently, there is an urgent demand to invent a novel calibration method to combine the advantages of these two classic calibration methods. References [13], [14] proposed an approach based on lasers and diffractive optical elements, in which a single beam of parallel laser light is divided into a plurality of parallel beams through a diffraction element, and then will produce multiple calibration targets at infinity. Although this method is suitable for both long and short focal length cameras, there are safety issues in the usage of laser and the technique is limited by the fabrication process of the diffraction element. In addition, the aperture of the colli-mated beam in ref. [14] is less than 5 cm, which will cause vignetting (pupil mismatch) while calibrating the large aper-ture aerial mapping camera. Therefore, on one hand this method is only suitable for small aperture camera calibration, on the other hand this method only can make calibration based on a single photo since the camera can't image the parallel light at multiple angles, which finally influence the calibration accuracy. Moreover, collimated light with different propagation directions has different energy distributions, resulting in different brightness of pixels at different positions of the calibration photo and introduces difficulties in extracting image coordinates.
The existing problems can be overcome by using the equipment of multiple pinhole collimator. Multiple pinhole collimator is a collimator with a multiple pinhole mask on its focal plane, thus it has the capability of generating collimated multi-beams with even energy distribution. Wei Zhang et al. [15] used a two-pinhole collimator to generate two X-ray collimated beams and update X-ray photon utilization efficiency. Benedikt Menz et al. [16] employed a multiple pinhole collimator with much more holes to generate multi-beams parallel light with large aperture and known directions. These applications [15], [16] show that multiple pinhole collimator is very suitable for camera calibration.
The aim of our research of to provide a compact and accu-rate calibration setup to satisfy the requirements of long focal length and large aperture camera calibration. In this paper, we employ a collimator and an elaborately designed multiple pinhole mask to generate the collimated light of a large aperture with known directions to simulate the calibration targets at infinity, moreover, the energy evenly distributed to multiple collimated beams which is equivalent to have calibration grids at infinity. Thanks to the large aperture of the collimator, the camera can image parallel light at multiple angles and ensure the image points cover the entire detector, which guarantee the calibration accuracy. The method combines the advantages of two classic methods and works well without dependence on large-scale calibration target or precise measuring equipment, such as delicate turntable and high-precision goniometers. The main calibration equipment used on this paper is collimator, which is a common optical equipment and has been used in many optical systems, therefore, the calibration procedure is efficient and low-cost, and thus is practical for engineering application.

II. CAMERA CALIBRATION WITH MULTIPLE PINHOLE COLLIMATOR
A. SCHEME OF THE MULTIPLE PINHOLE COLLIMATOR According to geometric optics, when an object at infinity passes through the optical lens, it will image itself on the focal plane of the lens. Conversely, placing a calibration object on the focal plane of the lens will produce parallel light representing the calibration target at infinity. According to the position of the calibration object on the focal plane of the lens and the focal length of the lens, the propagation angle of the parallel light can be accurately calculated [17].
A multiple pinhole mask is mounted on the focal plane of the collimator, and the light passing through the holes passes through the optical system of the collimator, which becomes a multi-beams parallel light. According to the position coordinates of the hole and the focal length of the collimator, the propagation angle of each parallel light can be accurately obtained [18]. The schematics of a multiple pinhole collimator are shown in Fig. 1. A spherical coordinate system O − XYZ and a Cartesian coordinate system O − xyz are constructed. Coordinate system origin O is the center of the multiple pinhole mask (z) is the optical axis of the collimator; X(x) is horizontal axis; Y (y) is vertical axis.
The propagation direction ϕ i and θ i of the collimated light P i in the spherical coordinate system O − XYZ are given by Eq. (1) where f is the focal length of the collimator. Convert the propagation direction of collimated light P i from the spherical coordinate system O − XYZ to the Cartesian coordinate system O − xyz as shown in Fig. 2.
39726 VOLUME 8, 2020   shows the schematics of calibration with multiple pinhole collimators. The multiple pinhole mask is placed on the focal plane of the collimator which generates multiple calibration targets at infinity. The aperture of the collimated light can up to ϕ =600mm, which is 12 times larger than the existing method using diffractive optical elements. Multiple pinhole mask has 16 pinholes with diameter of 5 mm. The pinholes are fabricated by photolithography, the technology is mature and cheap (about $150 for manufacturing) and the position error is less than 0.0002 mm.
The propagation directions of 16 collimated beams can be calculated by Eq. (2). Camera takes photos of the aperture of the multiple pinhole collimator at different angle and ensures images cover the whole detector of the mapping camera.
Considering that a light ray travels along a straight line [19], Object (P i ), image (m i ) and camera projection center (O c ) are linearly aligned as shown in Fig. 4(a). Incident beam P i and vector O c m i are collinear. According to the collinear equation [20], the relationship between P i and m i as Eq. (3): where, s is photography scale factor, H is projection matrix, which is as follow: where, C is inner orientation parameters matrix.
where, x 0 , y 0 are coordinates of the principal point and f is the principal distance of camera. In Eq. (4), R is matrix which represents the rotation of where, a i , b i , c i (i = 1, 2, 3) are direction cosines [20] of ϕ, ω, κ in Fig. 4(b), r 1 , r 2 , r 3 are column vectors of R. Substitute Eq. (5) and Eq. (6) into Eq. (3), the imaging model of the aerial mapping camera using multiple pinhole collimator is: Eq. (7) is the derivation of the relationship between object and image in calibration. Incident beam P i and its image m i is related by a 3×3 projection matrix H, which is the product of C and R.

) SOLUTION OF INNER ORIENTATION PARAMETER
The calibration model for camera is established based on R. Hartley et al. [6] as Eq. (9). Given an image of multiple collimated beams, a projection matrix H can be estimated (see Appendix A).
Defining coefficient matrix A as: As shown in Eq. (10), coefficient matrix A contains incident light P i and its image coordinate x i , y i . Since each incident light and its image have 2 equations, so 16 collimated beams bring about 32×9 coefficient matrix A. Solving Eq. (9), we have projection matrix H. Camera takes picture of the multiple pinhole collimator at n different angles, and gets n projection matrix H 1 , H 2 . . . H n . Setting Here G is a symmetric matrix, defined by a 6D vector g: where, G 11 , G 12 , G 22 , G 13 , G 23 , G 33 are elements in matrix G. (8), due to orthogonality of the rotation matrix R, we can get (see Appendix B): Setting: There is:  Matrix V is defined as: Eq. (15) is simplified as V · g T = 0, matrix V is composed by the elements of projection matrix H. Taking n pictures, there are H 1 , H 2 . . . H n . Substituting Eq. (16), we have 6n × 6 matrix V . Based on least squares, the best solution g is solved. The inner orientation parameters are solved by using Eq. (17): Matrix V in Eq. (15) is 6n × 6. According to Eq. (15), our method requires at least one photo to solve the inner orientation parameters, while adopting classical method, matrix V is 2n × 6. Ref. [9] requires at least three photos to solve the same problem.

3) CALCULATION OF DISTORTION COEFFICIENTS
Camera distortion can be divided into radial distortion, eccentric distortion, etc. Long-focus aerial mapping cameras generally require precise optical adjustment [22]. The eccentricity of the optical lens is small, and the main distortion type is radial distortion. Therefore, this paper only considers radial distortion. Considering too many kinds of distortions will lead to computational instability [23].
In ideal imaging, pinhole T has image t and its coordinate is (x, y). Because of radial distortion, the actual image coordinate is (x d , y d ) where k 1 , k 2 are radial distortion coefficients, and r 2 = x 2 d + y 2 d . The origin of the radial distortion is the origin of the camera.
Simplified to D.k = d. Considering there are 16 calibration points in each photo and calibration at n angles, Eq. (19) has 16n equations which can be solved by least squares method: 39728 VOLUME 8, 2020 Once k 1 and k 2 are estimated, we can refine the estimation of the projection matrix H by solving (9) with coefficient matrix A replaced by (19). Matrix V is then given according to the equation (16). Lastly, all the intrinsic elements can be computed by calculating the Eq. (15) and Eq. (17). We can alternate these two procedures until the convergence is achieved. Through this iterative method, the camera can be accurately calibrated even if there is a large distortion.

C. FLOWCHART OF THE CALIBRATION METHOD
The flowchart of the proposed method is shown in Figure 5. the input is the information about the multiple pinhole collimator and the output is the intrinsic parameters and the radial distortion coefficients.
There are three steps in this method. Firstly, ''preparation'', an elaborately designed multiple pinhole mask should be fabricated and mounted. The camera should be fixed facing the collimator. Secondly, ''data acquisition'', the camera should take photos for the aperture of the collimator at multiple angles and ensure that the image points cover the entire detector. Thirdly, ''estimation'', we estimate projection matrix H with coefficient matrix A, which is calculated by the parameters of multiple pinhole collimator and image coordinates. Once H is estimated, matrix V can be given by the elements of H. Finally, the inner orientation parameters, as well as radial distortion coefficients are solved. The camera can be accurately calibrated through the proposed iterative method.

III. EXPERIMENTS
The accuracy and reliability of the proposed method were verified by both computer simulation and real data. Detailed information about the aerial mapping camera and the multiple pinhole collimator is given in Table 1.
Firstly, we simulated the calibration accuracy by Monte Carlo algorithm. Then, the aerial mapping camera was calibrated by using the method of this paper. Moreover, the proposed method and classic Collimated light were compared.

A. COMPUTER SIMULATIONS
The simulation test device is shown in Fig. 5, whose camera was mounted on the turntable. In the simulation, the camera imaged the aperture of the collimator at 16 angles, which ensured that the image points of the parallel light cover the entire detector. In practice, calibration data from more widely distributed view angles is more helpful in reducing the estimation errors. Empirically, we found calibration data from more than 16 widely distributed view angles can yield satisfying results. Different from the conventional collimated light method, our method requires neither precise adjustment nor angle measurement of the camera relative to the parallel light.
The parameters of the simulation are summarized in Table 1. Two simulated experiments are performed. Section 1) examines the effect of measurement noise of the hole position on the estimated intrinsic parameters. Section 2) models the effect of projected image points on the estimated intrinsic parameters.

1) MEASUREMENT NOISE OF THE HOLE POSITION ON INTERNAL ORIENTATION ELEMENTS
The effect of measurement noise of the hole position on calibration result is tested by adding increasing levels of Gaussian noise, ranging from 0.001mm to 0.005mm. For each noise level, we perform 200 independent trials. The statistics of the error in the intrinsic parameter estimates are shown in Fig. 6.
As we can see from Fig. 6, errors increase with the noise level. For σ = 0.002mm, the errors in principal point is around 0.1 pixel and principal distance is around 0.19 pixel.

2) NOISE OF THE PROJECTED IMAGE POINTS ON INTERNAL ORIENTATION ELEMENTS
Gaussian noise with variance of 1/20 pixels to 1/5 pixels was added to projected image points and the results are shown VOLUME 8, 2020   For σ = 1/10 pixels (which is larger than the normal noise in practical calibration), the error in principal point is around 0.29 pixels and principal distance is around 0.5 pixels.

B. REAL DATA
The aerial mapping camera was calibrated using the method of this paper. Calibration facilities are shown in Fig. 8. The specifications of the collimator and camera are listed in Table 1. The camera was mounted on a turntable to adjust the angle of the camera with respect to the multiple pinhole collimator. In total, 16 angles were used to take photos and 256 image points were obtained. Fig. 9 shows the distribution of the image points.
Calibration results were calculated by combining image points acquired at all angles. The standard deviation of the residuals between model and measurement points on the entire measurement points is about 0.12 pixels (1.33µm) and the maximum residuals is less than 0.3 pixel (2.22µm). The results were compared with the calibration results taken by the standard collimated light method. We try our best to conduct two experiments with the same experimental conditions, such as the aperture of the collimated light, focal length of the collimator, the parameters of the aerial mapping camera,   temperature, pressure, humidity et al.. The only difference is that the standard collimated light method employs highprecision equipment, and we can work well without dependence on it. All the corresponding results are listed in Table 2, where n is the number of photos.
According to Table 2, the calibration results agree well with the collimated light method using a high precision and expensive calibration device, but our method uses fewer calibration photos, which means saving lots of time. In addition, it was shown that our method can separate the parameters of the interior orientation from the exterior orientation of the camera. Hence, a complex alignment of the calibration setup components is not necessary, which significantly simplifies the calibration process. On the contrary, the total calibration time will increase about 33% because of complex bundle adjustments in conventional collimated light method.
The time and money spent by the two methods in this experiment are shown in Table 3.
As we can see from Table 3, our method can achieve the same accuracy as that of the traditional collimated light method, but is cheap and fast, which meets the requirements of engineering application very well.
Our method is similar to the method proposed in [13], [14] but differs in two aspects: (1) The method proposed in [13], [14] generates the multi-beam parallel light by the diffractive element, which requires a laser, a collimator and a diffractive element etc. The generated collimated light has a large angle of view, but a small aperture, and an uneven energy distribution. In our method, the collimated light is generated by placing a porous plate on the focal plane of the collimator, and the aperture of the generated parallel light is limited only by the collimator, and can generate parallel light of 12 times of the aperture of [14] and the energy distribution is uniform. (2) Method proposed in [13], [14] is only suitable for cameras with small aperture and the camera can't image the parallel light at multiple angles because of the small aperture. Therefore, method proposed in [13], [14] can only do calibration based on a single photo. In our method, the camera can image parallel light at multiple angles and ensure the image points cover the entire detector, which provides better calibration accuracy.

IV. CONCLUSION
The low-cost and efficient calibration of aerial mapping camera is the key challenge encountered in earth observation applications. Traditional calibration methods require largescale calibration target, high-precision and expensive equipment. This paper proposed a new method for calibrating the aerial mapping camera only based on a multiple pinhole collimator without any dependence on complex and expensive instruments. Firstly, a collimator and an elaborately designed multiple pinhole mask are firstly used to generate the collimated light of a large aperture with known directions to simulate the calibration targets at infinity. Then, the camera takes photos for the aperture of the multiple pinhole collimator at multiple angles to ensure that the image points cover the entire detector. Thirdly, the final calibrated results are obtained by solving the data acquired from multiple angles. In general, highlights from two aspects can be drawn as follows: 1) On one hand, the proposed method does not require large-scale calibration target, high precision turntable or goniometer, etc. On the other hand, the calibration process is simplified and has fewer requirements on precision adjustments. Moreover, the proposed method can achieve the same accuracy as that of the traditional collimated light method which depends on the high precision calibration device. 2) In this method, the aperture of collimated light is 600 mm, which is 12 times larger than the existing method using diffractive optical elements and the energy distribution is more uniform. Such aperture is suitable to calibrate camera with long focal length and large aperture. After verifying by Monte-Carlo simulation and real experimental data we know that the proposed method can reach high accuracy calibration result without use of any largescale calibration target or expensive equipment. This method is not only suitable for large-aperture long-focus aerial mapping cameras, but also for other cameras with calibration requirements. The analysis results show that the proposed method has the advantages of efficient and low-cost, and thus is practical for aerial mapping camera applications.

sm = HP
Therefore, the 3-vectors m and HP have the same direction and differ in magnitude by a scale factor s. Due to vector cross product, the equation can be expressed as: The cross product gives a set of three equations: Since h jT .P= P T .h j (j =1, 2, 3), this gives explicitly equations as: Notably, although there are three equations, only two of them are independent (since the third row is obtained from VOLUME 8, 2020 the sum of x times the first row and y times the second), like: H is the product of C and R, which as follows: where h 1 , h 2 , h 3 are row vectors of H,h k = [h k1 h k2 h k3 ](k = 1, 2, 3) and r 1 , r 2 , r 3 are column vectors of R, due to orthogonality of the rotation matrix R, we can get: r T k r j = 0 r T k r k = r T j r j Then, we can get: Here G is a symmetric matrix, defined by a 6D vector g. Using vector g = G 11 G 12 G 22 G 13 G 23 G 33 T , we can get: Based on least squares, the best solution G is solved. ZHUANG ZHANG received the B.S. degree in applied physics from Hehai University, in 2013. He is currently an Engineer with the Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences. His current research interests include image processing and mapping techniques for the aerial remote camera. VOLUME 8, 2020