In this paper for the first time, we study the observability of the nonlinear system describing the IMU-camera calibration process. Specifically, after a brief review of the Lie derivatives and the observability rank condition (Section IV-A), in Section IV-B, we prove that the IMU-camera calibration system is *locally weakly observable* when at least two rotations about different axes are performed.

### A. Nonlinear Observability

Consider the state-space representation of the following infinitely smooth nonlinear system:
TeX Source
$$\cases{ \dot{{\bf x}} = {\bf f}({\bf x}, {\bf u}) \cr {\bf y} = {\bf h}({\bf x}) }\eqno{\hbox{(27)}}$$where is the state vector, is the vector of control inputs, and is the measurement vector, with *y*_{k} = *h*_{k}(**x**), *k* = 1, …, *m.*

If the process function, **f**, is input-linear [38], it can be separated into a summation of independent functions, each one corresponding to a different component of the control input vector. In this case, (27) can be written as
TeX Source
$$\cases{ \dot{{\bf x}} = {\bf f}_0({\bf x}) + {\bf f}_1({\bf x}) u_1 + \cdots + {\bf f}_l({\bf x}) u_l\cr {\bf y} = {\bf h}({\bf x}) }\eqno{\hbox{(28)}}$$where **f**_{0} is the zero-input function of the process model.

The zeroth-order Lie derivative of any (scalar) function is the function itself, i.e., . The first-order Lie derivative of function *h*_{k}(**x**) with respect to **f**_{i} is defined as
TeX Source
$$\eqalignno{{{\L}}_{{\bf f}_i}^{1} h_k({\bf x}) &= { \partial h_k({\bf x})\over \partial x_1} f_{i1}({\bf x}) + \cdots + { \partial h_k({\bf x})\over \partial x_n} f_{in}({\bf x}) \cr&= \nabla h_k({\bf x}) \cdot {\bf f}_i({\bf x})&\hbox{(29)}}$$where **f**_{i}(**x**) = [*f*_{i1}(**x**),…, *f*_{in}(**x**)]^{T}, ∇ represents the gradient operator, and '·' denotes the vector inner product. Considering that is a scalar function itself, the second-order Lie derivative of *h*_{k}(**x**) with respect to **f**_{i} is
TeX Source
$${{\L}}_{{\bf f}_i}^{2} h_k({\bf x}) = {{\L}}_{{\bf f}_i}^{1} \left({{\L}}_{{\bf f}_i}^{1} h_k({\bf x})\right) = \nabla {{\L}}_{{\bf f}_i}^{1} h_k({\bf x}) \cdot {\bf f}_i({\bf x}).\eqno{\hbox{(30)}}$$Higher order Lie derivatives are computed similarly. Additionally, it is possible to define mixed Lie derivatives, i.e., with respect to different functions of the process model. For example, the second-order Lie derivative of *h*_{k} with respect to **f**_{j} and **f**_{i}, given its first derivative with respect to **f**_{i}, is
TeX Source
$${{\L}}_{{\bf f}_j {\bf f}_i}^{2} h_k({\bf x}) = {{\L}}_{{\bf f}_j}^{1} \left({{\L}}_{{\bf f}_i}^{1} h_k({\bf x})\right) = \nabla {{\L}}_{{\bf f}_i}^{1} h_k({\bf x}) \cdot {\bf f}_j({\bf x}).\eqno{\hbox{(31)}}$$

Based on the preceding expressions for the Lie derivatives, the observability matrix is defined as the matrix with rows:
TeX Source
$${\cal O} {\buildrel \triangle \over =} \{{\nabla {{\L}}_{{\bf f}_i\cdots {\bf f}_j}^{\ell } h_k({\bf x}) \vert i, j=0,\ldots, l; k = 1,\ldots, m; \ell \in \hbox{IN} }\} .\eqno{\hbox{(32)}}$$

The important role of this matrix in the observability analysis of a nonlinear system is captured by the following proposition [17].

*Proposition 1 (Observability rank condition):* If the observability matrix [cf. (32)] of the nonlinear system defined in (27) is full rank, then the system is locally weakly observable.

*Remark 1:* Since the process and measurement functions [cf. (27)] are infinitely smooth, the observability matrix can have infinite number of rows. However, to prove that is full rank, it suffices to show that a subset of its rows are linearly independent.

In general, there exists no systematic method for selecting the suitable Lie derivatives and corresponding rows of when examining the observability of a system. Instead, this selection is performed by sequentially considering the directions of the state space along which the gradient of each of the candidate Lie derivatives provides information.

### B. Observability of IMU-Camera Calibration

In this section, we compute the observability matrix of the system describing the IMU-camera calibration process and prove that it is full rank.

First and in order to simplify the notation, we retain only few of the subscripts describing the variables in the system state vector [cf. (2)]:
TeX Source
$${{\bf x}(t)} = \left[\matrix{ \bar{q}_{ {I}}^{{ {T}}}\quad {\bf b}_g^{{ {T}}}\quad {\bf v}^{{ {T}}}\quad {\bf b}_a^{{ {T}}}\quad {\bf p}_{ {I}}^{{ {T}}}\quad \bar{q}_{ {C}}^{{ {T}}}\quad {\bf p}_{ {C}}^{{ {T}}} }\right]^{{ {T}}}.\eqno{\hbox{(33)}}$$

Then, we rearrange the nonlinear kinematic equations (3)– (6) in a suitable format for computing the Lie derivatives:
TeX Source
$$\left[\!\matrix{\dot{\bar{q}}_{ {I}}\cr \dot{{\bf b}}_g\cr \dot{{\bf v}}\cr \dot{{\bf b}}_a\cr \dot{{\bf p}}_{ {I}}\cr \dot{\bar{q}}_{ {C}}\cr \dot{{\bf p}}_{ {C}}}\!\right] \!=\! \underbrace{ \left[\matrix{ -{ 1\over 2} {\bm \Xi} (\bar{q}_{ {I}}) {\bf b}_g \cr {\bf 0}_{3\times 1} \cr {\bf g}\! -\! {{\bf C}}^{ {T}}(\bar{q}_{ {I}}){\bf b}_a \cr {\bf 0}_{3\times 1} \cr {\bf v}\cr {\bf 0}_{3\times 1} \cr {\bf 0}_{3\times 1} }\right]}_{{\bf f}_0}\! +\! \underbrace{\left[\matrix{ { 1\over 2} {\bm \Xi} (\bar{q}_{ {I}}) \cr {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} }\right]}_{\underline{{\bf f}}_1} {{\bm \omega} _m }\! +\! \underbrace{\left[\matrix{ {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} \cr {{\bf C}}^{ {T}}(\bar{q}_{ {I}})\cr {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} \cr {\bf 0}_{3\times 3} }\right]}_{\underline{{\bf f}}_2}{\bf a}_m\eqno{\hbox{(34)}}$$where *ω*_{m} and **a**_{m} are considered the control inputs, and
TeX Source
$${\bm \Xi} (\bar{q}) = \left[\matrix{ q_4 {\bf I}_{3\times 3} + \lfloor {\bf q}\, \times \rfloor \cr -{\bf q}^{ {T}}}\right] \hbox{with}\ \bar{q} =\left[\matrix{ {\bf q} \cr q_4 }\right].\eqno{\hbox{(35)}}$$Also, note that **f**_{0} is a 23 × 1 vector, while __f___{1} and __f___{2} are both compact representations of three vectors of dimension 23 × 1, i.e.,
TeX Source
$$\underline{{\bf f}}_1 {{\bm \omega} _m } = {\bf f}_{11} \omega _{m1} + {\bf f}_{12} \omega _{m2} + {\bf f}_{13} \omega _{m3}$$where, for *i* = 1, …, 3, **f**_{1i} denotes the *i* th column vector comprising __f___{1} and ω_{mi} is the *i* th scalar component of the rotational velocity vector.

A well-known result that we will use in the observability analysis of (34) is the following: When four or more ^{1} known features are detected in each calibration image processed by the filter, the *camera pose* is observable [39] and can be computed in closed-form [4]. Based on this fact, we replace the measurement equation [cf. (19)] with the following pair of inferred measurements of the camera pose expressed with respect to the global frame of reference:
TeX Source
$$\eqalignno{{}^{ {G}}\bar{q}_{ {C}}&= {\bm\xi}_1 ({\bf z}_1,{\bf z}_2,{\bf z}_3,{\bf z}_4) = {\bf h}_1^\ast ({\bf x}) = {\bf J} \bar{q}_{ {I}}\otimes \bar{q}_{ {C}}&\hbox{(36)}\cr {}^{ {G}}{\bf p}_{ {C}}&= {\bm\xi}_2 ({\bf z}_1,{\bf z}_2,{\bf z}_3,{\bf z}_4) = {\bf h}_2^\ast ({\bf x}) = {\bf p}_{ {I}}+ {{\bf C}}^{ {T}}(\bar{q}_{ {I}}) {\bf p}_{ {C}}&\hbox{(37)}}$$where is the rotational matrix corresponding to the quaternion , ⊗ denotes quaternion multiplication, and
TeX Source
$${\bf J} {\bar{q}}_{ {I}}= {\bar{q}}_{ {I}}^{-1}, {\bf J}\; {\buildrel \triangle \over =}\; \left[\matrix{ -{\bf I}_{3\times 3} & 0 \cr 0 & 1 }\right].\eqno{\hbox{(38)}}$$At this point, we should note that the functions ξ_{1} and ξ_{2} in (36) and (37) need not to be known explicitly. Instead, what is required for the observability analysis is their functional relation with the random variables, and **p**_{I}, and the unknown parameters, and **p**_{C}, appearing in the system's state vector.

Furthermore, we enforce the unit-quaternion constraints by employing the following additional measurement equations:
TeX Source
$$\eqalignno{ & h_3^\ast ({\bf x}) = \bar{q}_{ {I}}^{ {T}}\bar{q}_{ {I}}- 1 = 0&\hbox{(39)}\cr & h_4^\ast ({\bf x}) = \bar{q}_{ {C}}^{ {T}}\bar{q}_{ {C}}- 1 = 0.&\hbox{(40)}}$$

According to Remark 1, it suffices to show that a subset of the rows of the observability matrix [cf. (32)] are linearly independent. In the remaining of this section, we prove that the system described by (34) and (36)– (40) is observable by computing among the candidate zeroth-, first-, and second-order Lie derivatives of **h**_{1}∗, **h**_{2}∗, and *h*_{3}∗, the ones whose gradients ensure that is full rank.

1) *Zeroth-Order Lie Derivatives* (): By definition, the zeroth-order Lie derivative of a function is the function itself, i.e.,
TeX Source
$$\eqalignno{&{{{\L}}}^0 {\bf h}_1^\ast = {\bf h}_1^\ast = \bar{q}_{{ {I}}}^{-1} \otimes \bar{q}_{{ {C}}}&\hbox{(41)}\cr&{{{\L}}}^0 {\bf h}_2^\ast = {\bf h}_2^\ast = {\bf p}_{{ {I}}} + {{\bf C}}^{T}(\bar{q}_{{ {I}}}){\bf p}_{{ {C}}}&\hbox{(42)}\cr&{{{\L}}}^0 h_3^\ast = h_3^\ast = \bar{q}_{ {I}}^{ {T}}\bar{q}_{ {I}}- 1.&\hbox{(43)}}$$Therefore, the gradients of the zeroth-order Lie derivatives are exactly the same as the Jacobians of the corresponding measurement functions:
TeX Source
$$\eqalignno{&\nabla {{{\L}}}^0 {\bf h}_1^\ast = \left[\matrix{ {\cal R}(\bar{q}_{{ {C}}}) {\bf J} & {\bf 0}_{4\times 12} & {\cal L}({\bf J} \bar{q}_{{ {I}}}) & {\bf 0}_{4\times 3} }\right]&\hbox{(44)}\cr &\nabla {{{\L}}}^0 {\bf h}_2^\ast = \left[\matrix{{\bm \Psi} (\bar{q}_{{ {I}}},{\bf p}_{{ {C}}}) & {\bf 0}_{3\times 9} & {\bf I}_{3\times 3} & {\bf 0}_{3\times 4} & {{\bf C}}^{ {T}}(\bar{q}_{{ {I}}}) }\right]\qquad&\hbox{(45)}\cr &\nabla {{{\L}}}^0 h_3^\ast = \left[\matrix{ 2{\bar{q}}_{ {I}}^{ {T}} & {\bf 0}_{1\times 19} }\right]&\hbox{(46)}}$$where, for a quaternion and a vector **p**, we define
TeX Source
$$\eqalignno{ & {{\cal L}({\bar{q}})\; {\buildrel \triangle \over =}\; \left[\matrix{ q_4 {\bf I}_{3\times 3} - \lfloor {\bf q}\, \times \rfloor & {\bf q} \cr -{\bf q}^{ {T}} & q_4 }\right]}&\hbox{(47)}\cr & {{\cal R}({\bar{q}}) \;{\buildrel \triangle \over =}\; \left[\matrix{ q_4 {\bf I}_{3\times 3} + \lfloor {\bf q}\, \times \rfloor & {\bf q} \cr -{\bf q}^{ {T}} & q_4 }\right]}&\hbox{(48)}}$$and
TeX Source
$$\bm\Psi ({\bar{q}},{\bf p})\ {\buildrel \triangle \over =}\ { \partial \over \partial {\bar{q}}} {{\bf C}}^{ {T}}({\bar{q}}) {\bf p}. \eqno{\hbox{(49)}}$$Also, note that for deriving (44), we have used the following identities [23]:
TeX Source
$$\eqalignno{\bar{q}_{{ {I}}}^{-1} \otimes \bar{q}_{{ {C}}} &= {\cal R} (\bar{q}_{{ {C}}}) \bar{q}_{{ {I}}}^{-1} = {\cal R} (\bar{q}_{{ {C}}}) {\bf J} \bar{q}_{{ {I}}}\cr &={\cal L} ({\bf J}\bar{q}_{{ {I}}}^{-1}) \bar{q}_{{ {C}}} = {\cal L}({\bf J} \bar{q}_{{ {I}}}) \bar{q}_{{ {C}}}}$$
TeX Source
$${\cal O} = \left[\matrix{ \nabla {{{\L}}}^{0} {\bf h}_1^\ast \cr \nabla{{{\L}}}^{0} {\bf h}_2^\ast \cr \nabla {{{\L}}}_{{\bf f}_0}^1 {\bf h}_1^\ast \cr \nabla {{{\L}}}_{{\bf f}_0}^1 {\bf h}_2^\ast \cr \nabla{{{\L}}}_{\underline{{\bf f}}_{1ij}}^1 {\bf h}_2^\ast \cr \nabla{{{\L}}}^0 h_3^\ast \cr \nabla {{{\L}}}_{{\bf f}_0}^2 {\bf h}_2^\ast}\right] = \left[\matrix{ {\cal R}(\bar{q}_{{ {C}}}) {\bf J} & {\bf0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf0}_{4\times 3} & {\cal L}({\bf J} \bar{q}_{{ {I}}}) & {\bf0}_{4\times 3} \cr {\bm\Psi} (\bar{q}_{{ {I}}},{\bf p}_{{ {C}}}) &{\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} &{\bf I}_{3\times 3} & {\bf 0}_{3\times 4} & {{\bf C}}^{{T}}(\bar{q}_{{ {I}}})\cr {\bf X}_1 & -{ 1\over 2} {\cal R}({\bar{q}}_{ {C}}) {\bf J} {\bm\Xi} ({\bar{q}}_{ {I}}) & {\bf0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf X}_2 & {\bf 0}_{4\times 3} \cr {\bf X}_3 & {\bf X}_4 & {\bf I}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf0}_{3\times 4} & {\bf X}_5\cr {\bm\Gamma} _{ij}({\bar{q}}_{ {I}},{\bf p}_{ {C}}) & {\bf 0}_{6\times 3} & {\bf 0}_{6\times 3} & {\bf0}_{6\times 3} & {\bf 0}_{6\times 3} & {\bf 0}_{6\times 4} & {\bm\Upsilon} _{ij} ({\bar{q}}_{ {I}})\cr 2{\bar{q}}_{ {I}}^{ {T}} & {\bf0}_{1\times 3} & {\bf 0}_{1\times 3} & {\bf 0}_{1\times 3} & {\bf0}_{1\times 3} & {\bf 0}_{1\times 4} & {\bf 0}_{1\times 3}\cr {\bf X}_6 & {\bf X}_7 & {\bf 0}_{3\times 3} & -{{\bf C}}^{{T}}({\bar{q}}_{ {I}}) & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 4} &{\bf X}_8 }\right]\eqno{\hbox{(50)}}$$

2) *First-Order Lie Derivatives* (): The first-order Lie derivatives of **h**_{1}∗ and **h**_{2}∗ with respect to **f**_{0} are computed as (50), shown at the bottom of the page
TeX Source
$$\eqalignno{&{{{\L}}}^1_{{\bf f}_0} {\bf h}_1^\ast = \nabla {{{\L}}}^{0} {\bf h}_1^\ast \cdot {\bf f}_0 = {\textstyle -{ 1\over 2} } {\cal R}({\bar{q}}_{ {C}}) {\bf J} {\bm\Xi} ({\bar{q}}_{ {I}}) {\bf b}_g&\hbox{(51)}\cr &{{{\L}}}^1_{{\bf f}_0} {\bf h}_2^\ast = \nabla {{{\L}}}^{0} {\bf h}_2^\ast \cdot {\bf f}_0 = {\textstyle -{ 1\over 2} } {\bm \Psi} ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) {\bm\Xi} ({\bar{q}}_{ {I}}) {\bf b}_g + {\bf v}&\hbox{(52)}}$$while their gradients are given by
TeX Source
$$\eqalignno{&\nabla {{{\L}}}^1_{{\bf f}_0} {\bf h}_1^\ast = \left[\matrix{ {\bf X}_1 & -{ 1\over 2} {\cal R}({\bar{q}}_{ {C}}) {\bf J} {\bm\Xi} ({\bar{q}}_{ {I}}) & {\bf 0}_{4\times 9} & {\bf X}_2 & {\bf 0}_{4\times 3} }\right]\cr &\nabla {{{\L}}}^1_{{\bf f}_0} {\bf h}_2^\ast = \left[\matrix{ {\bf X}_3 & {\bf X}_4 & {\bf I}_{3\times 3} & {\bf 0}_{3\times 10} & {\bf X}_5 }\right].&\hbox{(53)}}$$In these last expressions, **X**_{i}, *i* = 1, …, 5, are matrices of appropriate dimensions (4 × 4 the first two, 3 × 4 the third one, and 3 × 3 the last two) which, regardless of their values, will be eliminated in the following derivations; hence, they need not be computed explicitly.

The next first-order Lie derivative of interest is that of **h**_{2}∗ with respect to __f___{1}, i.e., . At this point, we remind the reader that __f___{1} as defined in (34) is a compact representation of three column vectors. Similarly, we can also write the resulting Lie derivative in a compact form (i.e., a 3 × 3 matrix):
TeX Source
$${{{\L}}}^1_{\underline{{\bf f}}_1} {\bf h}_2^\ast = \nabla {{{\L}}}^{0} {\bf h}_2^\ast \cdot \underline{{\bf f}}_1 = { 1\over 2} {\bm\Psi} ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) {\bm\Xi} ({\bar{q}}_{ {I}}).\eqno{\hbox{(54)}}$$The gradients of the three columns of stacked together give
TeX Source
$$\nabla {{{\L}}}^1_{\underline{{\bf f}}_{1}}{\bf h}_2^\ast = \left[\matrix{ {\bm \Gamma} ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) & {\bf 0}_{9\times 16} & {\bm \Upsilon} ({\bar{q}}_{ {I}}) }\right]\eqno{\hbox{(55)}}$$where the matrices
TeX Source
$${{\bm\Gamma}} ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) = \left[\matrix{{{\bm\Gamma}} _1 ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) \cr{\bm\Gamma} _2 ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) \cr{\bm\Gamma} _3 ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) }\right] \qquad {\bm\Upsilon} ({\bar{q}}_{ {I}}) = \left[\matrix{ {\bm\Upsilon} _1({\bar{q}}_{ {I}}) \cr{\bm\Upsilon} _2({\bar{q}}_{ {I}}) \cr{\bm\Upsilon} _3({\bar{q}}_{ {I}}) }\right]\eqno{\hbox{(56)}}$$of dimensions 9 × 4 and 9 × 3, respectively, have block-row elements (for *i* = 1, …, 3)
TeX Source
$$\eqalignno{{\bm\Gamma} _i ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) &= { \partial \over \partial {\bar{q}}_{ {I}}} \left[\left({{{\L}}}^1_{\underline{{\bf f}}_{1}}{\bf h}_2^\ast \right) {\bf e}_i \right] \cr{\bm\Upsilon} _i ({\bar{q}}_{ {I}}) &= { \partial \over \partial {\bf p}_{ {C}}} \left[\left({{{\L}}}^1_{\underline{{\bf f}}_{1}}{\bf h}_2^\ast \right) {\bf e}_i \right]}$$with **e**_{1} = [1 0 0]^{T}, **e**_{2} = [0 1 0]^{T}, and **e**_{3} = [0 0 1]^{T}.

Note that inclusion of all the block-row elements of the gradient (55) in the observability matrix [cf. (50)] implies that all components of *ω*_{m} are nonzero. However, as it will become evident later on, in order to prove observability, *only two* of the elements of *ω*_{m} need to be nonzero. In such case, matrix will contain the block matrices
TeX Source
$${\bm\Gamma} _{ij} ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) = \left[\matrix{ {\bm\Gamma} _i ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) \cr {\bm\Gamma} _j ({\bar{q}}_{ {I}},{\bf p}_{ {C}}) }\right] \qquad {\bm\Upsilon} _{ij} ({\bar{q}}_{ {I}}) = \left[\matrix{ {\bm\Upsilon} _i({\bar{q}}_{ {I}}) \cr {\bm\Upsilon} _j({\bar{q}}_{ {I}}) }\right]\eqno{\hbox{(57)}}$$with *i*,*j* = 1, …, 3, *i* ≠ *j*, instead of and .

3) *Second-Order Lie Derivative* (): Finally, we compute the second-order Lie derivative of **h**_{2}∗ with respect to **f**_{0}:
TeX Source
$$\eqalignno{{{{\L}}}^2_{{\bf f}_0} {\bf h}_2^\ast &= {{{\L}}}^1_{{\bf f}_0}{{{\L}}}^1_{{\bf f}_0} {\bf h}_2^\ast = \nabla {{{\L}}}^1_{{\bf f}_0} {\bf h}_2^\ast \cdot {\bf f}_0 \cr &= -{ 1\over 2} {\bf X}_3 \, \bm\Xi ({\bar{q}}_{ {I}}) {\bf b}_g + {\bf g} - {{\bf C}}^{ {T}}({\bar{q}}_{ {I}}){\bf b}_a&\hbox{(58)}}$$and its gradient
TeX Source
$$\nabla {{{\L}}}^2_{{\bf f}_0} {\bf h}_2^\ast = \left[\matrix{ {\bf X}_6 & {\bf X}_7 & {\bf 0}_{3\times 3} & -{{\bf C}}^{ {T}}({\bar{q}}_{ {I}}) & {\bf 0}_{3\times 7} & {\bf X}_8}\right]\eqno{\hbox{(59)}}$$where the matrices **X**_{5}, **X**_{6}, and **X**_{7} (of dimensions 3 × 4 the first one and 3 × 3 the last two) will be eliminated in the ensuing derivations, and therefore, we do not need to compute them explicitly.

Stacking together all the previously computed gradients of the Lie derivatives, we form the observability matrix shown in (50).

In order to prove that the system described by (34) and (36)– (40) is observable, we employ the result of Proposition 1 and show that matrix is full rank (i.e. , the state space of the system is spanned by the gradients of the Lie derivatives of the measurement functions [17], [18]). Before presenting the main result of this section (cf. Lemma 3), we first state the following two lemmas whose proofs are detailed in Appendix I.

*Lemma 1:* The matrix
TeX Source
$${\bf A}_{ij} = \left[\matrix{ {\bm\Gamma} _{ij} & {\bm\Upsilon} _{ij} \cr 2{\bar{q}}_{ {I}}^{ {T}} & {\bf 0}_{1\times 3} }\right]\eqno{\hbox{(60)}}$$formed by the fifth and sixth block-row elements of the first and last block-columns of the observability matrix [cf. (50)], with Γ_{ij} and ϒ_{ij} defined in (57), is full rank.

*Corollary 1:* The matrix described by (60) is full rank if the IMU-camera rig is rotated about at least two different axes.

Note that only two block rows of and [cf. (56)]—the ones corresponding to two nonzero components of *ω*_{m}—are included in **A**_{ij} [cf. (60)]. Therefore, the third component of *ω*_{m} can be zero (i.e., no rotation around the corresponding axis) without affecting the observability properties of the system.

*Lemma 2:* For any unit-quaternions and , matrix is full rank.

*Lemma 3:* The observability matrix [cf. (50)] is full rank when the IMU-camera rig is rotated about at least two different axes.

*Proof:* Here, we provide a sketch of the proof based on block Gaussian elimination (for details please see [40]). We start by employing Lemma 1 and Corollary 1 to eliminate all the matrices in the first and last columns of . The next step is to eliminate **X**_{2} using , i.e., the (1,6) block element of in (50). Note that is unit quaternion and [cf. (38), (40), and (47)]. Finally, since is full rank (cf. Lemma 2), it can be used to eliminate **X**_{4} and **X**_{7}. Following these steps, reduces to
TeX Source
$${\left[\matrix{ {\bf 0}_{4\times 4} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf I}_{4\times 4} & {\bf 0}_{4\times 3} \cr {\bf 0}_{3\times 4} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf I}_{3\times 3} & {\bf 0}_{3\times 4} & {\bf 0}_{3\times 3}\cr {\bf 0}_{3\times 4} & {\bf I}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 4} & {\bf 0}_{3\times 3} \cr {\bf 0}_{1\times 4} & {\bf 0}_{1\times 3} & {\bf 0}_{1\times 3} & {\bf 0}_{1\times 3} & {\bf 0}_{1\times 3} & {\bf 0}_{1\times 4} & {\bf 0}_{1\times 3} \cr {\bf 0}_{3\times 4} & {\bf 0}_{3\times 3} & {\bf I}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 4} & {\bf 0}_{3\times 3}\cr {\bf I}_{4 \times 4} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 3} & {\bf 0}_{4\times 4} & {\bf 0}_{4\times 3}\cr {\bf 0}_{3\times 4} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 4} & {\bf I}_{3\times 3}\cr {\bf 0}_{3\times 4} & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 3} & -{{\bf C}}^{ {T}}({\bar{q}}_{ {I}}) & {\bf 0}_{3\times 3} & {\bf 0}_{3\times 4} & {\bf 0}_{3\times 3} }\right]}.\eqno{\hbox{(61)}}$$Considering that a property of a rotation matrix is that it is full rank (), it is easy to see that (61) is full rank, indicating that is also full rank.

*Corollary 2:* The system described by (34) and (36)– (40) is observable regardless of the linear motion of the IMU-camera rig.

This is evident from the fact that for proving Lemma 3, we did *not* use any Lie derivatives with respect to __f___{2} [cf. (34)]. Therefore, **a**_{m}, the measured linear acceleration can take arbitrary values without compromising the observability of the system. This observation has important practical implications when no significant linear motion is possible due to physical constraints (e.g., calibration of an IMU-camera rig in an indoor laboratory): the IMU-camera transformation can be accurately estimated even if no linear acceleration is exerted.

*Remark 2:* Since no noise is injected into the system along the directions of the IMU-camera transformation [cf. (6)], regardless of the observability of the system, the uncertainty of the IMU-camera transformation will never increase.

When the linearization of the IMU-camera calibration system is sufficiently accurate, this remark has the following important implication: running the estimation algorithm during periods when the observability conditions are not met (e.g., as a result of stopping the IMU-camera rig) will not decrease the accuracy of the IMU-camera estimates, although it might not improve their quality either. However, it is advisable to excite at least two degrees of rotational freedom for sufficiently long time at the beginning of the calibration process, so as to significantly reduce the error in the IMU-camera transformation and ensure the validity of the linear approximation.