Algebra of L-Banded Matrices

Convergence is a crucial issue in iterative algorithms. Damping is commonly employed to ensure the convergence of iterative algorithms. The conventional ways of damping are scalar-wise, and either heuristic or empirical. Recently, an analytically optimized vector damping was proposed for memory message-passing (iterative) algorithms. As a result, it yields a special class of covariance matrices called L-banded matrices. In this paper, we show these matrices have broad algebraic properties arising from their L-banded structure. In particular, compact analytic expressions for the LDL decomposition, the Cholesky decomposition, the determinant after a column substitution, minors, and cofactors are derived. Furthermore, necessary and sufficient conditions for an L-banded matrix to be definite, a recurrence to obtain the characteristic polynomial, and some other properties are given.


I. INTRODUCTION
V ARIOUS kinds of iterative algorithms are widely used in the fields of statistical signal processing, compressed sensing, communications, machine learning, coding theory, etc.For example, gradient descent algorithms [1], [2] are used for convex optimization, Jacobi and Gauss-Seidel algorithms [3], [4] for linear systems, and message passing algorithms [5]- [11] for graphical models.
An iterative algorithm can be generally represented as where x t is the t-th estimate for x.A non-memory iterative algorithm x t = f t (x t−1 ) can be seen as a special case of (1).How to guarantee convergence of iterative algorithms is a crucial problem.Damping is an efficient technique to ensure the convergence of iterative algorithms.Scalar damping [10] is commonly used in the existing literature, where ζ is a scalar damping factor determined empirically, and xt is the damped estimate initialized with x1 = x 1 .
Recently, a non-empirical and analytically optimized vector damping was proposed for memory approximate message passing (MAMP) algorithms [11] as Shunqi Huang and Brian M. Kurkoski are with the School of Information Science, Japan Institute of Science and Technology (JAIST), Nomi 923-1292, Japan (e-mail: {shunqi.huang,kurkoski}@jaist.ac.jp).Lei Liu is with the Zhejiang Provincial Key Laboratory of Information Processing, Communication and Networking, College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310007, China (e-mail: lei liu@zju.edu.cn).(Corresponding author: Lei Liu.) where ζ t is a damping vector, V t the covariance matrix of x 1 , • • •, x t , 1 the all-one vector, V −1 t 1 the column-wise sum of V −1 t , and 1 T V −1 t 1 the sum of all the entries in V −1 t .It was found in [11] that the covariance matrix Vt of the damped estimates x1 , • • •, xt has a special structure that the entries in each "L band" are identical.Meanwhile, this special structure was found independently in [12], and played an important role in the convergence proof of orthogonal/vector AMP [7], [8].This particular type of matrix is referred to as L-banded matrices as follows.
Definition 1 (L-banded Matrix [11], [12]).A matrix A ≡ [a i,j ] ∈ R n×n is said to be an L-banded matrix if That is, A can be written as It is crucial to emphasize that the matrices satisfying (4) are referred to as L-matrices when a i ∈ C and n → ∞ in [13]- [17].In other words, L-banded matrices can be regarded as L-matrices with real and finite entries.
Significantly, it was shown that any L-banded covariance matrix converges in [11], [18], [19], i.e., its diagonal entries converge (See Lemma 2).As a result, the convergence difficulties of general iterative algorithms were resolved in principle.The discussions in [11], [12], [18], [19] are limited to Lbanded covariance matrices.Although some properties of Lbanded matrices were included in previous works (see Section I-B), more algebraic properties remain unknown.
In this paper, we show that the structure of L-banded matrices gives rise to a number of algebraic properties, which are broader than those properties which have been uncovered thus far.This algebra of L-banded matrices, besides being potentially useful for signal processing applications, is also inherently interesting.These new properties include an analytic expressions for the LDL decomposition, the Cholesky decomposition, minors, cofactors, and the determinant of the matrix formed by replacing any one column.In addition, necessary and sufficient conditions for an L-banded matrix to be definite, a recurrence to obtain the characteristic polynomial, and some other properties are given.We also provide new derivations of the determinant and the inverse.Finally, a comparison of the time complexity of some operations on L-banded matrices and those of general matrices is given.

A. Notation
Boldface lowercase and boldface uppercase symbols denote column vectors and matrices, respectively.A ≡ [a i,j ] n×n denotes that A is an n×n matrix with (i, j)-th entries a i,j .I n denotes an n × n identity matrix."iff" is the abbreviation of "if and only if".We call the matrix in (4) an n × n L-banded matrix with [a 1 , • • •, a n ].In the following paragraphs, we assume that A is an n × n L-banded matrix with [a 1 , • • •, a n ] unless otherwise specified.
Lemma 5 (Inverse [16], [18], [19]).Suppose that That is, A −1 can be written as II. MAIN RESULTS In this section, we give our main results.Necessary and sufficient conditions for the definiteness of A are given in Section II-A.The LDL decomposition and the Cholesky decomposition are given in Section II-B.The minors, cofactors, and determinant after a column substitution are given in Section II-C.The characteristic polynomial is given in Section II-D.Some other properties are given in Section II-E.The new proofs of Lemma 4 and Lemma 5 are given in Section II-F and II-G, respectively. where (⋆) can be expanded as (#) can be expanded as Then, (8c) can be rewritten as Thus, we finish the proof.
Lemma 6 not only shows a simpler expression for the quadratic form, but also is the key to proving Theorem 1.
Theorem 1 (Definiteness).The following statements hold: From Lemma 6, ( 12) is equivalent to: For any real Note that ( are not all zeros since x 1 , • • • , x n are not all zeros.Thus, ( 13) is equivalent to (14) is equivalent to The proofs for the other cases are omitted since they are similar.
Compared to Lemma 1 and Lemma 2, Theorem 1 shows that the necessary conditions for A to be positive definite and positive semi-definite are also sufficient conditions.Though the second statement was proved in Lemma 3, Theorem 1 gives a different proof.In addition, necessary and sufficient conditions for A to be negative definite and negative semidefinite are listed in Theorem 1.

B. LDL decomposition and Cholesky decomposition
The LDL decomposition of a real symmetric matrix has the form LDL T , where L is a unit lower triangular matrix and D is a diagonal matrix.
Theorem 2 (LDL decomposition).Suppose that a k = 0, ∀k ∈ {1, • • •, n−1}.The LDL decomposition of A can be where L ≡ [l i,j ] n×n is a unit lower triangular matrix as and D is a diagonal matrix with diagonal entries That is, L and D can be written as • • • Proof.For i, j ∈ {1, • • •, n}, let b T i denote the i-th row of LD and l j denote the j-th column of L T .We have Let m = min(i, j), t = max(i, j), and Thus, we have proved that A = LDL T .
Proof.First, we want to show "the LDL decomposition of Then, the LDL decomposition of A can be A = LDL T , where Second, we want to show "... only if ...".We consider showing the contrapositive, i.e., the LDL decomposition of A does not exist if ∃k ∈ {1, • • •, p−1}, a k = 0, and use proof by contradiction.Assume that there exists an LDL decomposition A = LDL T if ∃k ∈ {1, • • •, p−1}, a k = 0. We can always find m ∈ {1, • • •, p−1} such that a m = 0 and a m+1 = 0.For i, j ∈ {1, • • •, n}, the i-th row of LD and the j-th column of Then, we can obtain a m+1 = b T m+1 l m = 0, which leads to a contradiction.Thus, we finish the proof.

Proposition 2. There exists a unique LDL decomposition of
Proof.For k ∈ {1, • • •, n−1}, the k-th leading principal submatrix of A is the matrix A with the last n−k rows and columns removed, which is an L-banded matrix with all the leading principal submatrices of A are invertible.Thus, from Theorem 4.1.3in [20], we can show that the LDL decomposition is unique.
Remark 1. Suppose that a n = 0.The sufficient condition of "there exists a unique LDL decomposition of A" in Proposition 2 is also the necessary condition.
The Cholesky decomposition of a real symmetric positive definite matrix has the form LL T , where L is a lower triangular matrix.

Theorem 3 (Cholesky decomposition). Suppose that A is positive definite. The Cholesky decomposition of A is
where L ≡ [ li,j ] n×n is a lower triangular matrix as Recall That is, L can be written as Proof.We have a 1 > • • • > a n > 0 since A is positive definite by Theorem 1.Then, from Theorem 2, the LDL decomposition of A is Note that the diagonal entries of D are all positive since a 1 > • • • > a n > 0. We can rewrite (22a) as Let L = LD 1/2 , it is easy to prove the theorem.

C. Minors, Cofactors and Column Substitution
For i, j ∈ {1, • • •, n}, let M i,j (A) be the (i, j)-th minor of A, C i,j (A) be the (i, j)-th cofactor of A, and C(A) ≡ [C i,j (A)] n×n be the cofactor matrix of A.
Theorem 4 (Minors and Cofactors).Suppose that A is invertible and n ≥ 2.Then, the cofactors and minors of A are Proof.Let adj(A) be the adjugate matrix [21] of A. Since A is symmetric, Since A is invertible, Then, from Lemma 5, we can prove the theorem.

Theorem 5 (Determinant after Column Substitution). Suppose that
where Proof.The Laplace expansion [20] along the column k of The k-th entry of A −1 b is and then we can easily prove the theorem.

D. Characteristic Polynomial
Theorem 6 (Characteristic Polynomial).Suppose that A is invertible.The characteristic polynomial of A is where f n can be obtained by a three-term recurrence: Lemma 5.In other words, the eigenvalues of A are the roots of f n .
Proof.Since A is invertible, Note that λA −1 − I is a tridiagonal matrix with entries Then, we can obtain the recurrence in (30b) by Theorem 2.1 in [22]. Since i.e.
Proof.For i, j ∈ {1, • • •, n}, let h T i denote the i-th row of H and a j denote the j-th column of A. We have Let t = max(i, j).The (i, j)-th entry of Q is ] n×n , t = max(i, j) and m = min(i, j).Then, Thus, we have proved the proposition.
Proposition 4. The linear combination of L-banded matrices is an L-banded matrix.
Proof.The linear combination of matrices can be seen element-wise.Thus, the linear combination of any number of L-banded matrices is still an L-banded matrix.

F. New Derivation for Determinant
For i, j ∈ {1, • • •, n}, let A i,j denote the matrix A with row i and column j removed.

G. New Derivation for Inverse
We give a new proof of Lemma 5 by mathematical induction.
(1) Base case: and we assume that the theorem holds.For n = p + 1, A is a (p + 1) × (p + 1) L-banded matrix with [a 1 , • • •, a p+1 ].Since A and A are invertible, for k ∈ {1, • • •, p}, a k = a k+1 , a p = 0 and a p+1 = 0. Let 1 ∈ R p be an all-ones vector.It is easy to verify that Then, we let Expressing the matrix inverse in block form, .Then, substitute (44) into (43b), we can find that A −1 is the same as the matrix in (6).Thus, the holds for n = p+1.

H. Complexity Comparison
We give a comparison of the time complexity of some operations on L-banded matrices and those of general real symmetric matrices.
In Table I, the notation "⋆" and " †" means the corresponding matrices are invertible and positive definite, respectively.In

TABLE I COMPARISON FOR THE COMPLEXITY OF OPERATIONS ON L-BANDED
MATRICES AND GENERAL SYMMETRIC MATRICES addition, we need to point out that some methods can find determinants of general matrices with complexity between O(n 2 ) and O(n 3 ).However, the most common methods, like LU decomposition or Bareiss algorithm, are in O(n 3 ).

III. CONCLUSIONS
In this paper, we gave many algebraic properties of Lbanded matrices.We expect that our findings in this research will contribute to the fields of mathematics, iterative signal processing, message-passing algorithms, and other relevant applications that employ the L-banded matrices.

For
k ∈ {1, • • •, n}, let A k ← b denote the matrix formed by replacing column k of A with b.
respectively.Then, column 1 and column 2 of A with row 1 removed are both [a 2 , • • •, a n ] T .Thus, |A 1,k | = 0 for k ≥ 3 since the column 1 and column 2 of A 1,k are the same.In addition, |A 1,1 | = |A 1,2 |.Then, we give a new proof of Lemma 4 by mathematical induction.(1) Base case: For n = 1, A = [a 1 ] so that |A| = a 1 .(2) Induction step: For n = p, A is a p × p L-banded matrix with [a 1 , • • •, a p ] and we assume that Lemma 4 holds.For n