The Bias-Compensated Proportionate NLMS Algorithm With Sparse Penalty Constraint

For compensating the bias caused by the noisy input which is always ignored by ordinary algorithms, two novel algorithms with zero-attraction (ZA) penalties are proposed in this paper. The first one constructs a bias-compensated term in the updating recursion of the zero-attraction proportionate normalized least mean square (PNLMS) algorithm which is named BC-ZA-PNLMS algorithm. The second one employs the bias-compensated term and the correntropy induced metric (CIM) constraint to renew the updating recursion of the PNLMS algorithm which is named BC-CIM-PNLMS algorithm. Both of these two algorithms are derived on the basis of unbiased criterion. Simulation examples are carried out, and the results indicate that the two newly developed unbiased algorithms outperform the related algorithms previously presented in other literatures for combating noisy input and measurement noises.


I. INTRODUCTION
Normalized least-mean-square (NLMS) algorithm is one of the popular adaptive filtering algorithms which are widely used in signal processing field, such as echo cancellation, system identification and linear prediction [1], [2]. Because of the simplicity and excellent performance of the NLMS, the family of the NLMS algorithms are extensively and deeply studied. It is well known that the channel responses of several practical systems such as the wireless communication system and echo path, are sparse, which means that most of the parameters to be estimated are insignificant and negligible, only small part of the parameters are remarkable and considerable [3]- [5]. For the purpose of taking advantages of the sparse characteristics in these systems, the proportionate type algorithms which obtain fast convergence speed by assigning an independent step-size to each coefficient have been developed [6]- [11]. Among these algorithms, the most popular one is the proportionate normalized least mean square (PNLMS) The associate editor coordinating the review of this manuscript and approving it for publication was Min Jia . algorithm [6]. Inspired by the compressed sensing, a type of zero attraction algorithm has been developed by introducing a penalty term into the updating recursion of the original adaptive filter algorithm [12]- [19]. The role of the sparse penalty is to attract the small value coefficients to zero, consequently, faster convergence speed can be obtained than that of original algorithms and even than that of the proportionate type algorithms. Among the zero attraction algorithms, the zero attraction NLMS (ZA-NLMS) algorithm is the most famous one [14]. Besides the traditional zero-attractors obtained from different norm penalty, the correntropy induced metric (CIM) method has been considered to construct a zero-attractor in [20]- [26], which is to measure the similarity between the two different variables. If one variable is zero, the CIM tends to be l 0 -norm. Thus, the CIM can be used to approximate the l 0 -norm to exploit the sparsity of the system.
Although the two categories algorithms which fall into proportionate type one and zero attraction type one can get smaller steady state error and faster convergence, which only focus on the output noise and do not consider the input noise. However, the input noise caused by modeling and instrument error is unavoidable, and the neglect of input noise may introduce estimation bias which decreases the accuracy accordingly. Using higher order model and increasing the signal-to-noise ratio (SNR) can mitigate the adverse impact of input noise in some degree but can never be eliminated completely [27]. To overcome this drawback, some improved algorithms for unbiased estimation have been investigated [28]- [39]. The total least squares (TLS) and the biascompensated least squares (TL) methods can handle the bias issue but suffer to computational inefficiency [40]. Therefore, the bias-compensated algorithms under least mean square criterion obtain more attention relay on their easy implementation and low complexity. In [28], the unbiased criterion for steady state is utilized to provide a simple approach of bias compensation via the statistical property of the input noise. Since then, unbiased algorithms deriving from the unbiased criterion have been developed. Among the existing ideas, the classical adaptive filter algorithms like the NLMS [1], [2], [41], PNLMS [6], constrained least-mean-square (CLMS) [42], least mean fourth (LMF) [43], [44], normalized subband adaptive filter (NSAF) [45], [46], proportionate least-meansquare/fourth (PLMS/F) [47]- [49], affine-projection-like (APL) [50], [51], normalized maximum correntropy criterion (NMCC) [20], [21], [52]- [54], fractional order normalized least mean square (FONLMS) [55]- [58] are improved by reducing the impacts of the input noise via appending a bias-compensated term into the updating equation of the primal algorithms. Furthermore, the stabilization for the biascompensated NLMS (BC-NLMS) algorithm based on unbiased criterion was also researched carefully in [59].
Inspired by the idea of bias-compensation and motivated by utilizing the sparsity of system impulse response, the novel algorithm is proposed to utilize both of the bias-compensated and sparsity to construct new updating recursion of the PNLMS algorithm in this paper. The bias-compensated term is derived from the unbiased criterion and incorporated with zero-attraction scheme. Besides the traditional zero attractor of the sign function, the CIM is also be employed to form the desired zero attractor. The two proposed algorithms are named BC-ZA-PNLMS and BC-CIM-PNLMS, respectively.

A. THE PNLMS ALGORITHM WITH ZERO ATTRACTOR
Consider a finite impulse response (FIR) system with L taps, and the weight vector w(k) denotes a columnvector with L rows. The system input signal is T , which is noisefree vector. The desired signal is described as n(k) represents the noise signal from the environment. The system estimation error is with an estimationŵ(k).
The PNLMS's update recursion iŝ with a step-size of µ, where ε 0 is to prevent the denominator from zero. Q(k) is a weight assignment matrix which assigns individual step-size to each tap corresponding to its magnitude, and it is described as The element q in the matric Q(k) is where The parameter ρ in (6) is used to keep the iteration going on when ŵ l (k) is much smaller than the previous item, and its value is always 1 L ∼ 5 L . The parameter δ in (6) is a small constant to promote the iteration running at beginning when all the coefficients are zeros.
The PNLMS can achieve fast convergence speed at initial stage of iteration which benefits from the step-size individual assignment of each coefficient. However, the advantage cannot be maintained at the later stage, which suffers from the slow convergence speed of the small value coefficients that are assigned small step-sizes. To accelerate the convergent of the small coefficients, the l 1 -norm regularization constraint is introduced to the optimization of the PNLMS algorithm to force the small coefficients to approach zeros rapidly. Then, the resulting updating recursion of the ZA-PNLMS algorithm is obtained aŝ It is observed that the first two terms of the iteration are the same as those of the PNLMS algorithm, and the last term of the sign function with a zero attraction strength control parameter γ is the constructed zero attractor.

B. REVIEW OF THE CIM
Recently, the CIM is introduced to the adaptive filter which acts as an approximation of l 0 -norm [20]- [26]. Compared with the global measurement of the mean square error (MSE), the correntropy focuses on local statistics. To describe the similarity of two random vectors, the correntropy of two vectors W and Z is defined as where κ(.) represents the kernel used to satisfy the Mercer's theorem. Among the various kernel functions, the Gaussian kernel is the most popular one, and it is described as where the kernel width is σ , and its size is equal to the range of a special set. The CIM is which is a nonlinear metric derived from the correntropy. Herein, make W toŵ(k) and make Z to zero, then, the degree to which the coefficient approaches zero can be obtained. This is lucky coincidence to the role of the zero attractor. Thus, the CIM can be chosen to be a sparse penalty to exploit the sparsity of the original adaptive algorithm. To simplify the expression, the CIM usually takes squared rather than the form of square root which is shown as Taking the derivation of CIM 2 ŵ(k), 0 with respect toŵ(k), we can get The updating recursion of the PNLMS algorithm with CIM penalty can be described aŝ Comparing to (7), we can find that γ CIM 2σ 2 is regarded as a zero attractor, and the algorithm with CIM constraint is also can be considered as a zero attraction type algorithm.

III. THE BIAS-COMPENSATED ZERO ATTRACTION ALGORITHMS WITH INPUT NOISE
The algorithms mentioned in the previous section can both exploit the system's sparse characteristic and further accelerate the convergence rate, nevertheless, the input noise of the filter which is unavoidable in practice is not taken into account. To mitigate the adverse impacts of the noisy input, an unbiased term is incorporated into the updating equation of the traditional algorithm.
Input signal containing noises is described as with n in (k) = [n in (k), n in (k − 1), n in (k − 2) · · · , n in (k − L+1)] T is the input noise (see Fig.1) that is assumed to be white Gaussian noise (WGN) and independent of v(k). The resulting error signal will be replaced by in (k)ŵ(k). (15) As can be seen from (15), the considering input noise causes additional bias compared with e(k) which is expressed in (2). Therefore, the updating equation of theŵ(k) shown in (7) is improper. To make the estimation accurate, a biascompensation term is introduced and is written as W BC . Then, the new updating equation of the BC-ZA-PNLMS algorithm turns tô (16) In order to figure out the concrete expression of W BC (k), the unbiased criterion proposed in [28] is employed which is described as where The unbiased criterion is only satisfied in steady state, and by this time the w * tends to zero. According to (16) and (18), we can get Taking the conditional expectation of both sides of (19) to get the same form in (17). It is found that the third term on the right-hand side of the equation can be omited in the following operation, avoiding introducing extra bias and being unnecessary for input noise elimination [35]. Then, the result of taking conditional expectation will be In steady state, the equation will become where Substituting (14) and (15) into (22), we will obtain . (23) To make the calculation tractable, some assumptions are given. First of all, the input noise n in (k) is independent of v(k) with mean and variance of zero and δ 2 in , respectively. Next, the observation noise n(k) is independent of n in (k) and v(k) with N (0, 1) distribution. In addition, both of x(k) and v(k) are uncorrelated to w * (k). Based on these assumptions, (23) can be rewritten as where E v(k)n in (k) Tŵ (k) |x(k) = 0, E n in (k)n T in (k)ŵ(k) |x(k) = δ 2 in E ŵ(k) |x(k) . (28) Substituting the result into (21), yields Then, the bias-compensation term can be calculated via (29) Substituting this result into (16), the updating equation of the bias-compensation ZA-PNLMS (BC-ZA-PNLMS) algorithm is obtained aŝ From (31), it can be found thatŵ(k  x T (k)Q(k)x(k)+ε is the bias-compensated term, and γ sgn(ŵ(k)) is the sparse penalty constraint.
Furthermore, the bias-compensation PNLMS algorithm with CIM constraint can be derived in the similar way. As a result, the updating equation of the BC-CIM-PNLMS is expressed aŝ which works more effectively benefits from the adjustable width of the kernel σ .  The key parameter δ 2 in in (31) and (32) is the variance of the input noise which usually can not be obtained directly in practice. So the methods of δ 2 in estimation are investigated in several literatures and are summarized in [59]. Herein, δ 2 in is replaced by δ 2 in (k) which is the instant value at kth iteration, and δ 2 in (k) can be estimated by following equations based on time ergodicity. where The parameter f in (34) and (35) is a forgetting parameter, and η is the pre-known input-output noise-ratio. From the derivation of the two algorithms, we can conclude that the BC-ZA-PNLMS and BC-CIM-PNLMS have both of fast convergence rate and low estimation error which benefits from the inserting of the bias-compensated term and sparse penalty constraint. However, the computational complexity is a little bit higher than the original algorithm. The recursion of the PNLMS requires 2L 2 + 2L − 1 additions, 2L 2 + 3L + 2 multiplications and L + 1 divisions. The BC-ZA-PNLMS requires 3L 2 +3L −1 additions, 3L 2 +5L +3 multiplications and L + 1 divisions. The BC-CIM-PNLMS requires 3L 2 + 4L −2 additions, 3L 2 +7L +7 multiplications, L +3 divisions and L exponentiation.   Figure 2.
We can conclude that the BC-ZA-PNLMS and BC-CIM-PNLMS algorithms have superior performance respect with estimation precision under different sparsities. In general, the algorithms with bias-compensation term behave better than those algorithms without bias-compensation term which is shown in Figure 2(b) and Figure 2(c). However, if the channel response is extremely sparse, the proportional algorithms show their superiority comparing with the non-proportional algorithms which is shown in Figure 2(a).
Example 2: The performance of the two developed algorithms for different SNRs are studied. The experimental environment and parameter settings are the same as those in Example 1. Herein, the SNR denotes the input signal variance and input noise variance ratio whose values are 0dB, 10dB, 20dB and 30dB, respectively. The MSD lines of the BC-ZA-PNLMS and BC-CIM-PNLMS algorithms are shown in Figure 3. It is evident from the result that the lower the SNR is, the worse the algorithm performs.
Example 3: The behaviors of BC-ZA-PNLMS and BC-CIM-PNLMS with different input signals are investigated. The input signals include WGN signal with distribution of N (0, 1), colored signal that is generated by WGN through a first order filter with a pole of 0.8, and the speech signal which lasts 8 seconds and is sampled by 8kHz, respectively. The SNR is set to 10dB and the sparsity is 4. The step-sizes of BC-ZA-NLMS and BC-NLMS are 0.4, and the step-sizes for other algorithms are 0.12. f is 0.1. The estimation behaviors of the algorithms with WGN, colored signal and speech signal are shown in Figure 4.
As shown from the results, the developed algorithms can still maintain the superior performance for dealing with the WGN, colored signal and speech signal. However, the learning line of speech signal which lasts 8 seconds is not smooth since it is time-varying. As a result, the results obtained from the speech signal do not look smooth enough [30], [36], [39], [59].
Example 4: For the sake of verifying the stability of the developed algorithms, different channel responses are considered and simulated. An echo path which is shown in Figure 5(a) has 256 taps. The performance is shown in Figure 5(b). The underwater communication channel shown in Figure 6(a) with 222 taps is considered [60]. The performance is shown in Figure 6(b). The simulation result validates the efficiency of the proposed algorithms for different applications.    Example 5: In this example, the values of the key parameters in the BC-ZA-PNLMS and BC-CIM-PNLMS algorithms are discussed. The appropriate valaue of zero attraction parameter γ in the BC-ZA-PNLMS algorithm is discussed, and the result is shown in Figure 7. It is observed that γ = 0.0001 is the best choice. In addition, the value of kernal width σ in the BC-CIM-PNLMS algorithm is discussed, and the result is shown in Figure 8. As can be seen from the result, both of the values σ = 0.1 and σ = 1 are the best ones.
Example 6: To illustrate the performance of the proposed algorithms in channel estimation more intuitively, the real response which is shown in Figure 6(a) is compared with the estimation one. Consider that the input signal is WGN and the SNR is 0dB which conforms the SNR of underwater communication channel. γ and γ CIM are 0.000001, and σ is 0.7. The comparison results are shown in Figure 9.
From the simulation results, we can find that both of the two novel algorithms can achieve good performance in estimating and tracking even at low SNR and complex environment.

V. CONCLUSION
The PNLMS algorithms with zero attraction scheme and biascompensated term are developed in this paper. The biascompensation term reduces the adverse effect of the noisy input which is always not considered by the traditional algorithms. The zero attractors incorporated into the PNLMS can accelerate the returning to zeros for the close-to-zero coefficients. The BC-ZA-PNLMS and BC-CIM-PNLMS algorithms have superior performance on both estimation accuracy and convergence speed. The derivations of the two algorithms are proposed and discussed on the basis of unbiased criterion, and the behaviors of the new algorithms are studied in a comprehensive way. Simulation results of numerical examples demonstrate the validity of bias compensation for suppressing the noisy input and the excellent performance of the two developed algorithms.