IEEE Xplore At-A-Glance
  • Abstract

Side Match Distortion Based Adaptive Error Concealment Order for 1Seg Video Broadcasting Application

Transmission of compressed video over error prone channels may result in packet losses or errors, which can significantly degrade the image quality. Such degradation even becomes worse in 1Seg video broadcasting application, which is widely used in Japan and Brazil for mobile phone TV service, where errors are drastically increased and huge conjunctive areas inside a picture may be corrupted. In this case the error concealment order—to decide which MB should be concealed earlier—may highly influence image quality. Aimed this problem, this paper proposes an adaptive concealment order based on well-known Boundary Matching Algorithm (BMA). The concealment order is carefully chosen according to a lost MB's priority, which is formulated considering a concealed MB's side match distortion: an MB with smaller distortion should be concealed earlier compared with an MB with larger distortion. As for formulation of side match distortion, not only the current corrupted MB's, but also the neighboring MB's, which is caused by error propagation, are included. Compared with reference work [10], the experiments show our proposal achieves better performance of video recovery under different error rate channel in 1Seg application.

SECTION I

Introduction

TRANSMISSION of compressed video over error prone channels such as wireless network may result in packet losses or errors in a received video stream. Such errors or losses do not only corrupt the current frame, but also propagate to the subsequent frames [1]. Several error control technologies, such as forward error correction (FEC), automatic retransmission request (ARQ) and error concealment (EC), have been proposed to solve this problem. Compared with FEC and ARQ, EC wins the favor since it doesn't need extra bandwidth and can avoid transmission delays [2].

The EC scheme attempts to recover the lost MBs by utilizing correlation from spatially or temporally adjacent macro blocks (MBs), i.e., spatial error concealment (SEC) or temporal error concealment (TEC) [2]. For TEC, which is focused on by this paper, several related works have been published to estimate the missing motion vector (MV) by using the correctly received MVs around the corrupted MB. In [3], the well-known boundary matching algorithm (BMA) is proposed to recover the MV from the candidate MVs by minimizing the side match distortion (Dsm) between the internal and external boundary of the reconstructed MB. This algorithm is adopted in H.264 reference software JM [4] and described in detail in [5], [6]. Based on BMA, several improved methods are proposed, such as [7], [8], [9].

It can be seen that, lost MV recovery is the most focus in published TEC methods, while the EC order—the decision of which MB should be concealed earlier inside the area of broken MBs—is rarely well considered. In BMA based [5], [6], the authors use the fixed concealment order, from boundary to center, column-by-column, inside a picture. In [10], the authors proposed an edge-sensitive order, which ranked each lost MB with their edge intensity, in the boundary of its available neighborhood. The edge intensity is extracted by sobel operator. This work is based on such observation that, in order to improve the image recovery, an MB which has stronger edge intensity (predominant edge is extracted) in the boundary of its available neighbors is going to be concealed earlier.

There are 2 problems in [10]'s work. Firstly, for the MBs with smooth boundary where no predominant edge may be extracted, it is hard to decide which MB should be concealed earlier, thus this algorithm sometimes is not efficient for such kind of MB's recovery. Secondly, edge property is not directly related with the priority determination. The more directly one is the side match distortion (Dsm ), which is the unique criterion for lost MV estimation in BMA based EC, see more detail in Section III. In other words, an MB with smaller Dsm is more likely to be concealed earlier.

Considering these 2 problems, in our work, Dsm is included in MB's priority formulation instead of edge intensity. Furthermore, for Dsm formulation, not only the distortion for current concealed MB, but also the potential propagated distortion for the neighboring lost MB are included.

The rest of this paper is organized as follows. Section II presents our work's motivation, which is for 1Seg application. Section III gives an overview of well-known BMA. Based on BMA, we present proposed EC order in Section IV. Finally Sections V and VI show our experiments and conclusion.

SECTION II

1Seg Application Based Motivation

Our work is targeted to 1seg application [11]. 1Seg is one of services of ISDB-T (Integrated Services of Digital Broadcasting- terrestrial) in Japan and Brazil. ISDB-T is designed so that each channel is divided into 13 segments. An HDTV broadcast signal occupies 12 segments, leaving the remaining (13th) segment for mobile receivers. Thus the mobile service is named with “1seg”. Due to errors drastically increased in wireless mobile terminal, EC is critical for 1Seg.

In 1Seg, H.264 format is specified for video compression. Table I shows the specification [11] for video compression and transmission in 1Seg related to this work.

Table 1
TABLE I Specification for Video Part in 1Seg Related to EC

According to Table I, even 1 broken packet may cause huge conjunctive area loss (from the start of broken packet to the end of the frame) inside a frame. This is because 1) slice mode and FMO mode are prohibited; 2) for QVGA sized video, each picture/slice usually consists of at least more than 1 TS packet (normally more than 3); and 3) variable length coding (VLC) based entropy coding is included in H.264. Fig. 1 shows a typical loss picture (foreman sequence) in 1Seg scenario, where green part is the destroyed area. In our experiment in Section V, we modified JM to support 1Seg.

Figure 1
Fig. 1. A typical broken frame in 1Seg.

As shown in Fig. 1, enormous areas are destroyed, therefore the concealment order in this area—to decide which MB should be concealed earlier—may highly influence the image quality of recovery. In Section IV, we will propose our solution toward this order problem.

SECTION III

Boundary Matching Algorithm (BMA) for TEC

The TEC method in JM uses fixed EC order, from frame boundary to center column-by-column. And for each lost MB, BMA [4], [5], [6] is utilized to recover the lost MV from candidate MVs (mvcan s) by minimizing the side match distortion (Dsm) between the IN-MB and OUT-MB, see Fig. 2. The IN-MB is projected by mvcan, which is the candidate MV belongs to one of 4 neighbors in current frame.

Figure 2
Fig. 2. BMA based TEC in H.264.

The Dsm is determined by the sum of absolute differences between the predicted MB and the neighboring MBs at the boundary, shown in Eq. (1).Formula TeX Source $$\eqalignno{&D_{sm} = {1\over N}\left\langle \sum^N_{i=1}\left\vert Y^{IN}_i (mv^{can}) -Y^{OUT}_i\right\vert\right\rangle\cr&where, mv^{can} \in \{mv^{top},mv^{bot},mv^{lft},mv^{rt}\}&\hbox{(1)}}$$

As shown in Fig. 2, N is the total number of the calculated boundary pixels, including correctly received (OK MB) and concealed boundary pixels in the neighbors. The winning prediction MV is the one which minimizes the side match distortion Dsm:Formula TeX Source $$mv^{win} = \mathop{\rm Arg\,Min}_{mv^{can} \in \{mv^{top},mv^{bot},mv^{lft},mv^{rt}\}}D_{sm}\eqno{\hbox{(2)}}$$

SECTION IV

Proposed EC Order

A. Overview of the Proposed EC Order

Fig. 3 shows the flow chart of EC process with proposed adaptive order. Note that, the traditional BMA is performed for the MB with the highest priority in each loop. Therefore, the key step in proposed order is the priority calculation for each lost MB.

Figure 3
Fig. 3. Flow of EC with proposed adaptive EC order.

B. Priority Formulation

It is inevitable that the current concealed MB may cause mismatch distortion compared with the original current MB. And it is also inevitable that the mismatch distortion of current MB will cause error propagation to the neighboring MB in the consequent concealment process, as the current concealed MB might be used to conceal the neighboring corrupted MBs. In BMA based TEC, the distortion can be expressed by side match distortion, since the original pixel is unavailable for a corrupted MB. Therefore, we can formulate the total side match distortion of a lost MB as follows:Formula TeX Source $$D_{sm\_total} = {D_{sm\_cur} + D_{sm\_nbr} \over 2}\eqno{\hbox{(3)}}$$where, Dsm_cur and Dsm_nbr are side match distortion for current MB itself, and the potential propagated side match distortion for neighboring corrupted MBs, respectively. Division by 2 is because the distortion should be normalized by pixel.

It is easy to know that, in order to improve the concealed image quality, an MB with smaller Dsm_total should be concealed earlier compared with an MB with larger Dsm_total. In other words, the MB with smaller Dsm_total should have higher priority in EC process. That it:Formula TeX Source $$priority = {1 \over D_{sm\_total}}\eqno{\hbox{(4)}}$$

In the next section, we will further formulate Dsm_total based on Eq. (3).

C. Distortion Formulation

Similar with Eq. (1) and (2), it is easy to formulate Dsm_cur as follows:Formula TeX Source $$\eqalignno{&D_{sm\_cur} = D_{sm}(mv^{can\_cur} = mv^{win\_cur})\cr&={\rm Min}_{mv^{can\_cur} \in \{mv^{top},mv^{bot},mv^{lft},mv^{rt}\}}{1\over N}\left\langle \sum^N_{i=1}\left\vert \sum^N_{i=1}Y^{IN\_cur}_i(mv^{can\_cur}) -Y^{OUT\_cur}_i\right\vert\right\rangle&\hbox{(5)}}$$

Note that, here the current distortion is calculated using the winning MV (mvwin), which minimizes Dsm of current concealed MB.

Due to existing of mismatch distortion in current concealed MB, the distortion may propagate to the neighboring MBs in succeeding EC process. Fig. 4 shows the scenario that the mismatch distortion in current MB propagates to its corrupted neighborhoods. Thinking about concealment of MBj which is the j-th MB next to current concealed MB, suppose the wining MV (mvwin_cur) of current concealed MB is chosen as the estimated lost MV (mvwin_nbrj) for the neighbor MBj, then the distortion of current MB should propagate to MBj. Therefore the propagated distortion can be formulated by calculating Dsm in MBj, shown in Eq. (6).Formula TeX Source $$\eqalignno{&D_{sm\_nbr} = \cases{{\rm if}\ mv_j^{win\_nbr}==mv^{win\_cur},\cr\qquad\quad {1\over M * N}\sum^M_{j=1}\displaystyle\sum^N_{i=1}\left\vert Y^{IN\_nbr}_{i,j}\left(mv_j^{win\_nbr}\right)-Y^{OUT\_nbr}_{i,j}\right\vert;\cr else,\cr\qquad\quad 0}\cr&{\rm where},\cr&mv^{win\_nbr}_j = \mathop{\rm Arg\,Min}_{mv^{can\_nbr} \in \{mv^{top},mv^{bot},mv^{lft},mv^{rt}\}}D^j_{sm\_nbr}&\hbox{(6)}}$$

YIN_nbri,j(mvjwin_nbr) and YOUT_nbri,j are shown in Fig. 4. N is the number of calculated pixels in MBj, M is the number of all neighboring MBs which are not concealed yet. Take Fig. 4 as an example, M is 2, which is the number of blue MBs.

Figure 4
Fig. 4. Illustration of Dsm_nxt calculation.

Given Eq. (5) and (6), the priority in Eq. (4) can be finally calculated.

SECTION V

Experiments

The proposed algorithm is evaluated based on the H.264 codec under the specification of 1Seg application. The JM9.1 reference software is used in the experiment, which is modified to support the packet loss in 1Seg application, while traditional JM can only support slice loss. Note that here the slice normally is divided into several TS packets. 300 frames of foreman, 90 frames for stefan, and 250 frames for flower sequences in QVGA format are encoded. I frame is encoded every 30 frames and no B frame is used. Slice mode and FMO mode are disabled. No transmission errors occur in I frames. For each P frame, a number of packets are randomly destroyed to simulate the transmission under the error rate of 10%, 20%.

As the objective comparison shown in Table II, the average PSNR of all decoded frames using 3 different methods are presented, i.e., fixed concealment order of JM suggested by H.264, adaptive order based on [10], and adaptive order based on proposal. Although 1Seg is not mentioned in [10], but their algorithm can still be easily used in 1Seg application, since order problem is common. Experiments show our algorithm can provide higher PSNR performance compared with [10].

Table 2
TABLE II Average PSNR Comparision

The subjective comparison is shown in Figs. 5 and 6. Frame #67 for stefan and frame #189 for foreman are presented respectively. There are still some mismatches in method of [10], and proposed method can avoid them efficiently. Note that for Stefan, we marked the mismatched area with circle in (c) and (d). Also note that for the sky area of foreman which is smooth, the artifacts are obvious using method in [10], while almost avoided using proposed method.

Figure 5
Fig. 5. Frame #67 in stefan with error rate of 20%. (a) Original, (b) NoEC, 8.49 dB, (c) Fixed order, 27.60 dB, (d) Ref [10], 28.22 dB, (e) Proposed, 29.72 dB.
Figure 6
Fig. 6. Frame #189 in foreman with error rate of 10%. (a) Original, (b) NoEC, 3.10 dB, (c) Fixed order, 15.75 dB, (d) Ref [10], 16.02 dB, (e) Proposed, 19.097 dB.
SECTION VI

Conclusion

This paper proposed an adaptive concealment order based on well-known boundary matching algorithm. The concealment order is carefully chosen according to a lost MB's priority, which is formulated with side match distortion. For side match distortion formulation, not only the distortion for current concealed MB, but also the propagated distortion for the neighboring lost MB are included. Compared with reference work [10], the experiments under 1Seg application show our proposal achieves better performance of video recovery under different error rate channel.

Acknowledgment

This work was supported by “Global COE Program of Waseda Univ.” of MEXT, Japan, and CREST of JST, Japan.

Footnotes

Jun Wang, Yichun Tang, and Satoshi Goto are with the Graduate School of Information, Production and Systems, Waseda University, Japan.

Shen Li and Shunichi Ishiwata are with the Center for Semiconductor Research and Development, Toshiba, Japan.

References

1. Error concealment techniques for digital TV

J.-W. Suh, et al.

IEEE Trans. Broadcasting, Vol. 48, pp. 209–306, 2002-12

2. Error control and concealment for video communication: A review

Y. Wang, et al.

Proc. of IEEE, pp. 947–997, 1998-05

3. Recovery of lost or erroneously received motion vectors

W. M. Lam, et al.

Proceeding of ICASSP, Vol. 5, pp. 417–420, 1993

5. Non-normative error concealment algorithms

V., Varsa, ITU-TVCEG-N62, 2001

6. The error concealment feature in the H.26L test model

Y. K., Wang, Proc. ICIP, 2002-09, 2, pp. 729–732

7. H.264 video communication based refined error concealment schemes

Y. Xu, Y. Zhou

IEEE Trans. Consum. Electron., vol. 50, issue (4), p. 1135–1141, 2004-11

8. Refined boundary matching algorithm for temporal error concealment

T. Chen

Proc. Packet Video, 2002, 875–887

9. An efficient error concealment implementation for MPEG4 video streams

T. S. Valente, C. Dufour, F. Groliere, D. Snook

IEEE Trans. Consum. Electron., vol. 47, issue (3), p. 568–578, 2001-08

10. Enhanced temporal error concealment algorithm with edge-sensitive processing order

T., Wu, Proc. ISCAS, 2008-06, pp. 3466–3469

11. Data Coding and Transmission Specification for Digital Broadcasting—ARIB STANDARD, ARIB STD-B24 V.5.1

Vol. 1, (English Translation)

Authors

No Photo Available

Jun Wang

No Bio Available
No Photo Available

Yichun Tang

No Bio Available
No Photo Available

Shen Li

No Bio Available
No Photo Available

Shunichi Ishiwata

No Bio Available
No Photo Available

Satoshi Goto

No Bio Available

Cited By

No Citations Available

Keywords

INSPEC: Non-Controlled Indexing

No Keywords Available

Authors Keywords

No Keywords Available

More Keywords

No Keywords Available

Corrections

No Corrections

Media

No Content Available

Indexed by Inspec

© Copyright 2011 IEEE – All Rights Reserved