TRANSMISSION of compressed video over error prone channels such as wireless network may result in packet losses or errors in a received video stream. Such errors or losses do not only corrupt the current frame, but also propagate to the subsequent frames [1]. Several error control technologies, such as forward error correction (FEC), automatic retransmission request (ARQ) and error concealment (EC), have been proposed to solve this problem. Compared with FEC and ARQ, EC wins the favor since it doesn't need extra bandwidth and can avoid transmission delays [2].

The EC scheme attempts to recover the lost MBs by utilizing correlation from spatially or temporally adjacent macro blocks (MBs), i.e., spatial error concealment (SEC) or temporal error concealment (TEC) [2]. For TEC, which is focused on by this paper, several related works have been published to estimate the missing motion vector (MV) by using the correctly received MVs around the corrupted MB. In [3], the well-known boundary matching algorithm (BMA) is proposed to recover the MV from the candidate MVs by minimizing the side match distortion (*D*_{sm}) between the internal and external boundary of the reconstructed MB. This algorithm is adopted in H.264 reference software JM [4] and described in detail in [5], [6]. Based on BMA, several improved methods are proposed, such as [7], [8], [9].

It can be seen that, lost MV recovery is the most focus in published TEC methods, while the EC order—the decision of which MB should be concealed earlier inside the area of broken MBs—is rarely well considered. In BMA based [5], [6], the authors use the fixed concealment order, from boundary to center, column-by-column, inside a picture. In [10], the authors proposed an edge-sensitive order, which ranked each lost MB with their edge intensity, in the boundary of its available neighborhood. The edge intensity is extracted by sobel operator. This work is based on such observation that, in order to improve the image recovery, an MB which has stronger edge intensity (predominant edge is extracted) in the boundary of its available neighbors is going to be concealed earlier.

There are 2 problems in [10]'s work. Firstly, for the MBs with smooth boundary where no predominant edge may be extracted, it is hard to decide which MB should be concealed earlier, thus this algorithm sometimes is not efficient for such kind of MB's recovery. Secondly, edge property is not directly related with the priority determination. The more directly one is the side match distortion (*D*_{sm} ), which is the unique criterion for lost MV estimation in BMA based EC, see more detail in Section III. In other words, an MB with smaller *D*_{sm} is more likely to be concealed earlier.

Considering these 2 problems, in our work, *D*_{sm} is included in MB's priority formulation instead of edge intensity. Furthermore, for *D*_{sm} formulation, not only the distortion for current concealed MB, but also the potential propagated distortion for the neighboring lost MB are included.

The rest of this paper is organized as follows. Section II presents our work's motivation, which is for 1Seg application. Section III gives an overview of well-known BMA. Based on BMA, we present proposed EC order in Section IV. Finally Sections V and VI show our experiments and conclusion.

SECTION II

## 1Seg Application Based Motivation

Our work is targeted to 1seg application [11]. 1Seg is one of services of ISDB-T (Integrated Services of Digital Broadcasting- terrestrial) in Japan and Brazil. ISDB-T is designed so that each channel is divided into 13 segments. An HDTV broadcast signal occupies 12 segments, leaving the remaining (13th) segment for mobile receivers. Thus the mobile service is named with “1seg”. Due to errors drastically increased in wireless mobile terminal, EC is critical for 1Seg.

In 1Seg, H.264 format is specified for video compression. Table I shows the specification [11] for video compression and transmission in 1Seg related to this work.

According to Table I, even 1 broken packet may cause huge conjunctive area loss (from the start of broken packet to the end of the frame) inside a frame. This is because 1) slice mode and FMO mode are prohibited; 2) for QVGA sized video, each picture/slice usually consists of at least more than 1 TS packet (normally more than 3); and 3) variable length coding (VLC) based entropy coding is included in H.264. Fig. 1 shows a typical loss picture (foreman sequence) in 1Seg scenario, where green part is the destroyed area. In our experiment in Section V, we modified JM to support 1Seg.

As shown in Fig. 1, enormous areas are destroyed, therefore the concealment order in this area—to decide which MB should be concealed earlier—may highly influence the image quality of recovery. In Section IV, we will propose our solution toward this order problem.

SECTION III

## Boundary Matching Algorithm (BMA) for TEC

The TEC method in JM uses fixed EC order, from frame boundary to center column-by-column. And for each lost MB, BMA [4], [5], [6] is utilized to recover the lost MV from candidate MVs (*mv*^{can} *s*) by minimizing the side match distortion (*D*_{sm}) between the IN-MB and OUT-MB, see Fig. 2. The IN-MB is projected by *mv*^{can}, which is the candidate MV belongs to one of 4 neighbors in current frame.

The *D*_{sm} is determined by the sum of absolute differences between the predicted MB and the neighboring MBs at the boundary, shown in Eq. (1).
TeX Source
$$\eqalignno{&D_{sm} = {1\over N}\left\langle \sum^N_{i=1}\left\vert Y^{IN}_i (mv^{can}) -Y^{OUT}_i\right\vert\right\rangle\cr&where, mv^{can} \in \{mv^{top},mv^{bot},mv^{lft},mv^{rt}\}&\hbox{(1)}}$$

As shown in Fig. 2, *N* is the total number of the calculated boundary pixels, including correctly received (OK MB) and concealed boundary pixels in the neighbors. The winning prediction MV is the one which minimizes the side match distortion *D*_{sm}:
TeX Source
$$mv^{win} = \mathop{\rm Arg\,Min}_{mv^{can} \in \{mv^{top},mv^{bot},mv^{lft},mv^{rt}\}}D_{sm}\eqno{\hbox{(2)}}$$

SECTION IV

## Proposed EC Order

### A. Overview of the Proposed EC Order

Fig. 3 shows the flow chart of EC process with proposed adaptive order. Note that, the traditional BMA is performed for the MB with the highest priority in each loop. Therefore, the key step in proposed order is the priority calculation for each lost MB.

### B. Priority Formulation

It is inevitable that the current concealed MB may cause mismatch distortion compared with the original current MB. And it is also inevitable that the mismatch distortion of current MB will cause error propagation to the neighboring MB in the consequent concealment process, as the current concealed MB might be used to conceal the neighboring corrupted MBs. In BMA based TEC, the distortion can be expressed by side match distortion, since the original pixel is unavailable for a corrupted MB. Therefore, we can formulate the total side match distortion of a lost MB as follows:
TeX Source
$$D_{sm\_total} = {D_{sm\_cur} + D_{sm\_nbr} \over 2}\eqno{\hbox{(3)}}$$where, *D*_{sm_cur} and *D*_{sm_nbr} are side match distortion for current MB itself, and the potential propagated side match distortion for neighboring corrupted MBs, respectively. Division by 2 is because the distortion should be normalized by pixel.

It is easy to know that, in order to improve the concealed image quality, an MB with smaller *D*_{sm_total} should be concealed earlier compared with an MB with larger *D*_{sm_total}. In other words, the MB with smaller *D*_{sm_total} should have higher priority in EC process. That it:
TeX Source
$$priority = {1 \over D_{sm\_total}}\eqno{\hbox{(4)}}$$

In the next section, we will further formulate *D*_{sm_total} based on Eq. (3).

### C. Distortion Formulation

Similar with Eq. (1) and (2), it is easy to formulate *D*_{sm_cur} as follows:
TeX Source
$$\eqalignno{&D_{sm\_cur} = D_{sm}(mv^{can\_cur} = mv^{win\_cur})\cr&={\rm Min}_{mv^{can\_cur} \in \{mv^{top},mv^{bot},mv^{lft},mv^{rt}\}}{1\over N}\left\langle \sum^N_{i=1}\left\vert \sum^N_{i=1}Y^{IN\_cur}_i(mv^{can\_cur}) -Y^{OUT\_cur}_i\right\vert\right\rangle&\hbox{(5)}}$$

Note that, here the current distortion is calculated using the winning MV (*mv*^{win}), which minimizes *D*_{sm} of current concealed MB.

Due to existing of mismatch distortion in current concealed MB, the distortion may propagate to the neighboring MBs in succeeding EC process. Fig. 4 shows the scenario that the mismatch distortion in current MB propagates to its corrupted neighborhoods. Thinking about concealment of MB_{j} which is the j-th MB next to current concealed MB, suppose the wining MV (*mv*^{win_cur}) of current concealed MB is chosen as the estimated lost MV (*mv*^{win_nbr}_{j}) for the neighbor MB_{j}, then the distortion of current MB should propagate to MB_{j}. Therefore the propagated distortion can be formulated by calculating *D*_{sm} in MB_{j}, shown in Eq. (6).
TeX Source
$$\eqalignno{&D_{sm\_nbr} = \cases{{\rm if}\ mv_j^{win\_nbr}==mv^{win\_cur},\cr\qquad\quad {1\over M * N}\sum^M_{j=1}\displaystyle\sum^N_{i=1}\left\vert Y^{IN\_nbr}_{i,j}\left(mv_j^{win\_nbr}\right)-Y^{OUT\_nbr}_{i,j}\right\vert;\cr else,\cr\qquad\quad 0}\cr&{\rm where},\cr&mv^{win\_nbr}_j = \mathop{\rm Arg\,Min}_{mv^{can\_nbr} \in \{mv^{top},mv^{bot},mv^{lft},mv^{rt}\}}D^j_{sm\_nbr}&\hbox{(6)}}$$

*Y*^{IN_nbr}_{i,j}(*mv*_{j}^{win_nbr}) and *Y*^{OUT_nbr}_{i,j} are shown in Fig. 4. *N* is the number of calculated pixels in MB_{j}, *M* is the number of all neighboring MBs which are not concealed yet. Take Fig. 4 as an example, *M* is 2, which is the number of blue MBs.

Given Eq. (5) and (6), the priority in Eq. (4) can be finally calculated.

The proposed algorithm is evaluated based on the H.264 codec under the specification of 1Seg application. The JM9.1 reference software is used in the experiment, which is modified to support the packet loss in 1Seg application, while traditional JM can only support slice loss. Note that here the slice normally is divided into several TS packets. 300 frames of foreman, 90 frames for stefan, and 250 frames for flower sequences in QVGA format are encoded. I frame is encoded every 30 frames and no B frame is used. Slice mode and FMO mode are disabled. No transmission errors occur in I frames. For each P frame, a number of packets are randomly destroyed to simulate the transmission under the error rate of 10%, 20%.

As the objective comparison shown in Table II, the average PSNR of all decoded frames using 3 different methods are presented, i.e., fixed concealment order of JM suggested by H.264, adaptive order based on [10], and adaptive order based on proposal. Although 1Seg is not mentioned in [10], but their algorithm can still be easily used in 1Seg application, since order problem is common. Experiments show our algorithm can provide higher PSNR performance compared with [10].

The subjective comparison is shown in Figs. 5 and 6. Frame #67 for stefan and frame #189 for foreman are presented respectively. There are still some mismatches in method of [10], and proposed method can avoid them efficiently. Note that for Stefan, we marked the mismatched area with circle in (c) and (d). Also note that for the sky area of foreman which is smooth, the artifacts are obvious using method in [10], while almost avoided using proposed method.

This paper proposed an adaptive concealment order based on well-known boundary matching algorithm. The concealment order is carefully chosen according to a lost MB's priority, which is formulated with side match distortion. For side match distortion formulation, not only the distortion for current concealed MB, but also the propagated distortion for the neighboring lost MB are included. Compared with reference work [10], the experiments under 1Seg application show our proposal achieves better performance of video recovery under different error rate channel.

### Acknowledgment

This work was supported by “Global COE Program of Waseda Univ.” of MEXT, Japan, and CREST of JST, Japan.