A NOVEL MULTI-STAGE MOTION VECTOR PROCESSING METHOD FOR MOTIONCOMPENSATED FRAME INTERPOLATIONAi-Mei Huang and Truong NguyenVideo Processing Lab ECE Dept, UCSD, La Jolla, CA 92093This paper appears in:  Image Processing, 2007. ICIP 2007. IEEE International Conference on

Overview

Introduction Block diagram of the proposed

algorithms Prediction residual energy analysis The proposed multi-stage motion

vector processing method Simulation results Conclusions

Introduction

Motion-compensated frame interpolation (MCFI) improves temporal quality by increasing the frame rate at the decoder.

Frame interpolation for compressed video remains a problem due to the use of improper MVs are often generated.

The proposed algorithms preserve the object structure information but also produce a smoother motion vector field (MVF).

Block diagram of the proposed algorithms

Prediction residual energy analysis(1/5)

In [4], we have discussed that there exists a strong correlation between MV reliability and its associated residual energy.

These high residual energies regions are distributed over where object edges are located.

Let vm,n denote the MV of each 8×8 block. We classify vm,n into three different reliability levels, reliable, possibly reliable.

Prediction residual energy analysis(2/5)

For a MB (16×16) with only one MV, we simply assign the same MV to all four 8×8 blocks(bm,n):

If Em,n ≧ ε1 , it will be considered as unreliable(L1).

Consider intra-coded MBs as unreliable(L1). The neighboring of L1 MBs or MVs in the same

MB will be classified as possibly reliable (L2). Other MBs will be classified as reliable (L3).

Prediction residual energy analysis(3/5)

Motion Vector Reliability Map

Prediction residual energy analysis(4/5) Analyze the connectivity of the

unreliable MVs in MVRM and create a MB merging map. If a MB that has unreliable MVs connecting

to other unreliable MVs in vertical, horizontal or diagonal directions in adjacent MBs, these MBs will be merged.

The merging process is performed on a MB basis using MVRM, and all MBs will be examined in a raster scan order.

The 32×32 block size is the maximum for merging.

Prediction residual energy analysis(5/5)

The diagonal direction is not considered for intra-intra MB merging, because the possibility for two diagonal intra-coded MBs belonging to the same object is lower.

(c) MV reliability classification map. Unreliable and reliable MVs are marked in yellow and white colors, respectively. Intra-coded MBs are marked in cyan color.

(d) MB merging map.

Block diagram of the proposed algorithms

The proposed multi-stage motion vector processing method(1/4) Find the best MV for each merged group:

If the ABPD of v*b is less than a threshold ε2 , assign v*b to the merged MBs in Cu.

Otherwise, drop the selected MV(v*b) and wait until a proper MV propagates to its neighborhood in next iteration.

Process stops until all merged groups have been assigned new MVs.

S denotes the reliable MVs in merge group & adjacent blocks.Cu denotes the merged group.

The proposed multi-stage motion vector processing method(2/4) Reclassify MV reliability based on BPD

resulted from the selected MV. BPD(m, n) of each 8×8 block is obtained

by simply summing up difference error like Eq(1). If BPD(m, n) is higher than ε3 ,vm,n is

unreliable(L1). Otherwise, other MVs will be classified as

reliable(L2).

If the MB consists of multiple motion, those unreliable MV can be easily detected by BPD.

Block diagram of the proposed algorithms

The proposed multi-stage motion vector processing method(3/4) For those unreliable MVs of 8×8 blocks in the

updated MVRM, correct them by using a reliability and similarity constrained vector median filter:

S contains the neighboring MVs centered at vm,n

di,j denotes the distance between vi,j and vm,n

Vm,n

The proposed multi-stage motion vector processing method(3/4) For those unreliable MVs of 8×8 blocks in the

updated MVRM, correct them by using a reliability and similarity constrained vector median filter:

Two MVs are considered to be similar if the angle distance(di,j) is below a threshold, ε4.

Before updating v*m,n in MVF2, check on BPD of v∗m,n to ensure that error energy is descended.

If the energy check fail, correct it in next iteration.

S contains the neighboring MVs centered at vm,n

di,j denotes the distance between vi,j and vm,n

Block diagram of the proposed algorithms

MV smoothing process in [3] to reduce visual artifacts due to high BPD.

On the frame boundary, using unidirectional interpolation based on the directions of MVs:

The proposed multi-stage motion vector processing method(4/4)

Simulation results(1/3)

Simulation results(2/3)

Simulation results(2/3)

CONCLUSIONS

We propose a novel algorithm based on the received information for MCFI .

Accomplishing the concept of object motion without complex motion estimation.

The method outperforms other conventional methods on both objective and subjective video quality.