image sequence coding by split and merge patrice willemin, todd r. reed and murat kunt presented by:...

Image Sequence Codingby Split and Merge

Patrice Willemin, Todd R. Reed and Murat Kunt

Presented by: Idan Shatz

Outline

• Split and Merge Coding– For Still Images

• Split and Merge Coding– For Image Sequence

• Conclusions

Image Coding

• First Generation– Pels coding (prediction or transform)

• Second Generation– Object Based coding– Region based coding

• Split and Merge Method

Split & Merge

• Split…

• Recursive algorithm– Approximate a region by a 2D polynomial– If (approximation error > threshold)

• Split the region into 4 sub-regions• Approximate each sub-region

Split Example

Original Image Split, depth=1

Split Example

Original Image Split, depth=2

Split Example

Original Image Split, depth=3

Split Example

Original Image Split, depth=4

Split Example

Original Image Split, depth=5

Split Example

Original Image Split, depth=6

Split Example

Original Image Split, depth=7

Split Example

Original Image Split, Final resultsThreshold=100,000

The threshold100,000 50,000 20,000 10,000 5,000

2,000 1,000 500 200 100

5,000

100,000

100

102

104

106

108

0

500

1000

1500

2000

2500

3000

3500

4000

4500

# of

reg

ions

Threshold

The threshold

Split & Merge

• An improvement of the Split algorithm– At each step decide between x-split and y-split

• produces fewer regions

Split Example (2)

Original Image Split, depth=1

Split Example (2)

Original Image Split, depth=2

Split Example (2)

Original Image Split, depth=3

Split Example (2)

Original Image Split, depth=4

Split Example (2)

Original Image Split, depth=5

Split Example (2)

Original Image Split, depth=6

Split Example (2)

Original Image Split, depth=7

Split Example (2)

Original Image Split, depth=8

Split Example (2)

Original Image Split, depth=9

Split Example (2)

Original Image Split, depth=10

Split Example (2)

Original Image Split, depth=11

Split Example (2)

Original Image Split, depth=12

Split Example (2)

Original Image Split, depth=13

Split Example (2)

Original Image Split, Final resultsThreshold = 100,000

Split Example (Comparison)

Split – 4 sub regionsThreshold = 100,000# regions = 613

Split – 2 sub regionsThreshold = 100,000# regions = 387

Comparison

100

102

104

106

108

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Threshold

# of

reg

ions

4-split2-split

Image Coding

• Each region is codes by:– The Border– 2D polynomial coefficients

The borders of the regions

• BSP-Tree– Binary space partition tree– Tree Nodes:

• “X” – mark an X-split• “Y” – mark an Y-split• “0” – mark an Region

Regions Borders

• Example of BSP-Tree

Y

X 0

0 Y

X

0 0

0

Storing a BSP Tree

• N regions 2N-1 BSP-Nodes

• Storing the tree in preorder

Y

X 0

0 Y

X

0 0

0

Y X Y X 00 0 0 0

Image Coding

• For N regions– Border (BSP) – N*3[bit]– 2D polynomial coefficients – N*24[bit]

• 8[bit] x 3 – gray level.• 8[bit] x 3 x 3 - for RGB.

Image Coding

• Without Compression– 196,608[Byte]

• With Compression– # regions = 387– 3628[Byte]

• Compression Raito:– 1 to 54

Split – 2 sub regions

Split & Merge

• Merge…– For each region:

• 3[bit] border << 54[bit] Coefficients

• Reduce the number of regions– Merging neighbor regions with similar

coefficients.

Merge Example (Comparison)

Split – 4 sub regionsThreshold = 100,000# regions = 613After Merge# region = 411

Split – 2 sub regionsThreshold = 100,000# regions = 387After Merge# regions = 348

Comparison

100

101

102

103

104

105

106

107

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Threshold

# of

reg

ions

split4split2split4 + mergesplit2 + merge

The borders of the regions

• The borders are more complex– Simple BSP Tree is not enough

• Therefore– BSP Tree– Regions labels

Regions Labeling

• How many bits are needed for labeling?– ~1000 regions ~10[bit]

• Non-neighbor regions can have the same labeling– ~3[bit] for labeling– Does not depend on the

number of regions

Image Coding

• For N regions made from M BSP-regions– Border (BSP) – M*3[bit]– Labeling – M*3[bit]– 2D polynomial coefficients – N*24[bit]

• 8[bit] x 3 – gray level.• 8[bit] x 3 x 3 - for RGB.

Image Coding

• Without Compression– 196,608[Byte]

• With Compression– # BSP regions = 387– # regions = 348– 3446[Byte]

• Compression Raito:– 1 to 57 (>54)

Split – 2 sub regionsAnd merging

• Same Idea.– Still Image (x,y)– Video (x,y,t)

• Split Stage– Decide between x-split, y-split and t-split.

• Merge Stage– Same

Image Sequence Coding

y

x

t

Real Time Transmission

• Compress all movie at once– Achieving best compression.– However

• can’t be transmitted on-line• Very high complexity

• Deal with chunks of 5~25 frames– Compression ratio decreases– Can be transmitted on-line

Conclusions

• Still Images compression– Achieves good results with good compression

ratio.– 2 major defects due to region boundaries:

• Disappearance of important boundaries• Border artifacts between regions.

– Merge Regions• using the same threshold has little effect on performance• adaptive threshold for achieving a giving compression

ratio

Conclusions

• Image Sequence compression– Achieves good results

• At the article: 1 to 200 comp. ratio.

– But, compression ratio decreases on high movement scenes.

• compression ratio still images.• Merging regions can be used to balance the buffer size.

– Hard to decode at real-time.• Must be coded at frame chunks.• Decoding process is hard to implement in real time

environment.

Go To…• Still Images

– Split• to 4 sub-regions• to 2 sub-regions• Threshold (2)• Compression

– Merge• Threshold• Compression

• Image Sequence• Conclusions