使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術
DESCRIPTION
Progressive Side Information Refinement with Non-Local Means Denoising in Distributed Video Coding. Wang , Pin-Hsiang 王品翔 Advisor: Prof. Wu, Ja -Ling 吳家麟 教授 2011/10/13. 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術. CMLab , CSIE, NTU. Outline. Introduction and Motivation DVC Architecture Overview - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/1.jpg)
Progressive Side Information Refinement with Non-Local Means
Denoising in Distributed Video Coding
使用於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術Wang, Pin-Hsiang 王品翔
Advisor: Prof. Wu, Ja-Ling 吳家麟 教授2011/10/13
CMLab, CSIE, NTU
![Page 2: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/2.jpg)
Introduction and Motivation DVC Architecture Overview Proposed Side Information Refinement
Framework Experimental Result Conclusions and Future Work
Outline2
CMLab, CSIE, NTU
![Page 3: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/3.jpg)
Emerging application
CMLab, CSIE, NTU
3
mobile cameras phone
Wireless sensor network
Video surveillance
mobile video conference
Requiring low complexity and power-efficient encoder…
![Page 4: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/4.jpg)
4
CMLab, CSIE, NTU
Emerging application
Conventional video coding (e.g. H.264/AVC, MPEG-2)- Inherent high complexity encoder, low complexity decoder
Requiring low complexity and power-efficient encoder…
Distributed video coding (DVC)- New video coding paradigm shifts complexity from encoder to decoder
![Page 5: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/5.jpg)
Application of DVC5
DVC to H.264 Transcoder
CloudComputational Resource
DVC encoder(Low Complexity)
H.264 decoder(Low Complexity)
DVC encoded bitstream
H.264 encoded bitstream
CMLab, CSIE, NTU
Make the Clients slimmer & thinner
![Page 6: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/6.jpg)
Distributed Video Coding
𝑺𝒐𝒖𝒓𝒄𝒆 𝑿
𝑺𝒐𝒖𝒓𝒄𝒆𝒀
Joint Encoder
Joint Decoder
𝑿𝒀
𝑹𝑿+𝑹𝒀 ≥𝑯 (𝑿 ,𝒀 ) Statistical dependency
𝑺𝒐𝒖𝒓𝒄𝒆 𝑿
𝑺𝒐𝒖𝒓𝒄𝒆𝒀
Encoder X
Joint Decoder
𝑿𝒀
𝑹𝑿+𝑹𝒀 ≥𝑯 (𝑿 ,𝒀 ) Statistical dependency is not exploited
Encoder Y
Conventional video coding paradigm
Slepian-Wolf theorem
CMLab, CSIE, NTU
Slepian-Wolf Theorem (1973, Lossless coding)
Wyner-Ziv Theorem (1976, Lossy coding)
6
![Page 7: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/7.jpg)
Distributed Video Coding7
SourceEncoder
SourceEncoder
Quantizer𝑺𝒐𝒖𝒓𝒄𝒆 𝑿
𝑺𝒐𝒖𝒓𝒄𝒆𝒀
SourceEncoder 𝑿
𝒀
Statistical dependency is not exploited
Quantizer
SourceEncoder
Correlation is exploited at the decoder side
Joint DecoderEncoder X
Encoder Y
Parity bitsChannelEncoder
ChannelDecoder
side information (SI)
Side Information Estimation
𝒀
Virtual channel
CMLab, CSIE, NTU
DVC is also called Wyner-Ziv video coding (WZVC)
Corrupted version of X
Wyner-Ziv Theorem (1976, Lossy coding)
Quality
![Page 8: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/8.jpg)
Motivation8
CMLab, CSIE, NTU
Past reference frame Future reference frame
Source XEncoder-side
Decoder-side
F( t-1 ) F( t+1 )F( t )
Side Information Estimation
F( t )
![Page 9: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/9.jpg)
Motivation9
CMLab, CSIE, NTU
Past reference frame Future reference frame
Source XEncoder-side
Decoder-side
F( t-1 ) F( t+1 )F( t )
Frame interpolation (Decoder-side ME)
F( t )
![Page 10: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/10.jpg)
Motivation10
CMLab, CSIE, NTU
Past reference frame Future reference frame
Source XEncoder-side
Decoder-side
F( t-1 ) F( t+1 )F( t )
Side information
F( t )
![Page 11: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/11.jpg)
Motivation11
CMLab, CSIE, NTU
Past reference frame Future reference frame
Source XEncoder-side
Decoder-side
F( t-1 ) F( t+1 )F( t )
Side informationLimitation
F( t )
Most reported WZ codecs have a poor RD performance for high motion and large GOP size sequences
![Page 12: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/12.jpg)
Non-Local Means Side Information Refinement framework (NLM-SIR) for DVC is proposed
NLM-SIR framework for DVC
12
CMLab, CSIE, NTU
Improve the SI quality to better rate-distortion (RD) performance of WZVC
Overcoming some of the limitations about current SI estimation methods in WZVC
![Page 13: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/13.jpg)
Introduction and Motivation DVC Architecture Overview Proposed Side Information Refinement
Framework Experimental Result Conclusions and Future Work
Outline13
CMLab, CSIE, NTU
![Page 14: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/14.jpg)
Reference codec : DISCOVER codec (Distributed coding for video
services) X. Artigas et al., “The DISCOVER codec: architecture,
techniques and evaluation”, PCS, 2007 Feedback channel based transform domain WZ
codec
DVC Architecture Overview14
CMLab, CSIE, NTU
![Page 15: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/15.jpg)
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
DISCOVER Codec Architecture
Soft Input
KeyFrame
KeyFrame
WZFrame
GOP size 2
WZFrame
WZFrame
WZFrame
GOP size 4
KeyFrame
KeyFrame
Ref. X. Artigas et al., PCS, 2007
15
![Page 16: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/16.jpg)
Quantization16
DCT coefficients bands
DCT coefficient band b1 : { S11, S2
1, S31, …SN
1 }
DCT coefficient band b2 : { S12, S2
2, S32, …SN
2 }
DCT coefficient band b16 : { S116, S2
16, S316, …SN
16 }
…DC band
AC bands
Block1
S11 S1
2 S16 S1
7
S13 S1
5 S18 S1
13
S14 S1
9 S112 S1
14
S110 S1
11 S115 S1
16
Block2
S21 S2
2 S26 S2
7
S23 S2
5 S28 S2
13
S24 S2
9 S212 S2
14
S210 S2
11 S215 S2
16
Block3
S31 S3
2 S36 S3
7
S33 S3
5 S38 S3
13
S34 S3
9 S312 S3
14
S310 S3
11 S315 S3
16
…
![Page 17: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/17.jpg)
DISCOVER Codec Architecture
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Soft Input
Ref. X. Artigas et al., PCS, 2007
17
![Page 18: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/18.jpg)
Bit plane Extraction18
0 0 1 0 0 0 0 0 0 1
0 0 0 0 0 1 1 1 1 0
Bit planes of DC band:
Bit plane 1:
Bit plane 2:
Bit plane 3:
Bit plane 4:
Bit plane 5:
Independently Channel Encode (LDPCA)
4 6
7
0 6
3
1 7
7
30 1
5
For each DCT coefficient band…
MSB
LSBZig-zag order
![Page 19: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/19.jpg)
DISCOVER Codec Architecture
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Soft Input
Ref. X. Artigas et al., PCS, 2007
19
![Page 20: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/20.jpg)
DISCOVER Codec Architecture
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Soft Input
Forward motion estimation
Ref. X. Artigas et al., PCS, 2007
20
![Page 21: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/21.jpg)
DISCOVER Codec Architecture
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Soft Input
Bidirectional motion estimation & compensation
Ref. X. Artigas et al., PCS, 2007
21
![Page 22: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/22.jpg)
DISCOVER Codec Architecture
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Soft Input
Ref. X. Artigas et al., PCS, 2007
22
![Page 23: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/23.jpg)
DISCOVER Codec Architecture
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Soft Input
Ref. X. Artigas et al., PCS, 2007
Laplacian distribution
23
![Page 24: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/24.jpg)
DISCOVER Codec Architecture
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Soft InputConditional bit probabilities
Ref. X. Artigas et al., PCS, 2007
Iterative decoding (band by band, bitplane by bitplane)
24
![Page 25: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/25.jpg)
𝑧𝑖+1
Reconstruction25
CMLab, CSIE, NTU
: Side Information
: Reconstructed value
𝑧𝑖
Quantization Interval
Boundary reconstruction method :
Case1 Case2 Case3
![Page 26: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/26.jpg)
DISCOVER Codec Architecture
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Soft Input
Poor RD performance for high motion and large GOP size sequences
Room for improvement
Ref. X. Artigas et al., PCS, 2007
26
![Page 27: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/27.jpg)
Introduction and Motivation DVC Architecture Overview Proposed Side Information Refinement
Framework Experimental Result Conclusions and Future Work
Outline27
CMLab, CSIE, NTU
![Page 28: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/28.jpg)
NLM-SIR framework28
Original XWZ frame (at the encoder)
Side information YWZ (at the decoder)
Correlation noise N between Original XWZ frame and side information
Observed model : Reduce noise to achieve better SI quality Denoising problem
![Page 29: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/29.jpg)
NLM-SIR framework29
Iterative decodingIterative decoding (band by band)
New information about original WZ frame is not exploited Progressive available during decoding Not available at the time the initial SI was estimated
![Page 30: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/30.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
30
![Page 31: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/31.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
Initial SI (YWZ) is always used to decoding the DC band
DC band decoding
31
![Page 32: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/32.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
AC bands decoding
32
![Page 33: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/33.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
Other decoding iterations
NLM-SIR
Partially decoded WZ frame
Refined Side Information
33
![Page 34: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/34.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
Other decoding iterations
Decoded coefficients
Copy from Initial SI coefficients
34
![Page 35: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/35.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
Other decoding iterations
35
![Page 36: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/36.jpg)
𝐸𝑛=∑𝑥=0
3
∑𝑦=0
3
(𝑌 𝑛 [ 𝑥 , 𝑦 ]−𝑃𝑛 [ 𝑥 , 𝑦 ] )2
𝐸𝑛≥ h h𝑇 𝑟𝑒𝑠 𝑜𝑙𝑑𝐸𝑛< h h𝑇 𝑟𝑒𝑠 𝑜𝑙𝑑Fine SI blocks :
Noisy SI blocks :
Candidate Block Selection
(1) Noise Computation :
(2) Block Selection for Refinement:
Noise indicator
Selection of the SI blocks which are worthwhile of refining Initial guess by the side information creation
process has basically failed
36
![Page 37: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/37.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
Other decoding iterations
37
![Page 38: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/38.jpg)
Non-Local Means Images possess repeating elements and
patches
38
![Page 39: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/39.jpg)
Non-Local Means Refinement
CMLab, CSIE, NTU
39
Partially decoded frame PWZ
𝐷(𝑥 , 𝑦 ) [𝑚 ,𝑛 ]= ∑𝜇 ,𝑣=¿𝑑𝑚 ,… ,−𝑑𝑚
(𝑃 [ 𝑥+𝜇 , 𝑦+𝑣 ]−𝑃 [𝑚+𝜇 ,𝑛+𝑣 ] )2
𝑌 𝑤𝑧 [ 𝑥 , 𝑦 ]= ∑[𝑚 ,𝑛 ]∈𝑆
𝑊 (𝑥 , 𝑦 ) [𝑚 ,𝑛 ] ∙𝑃 [𝑚 ,𝑛 ]
NLM algorithm :
Similarity measurement
![Page 40: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/40.jpg)
Non-Local Means Refinement
40
Smoothing parameter h :
Partially decoded frame PWZ
𝑊 (𝑥 ,𝑦 ) [𝑚 ,𝑛 ]= 1𝑁 [ 𝑥 , 𝑦 ]
∙𝑒𝑥𝑝(− 𝐷(𝑥 ,𝑦 ) [𝑚 ,𝑛 ]h2 )
Normalized term :
Weight assignment :
![Page 41: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/41.jpg)
Non-Local Means Refinement
41
Partially decoded frame PWZ
𝑊 (𝑥 ,𝑦 ) [𝑚 ,𝑛 ]= 1𝑁 [ 𝑥 , 𝑦 ]
∙𝑒𝑥𝑝(− 𝐷(𝑥 ,𝑦 ) [𝑚 ,𝑛 ]h2 )
Assigned the same value as themaximum of the other weightsobserved in the searching window
Weight assignment :
, Central weight :
![Page 42: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/42.jpg)
Non-Local Means Refinement
42
𝐹 (𝑡−1 ) 𝐹 (𝑡 ) 𝐹 (𝑡+1 )
Past decoded frame Future decoded framePartially decoded frame
Searching for more similar patches…
![Page 43: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/43.jpg)
Non-Local Means Refinement
43
Take into account of the temporal similar patches
𝑌 𝑤𝑧 [ 𝑥 , 𝑦 ]=∑𝑡
∑[𝑚 ,𝑛 ]∈𝑆
𝑊 (𝑥 , 𝑦 ) [𝑚 ,𝑛 ,𝑡 ] ∙𝑅𝑒𝑓 [𝑚 ,𝑛 , 𝑡 ]
𝐹 (𝑡−1 ) 𝐹 (𝑡 ) 𝐹 (𝑡+1 )
Past decoded frame Future decoded framePartially decoded frame
![Page 44: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/44.jpg)
Parameter Setting of NLM44
𝐹 (𝑡−1 ) 𝐹 (𝑡 ) 𝐹 (𝑡+1 )
Past decoded frame Future decoded framePartially decoded frame
Neighborhood size (Patch size) : Search window size :
![Page 45: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/45.jpg)
Parameter Setting of NLM
CMLab, CSIE, NTU
45
X’F
X’BMC Residue frame
Motion compensated residual frame
Correlation noise
𝑀𝐶𝑅𝐹 [ 𝑥 , 𝑦 ]=12 (𝑋 ¿¿𝑃 [𝑥+𝑚𝑣𝑥𝑃 , 𝑦+𝑚𝑣 𝑦𝑃 ]−𝑋 𝐹 [𝑥+𝑚𝑣 𝑥𝐹 , 𝑦+𝑚𝑣 𝑦𝐹 ])¿
Smoothing parameter : First iteration : Afterward iterations :
![Page 46: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/46.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
Other decoding iterationsUpdate Noise Distribution Model :
Y ’(u,v) Y(u,v)
X(u,v)
Prob
abilit
y
Coefficient Value
Refined SI coefficient
46
![Page 47: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/47.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
Other decoding iterationsFailed refinement detection
(q+1) W‧ kq‧Wk
Quantization bin
refined SIrefined SI unrefined SI
47
![Page 48: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/48.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
Progressive refinement
# of decoded bands
PSNR
, dB
SI quality of 13th frame in Foreman
48
![Page 49: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/49.jpg)
WZ Encoder WZ Decoder
Soft InputY’
WZ
YDCT
XDCT
X’F
X’P
XK
XWZ
YWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Reconstruction IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
NLM Refinement
Candidate Block
Selection
PWZ
Proposed Codec Architecture
NLM-SIR
Bitrate savings
Quality gains
49
![Page 50: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/50.jpg)
Introduction and Motivation DVC Architecture Overview Proposed Side Information Refinement
Framework Experimental Result Conclusions and Future Work
Outline50
CMLab, CSIE, NTU
![Page 51: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/51.jpg)
Test sequences :
QCIF, 15Hz, all frames GOP Size 2, 4 and 8 Only luminance component is used
Test materials
CMLab, CSIE, NTU
51
Soccer Foreman Coastguard Hall Monitor
Motion: High Low
![Page 52: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/52.jpg)
PSNR Temporal EvolutionAvg : 1.50 dBMax : 3.12 dB
Avg : 0.88 dBMax : 1.91 dB
Avg : 0.42 dBMax : 2.05 dB
Avg : 0.13 dBMax : 0.67 dB
Achieving better decoded quality, especially for high motion content zones
Scenario : GOP = 8, Q8
52
![Page 53: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/53.jpg)
Bitrate Temporal EvolutionAvg : 5.58 kbitsMax : 13.90 kbits
Avg : 4.78 kbitsMax : 9.36 kbits
Avg : 1.23 kbitsMax : 7.73 kbits
Avg : 0.32 kbitsMax : 1.15 kbits
Scenario : GOP = 8, Q8
53
![Page 54: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/54.jpg)
Overall RD Performance (GOP=8)
2.5 dB 2.0 dB
0.6 dB 0.3 dB
54
![Page 55: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/55.jpg)
Overall RD Performance (GOP=4) 55
![Page 56: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/56.jpg)
Overall RD Performance (GOP=2) 56
![Page 57: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/57.jpg)
Decoding Time Complexity57
Foreman Soccer Coastguard Hall0
2000
4000
6000
8000
10000
12000
14000
16000
10872 10772 11243
2298
1548914755
11340
2520
With NLM-SIRWithout NLM-SIR
Tota
l dec
odin
g ti
me
(Sec
.)
Q8, GOP = 8, whole sequence (150 frames)
There tests were performed on an Intel Core2 Quad Processor at 2.40 GHz with 4.0 GB of RAM (Windows 7 operating system)
Usage of better SI results in fewer rate requests and thus fewer LDPCA decoder runs
![Page 58: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/58.jpg)
Decoding Time Complexity58
Foreman Soccer Coastguard Hall0
1000
2000
3000
4000
5000
6000
7000
49565382
3836
975
5485
6457
3521
899
With NLM-SIRWithout NLM-SIR
Tota
l dec
odin
g ti
me
(Sec
.)
Q4, GOP = 8, whole sequence (150 frames)
There tests were performed on an Intel Core2 Quad Processor at 2.40 GHz with 4.0 GB of RAM (Windows 7 operating system)
![Page 59: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/59.jpg)
Highly parallelized decoder
DISPAC codec (Distributed video coding with parallelized design for cloud computing)
NLM-SIR for DISPAC Codec59
CMLab, CSIE, NTU
State-of-the-art RD performance - Effective integrate numerous advanced tools
![Page 60: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/60.jpg)
Overall RD Performance (GOP=8) 60
![Page 61: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/61.jpg)
Decoding Time Complexity61
Foreman Soccer Coastguard Hall0
2000
4000
6000
8000
10000
12000
14000
16000
10872 10772 11243
2298
315 353 338 200319 350 352 202
DISCOVERDISPACDISPAC with NLM-SIR
Tota
l dec
odin
g ti
me
(Sec
.)
Q8, GOP = 8, whole sequence (150 frames)
4core + GPU
CMLab, CSIE, NTUThere tests were performed on an Intel Core2 Quad Processor at 2.40 GHz with 4.0 GB of RAM (Windows 7 operating system)
The processing of NLM-SIR can be highly parallelized (CUDA)
![Page 62: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/62.jpg)
Introduction and Motivation DVC Architecture Overview Proposed Side Information Refinement
Framework Experimental Result Conclusions and Future Work
Outline62
CMLab, CSIE, NTU
![Page 63: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/63.jpg)
A NLM-based side information refinement (NLM-SIR) framework for WZVC is proposed
Conclusion63
CMLab, CSIE, NTU
Universally applicable in most DCT domain WZVC schemes
Provide significant RD gain over existing WZVC framework, notably for the conditions where usually WZVC performs worse
Introduce negligible overhead on the decoding time, and the processing module can be highly parallelized
![Page 64: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/64.jpg)
Future work64
CMLab, CSIE, NTU
Spatial adaptive parameter setting of NLM-SIR during the decoding
A more suitable and powerful transform-domain denoising algorithm could be considered to substitute for NLM
![Page 65: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/65.jpg)
Thank You
CMLab, CSIE, NTU
65
![Page 66: 使用 於分散式視訊編碼之非區域平均去雜訊循序旁資訊改善技術](https://reader036.vdocuments.mx/reader036/viewer/2022082215/56815b2c550346895dc8f20a/html5/thumbnails/66.jpg)
DISPAC codec with NLM-SIR
WZ Encoder
XDCT
YDCT
X’F
X’P
XK
XWZYWZ
Uniform Quantizer
DCT
LDPCA Encoder Buffer
CRC Gen
LDPCA Decoder
CRC Check
Multi-SI Reconstruction
IDCT
Correlation Noise Modeling
H.264/AVC Intra Encoder
H.264/AVC Intra Decoder
Frame Buffer
Side Information Creation
DCT
CRC-8
Feedback Channel
WZ Bitstream
Slepian-Wolf Encoder Slepian-Wolf Decoder
WZ Frames
Key Frames
Decoded WZ
Frames
Decoded Key
Frames
Bitplanes
WZ Decoder
Block Mode Selection
Soft Input
Deblocking Filter
Motion Learning SI Refinement
Non-Local Means SI Refinement
PWZY’
WZ
Y’’WZ
66