Slide 1

Speed up in feedback channel for a LDPCA base distributed video coding system on mobile device Chen,chun-yuan Advisor: Prof. Wu, Ja-Ling 2012/5/18 Slide 2 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 3 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 4 Digital Video Service Video compression is an essential component of broadcast and entertainment media Multimedia video everywhere! mobile video conference mobile cameras phone Video surveillance Wireless sensor network Slide 5 Slide 6 Emerging application mobile cameras phone Wireless sensor network Video surveillance mobile video conference Requiring low complexity and power-efficient encoder Slide 7 Emerging application Conventional video coding (e.g. H.264/AVC, MPEG-2) - I nherent high complexity encoder, low complexity decoder Requiring low complexity and power-efficient encoder Distributed video coding (DVC) - New video coding paradigm shifts complexity from encoder to decoder Slide 8 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 9 Conventional Video Codec MPEG-2, H.264, HEVC(H.265) ENCODER DECODERLightweight Heavyweight Slide 10 Distributed Video Coding (DVC) A new paradigm for video compression ENCODER DECODER Lightweight Heavyweight Slide 11 Application of DVC Video conferencing with mobile devices DVC to H.264 Transcoder Cloud Computational Resource DVC encoder (Low Complexity) H.264 decoder (Low Complexity) DVC encoded bitstream H.264 encoded bitstream Slide 12 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 13 Distributed Video Coding R Y =H (Y) Source encoder Correlation is exploited by motion estimation R Y =H (X|Y) Source encoder H (X|Y)H (Y) -------Distributed------- Slepian-Wolf Theorem (1973) Wyner-Ziv Theorem (1976) Channel encoder Source decoder Side Information creation Channel decoder X Y Virtual channel H (X, Y) Slide 14 Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, Rate-Adaptive Codes for Distributed Source Coding,EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder Slide 15 Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, Rate-Adaptive Codes for Distributed Source Coding,EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder Key frame Key frame WZ frame WZ frame WZ frame GOP size 4 Key frame Key frame WZ frame GOP size 2 Slide 16 Quantization DCT coefficients bands DCT coefficient band b1 : { S 1 1, S 2 1, S 3 1, S N 1 } DCT coefficient band b2 : { S 1 2, S 2 2, S 3 2, S N 2 } DCT coefficient band b16 : { S 1 16, S 2 16, S 3 16, S N 16 } DC band AC bands Block1 S11S11 S12S12 S16S16 S17S17 S13S13 S15S15 S18S18 S 1 13 S14S14 S19S19 S 1 12 S 1 14 S 1 10 S 1 11 S 1 15 S 1 16 Block2 S21S21 S22S22 S26S26 S27S27 S23S23 S25S25 S28S28 S 2 13 S24S24 S29S29 S 2 12 S 2 14 S 2 10 S 2 11 S 2 15 S 2 16 Block3 S31S31 S32S32 S36S36 S37S37 S33S33 S35S35 S38S38 S 3 13 S34S34 S39S39 S 3 12 S 3 14 S 3 10 S 3 11 S 3 15 S 3 16 Slide 17 Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, Rate-Adaptive Codes for Distributed Source Coding,EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder Slide 18 Quantization Q8 2^7 2^6 2^5 2^4 2^3 2^2 63bits for one block Q8 2^52^4 2^3 2^2 2^32^2 2^32^2 37bits for one block Slide 19 Bit plane Extraction 0010000001 00000 11110 Bit planes of DC band: Bit plane 1: Bit plane 2: Bit plane 3: Bit plane 4: Bit plane 5: Independently Channel Encode (LDPCA) 46 7 06 3 1 7 7 30 1 5 For each DCT coefficient band MSB LSB Zig-zag order Slide 20 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 21 LDPC decoding Sum-Product Algorithm (Message Passing) Side Information (real number) + 0 - 1 467 3521 decode output hard decision a b c d e f g a 25 b 25 c 25 d 25 e 25 f 25 g 25 Vertical processing Horizontal processing a 1 b 1 c 1 d 1 e 1 f 1 g 1 1 2 3 4 5 6 7 011011 011 From DVC encoder (syndrome bits) a b c d e f g Kschischang, F.R., Frey, B.J., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm. IEEE Trans. Inform. Theory Slide 22 LDPC decoding Sum-Product Algorithm (Message Passing) Side Information (real number) + 0 - 1 467 3521 decode output hard decision a b c d e f g a 25 b 25 c 25 d 25 e 25 f 25 g 25 Vertical processing Horizontal processing a 1 b 1 c 1 d 1 e 1 f 1 g 1 011 From DVC encoder (syndrome bits) Kschischang, F.R., Frey, B.J., and Loeliger, H.-A. 2001. Factor graphs and the sum-product algorithm. IEEE Trans. Inform. Theory Slide 23 Sum-Product Algorithm Vertical Processing ABCDE F GIHJ KLOMN 011011 Z P a b c de f g P = K + F + a Z = F + P + a Slide 24 Sum-Product Algorithm Horizontal Processing 011011 PQRST UVXWY ZADBC H a b c d e f g K Slide 25 LDPC Accumulate (LDPCA) codes Rate adaptivity D. Varodayan et al., "Rate-adaptive codes for distributed source coding," EURASIP Signal Processing Journal, Special Section on Distributed Source Coding, 2006 Slide 26 65 LDPC codes 1584 (Side information) 48~1584 (from encoder) 4752 edges 1584 3 99 Slide 27 65 LDPC codes 3 Slide 28 48bits parity bits 1584bits 48 24 LdpcaEncode parity Bits bitplane ! Buffer bitplanes 24 Slide 29 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 30 Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, Rate-Adaptive Codes for Distributed Source Coding,EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder Slide 31 Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, Rate-Adaptive Codes for Distributed Source Coding,EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Encoder Channel Decoder LDPC Encoder LDPC Decoder Slide 32 Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, Rate-Adaptive Codes for Distributed Source Coding,EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Decoder LDPC Decoder Key frame request WZ Bit stream 1th bitstream&CRC 63th bitstream&CRC Slide 33 Distributed Video Coding D. Varodayan, A. Aaron, and B. Girod, Rate-Adaptive Codes for Distributed Source Coding,EURASIP Signal Processing Journal, Special Issue on Distributed Source Coding,,November 2006. Channel Decoder LDPC Decoder Key frame request WZ Bit stream 1th bitstream&CRC 63th bitstream&CRC Slide 34 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 35 Sequence nameSI genLDPCA DecFeedback channelEtc. foreman9.3123.3283.098.88 soccer6.0495.31313.678.68 coastguard14.7598.06310.5915.46 hall3.87109.89348.565.25 Slide 36 Time ratio Slide 37 Amdahl's law Slide 38 Maximum speedup can be reached by improving the most critical part of the system Feedback channel in the DVC coding. 29% 66.6% Slide 39 Amdahl's law Maximum speedup can be reached by improving the most critical part of the system Feedback channel in the DVC coding. 95.6% Slide 40 Amdahl's law Maximum speedup can be reached by improving the most critical part of the system Feedback channel in the DVC coding. estimate syndromes size per bitplane 92.8% 1.7 X Slide 41 Amdahl's law Maximum speedup can be reached by improving the most critical part of the system Feedback channel in the DVC coding. estimate syndromes size per WZ frame 82.8% 4.6 X Slide 42 Amdahl's law Maximum speedup can be reached by improving the most critical part of the system LDPC decoding in the DVC decoder. 86%~94% 29%~36% QCIF Slide 43 Methods to speed up at feedback channel So we propose two methods to decrease time consuming at feedback channel. Estimate the syndromes size per WZ frame Estimate the syndromes size per bitplane Slide 44 Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number Slide 45 Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number Slide 46 The number of errors increases as the bitplanes signicance becomes lower since the correlation between the corresponding bitplanes of corresponding DCT bands of the side information and the WZ frames becomes weaker. Slide 47 Moreover, the lower is the correlation, the higher the amount of parity bits necessary for successful decoding. For example, for the Soccer sequence, the number of decoder requests is higher when compared to the Coast Guard sequence since each bitplane of the side information has a higher number of errors when compared to the Soccer sequence due to the lower quality side information. Slide 48 The highest number of requests for the highest number of errors happens for the last bitplane of the DC band in the Soccer sequence, where about 20 requests are needed to correct about 500 errors (32% of errors). This is expected since the correla- tion between the original frames and the correspond- ing side information is lower for the least signicant bitplanes of the lower frequency coefcients; note that these coefcients have more bitplanes to code due to the smaller quantization bin size. Slide 49 Estimate by previous bands 1.formula 2.time ratio Slide 50 Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number Slide 51 Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number Slide 52 Number of requests per bitplane per band DC AC1AC2 AC3 AC4AC5 AC6AC7AC8 AC9AC10 AC11 AC12 AC13 AC14 Bitplane Number Slide 53 3.5 DC AC1AC2 AC3 AC4AC5AC6 AC7AC8AC9 AC10AC11AC12AC13AC14 Bitplane Number Slide 54 Slide 55 formula (ES acn bt ) WZn = ( S dc bt-1 ) WZn WZn means the nth WZ frame. acn means the nth AC band. bt means the bitplane index of this AC band. Which ES acn bt mean the estimated syndrome size for the btth bitplane in nth AC band. dc means the DC band. S dc bt-1 means the syndromes bits for bt-1th bitplane in DC band. Slide 56 formula (ES acn bt ) WZn = ( S dc bt-1 ) WZn P(acn) WZn means the nth WZ frame. acn means the nth AC band. bt means the bitplane index of this AC band. Which ES acn bt mean the estimated syndrome size for the btth bitplane in nth AC band. dc means the DC band. S dc bt-1 means the syndromes bits for bt-1th bitplane in DC band. P(acn) is the penalty function depends on correlation between DC band and the correspond AC band Slide 57 Speed up ratio 406.39->237.87=1.708 times Slide 58 Amdahl's law Maximum speedup can be reached by improving the most critical part of the system Feedback channel in the DVC coding. 95.6% Slide 59 Amdahl's law Maximum speedup can be reached by improving the most critical part of the system Feedback channel in the DVC coding. 92.8% 1.7 X Slide 60 Estimate by reference frame 1.formula 2.time ratio Slide 61 formula (ES acn bt ) WZn = ( S dc bt-1 ) WZn WZn means the nth WZ frame. acn means the nth AC band. bt means the bitplane index of this AC band. Which ES acn bt mean the estimated syndrome size for the btth bitplane in nth AC band. dc means the DC band. S dc bt-1 means the syndromes bits for bt-1th bitplane in DC band. Slide 62 formula (ES bt ) WZn =(S bt ) WZ(n-GOPsize) WZn means the WZ frames index n-GOPsize means the correspond WZ frame in the previous GOP. bt is the bitplane index. (ES bt ) WZn means the estimated syndromes bit for the btth bitplane of WZn frame. (S bt ) WZ(n-GOPsize) means the syndromes Slide 63 formula (ES bt ) WZn =(S bt ) WZ(n-GOPsize) -P(Residual GOPnum, Residual GOPnum-1 ) WZn means the WZ frames index n-GOPsize means the correspond WZ frame in the previous GOP. bt is the bitplane index. (ES bt ) WZn means the estimated syndromes bit for the btth bitplane of WZn frame. (S bt ) WZ(n-GOPsize) means the syndromes Slide 64 Formula(cont.) P(Residual GOPnum, Residual GOPnum-1 ) Mean the penalty function depends on the residual of this and previous GOP, which Residual is evaluated by the residual value of the middle frame of one GOP. Slide 65 Time ratio 406.39->70.36=5.77 times Slide 66 Amdahl's law Maximum speedup can be reached by improving the most critical part of the system Feedback channel in the DVC coding. 95.6% Slide 67 Amdahl's law Maximum speedup can be reached by improving the most critical part of the system Feedback channel in the DVC coding. 82.8% 4.6 X Slide 68 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 69 Test condition 12 CPU, 24 processor Intel(R) Xeon(R) CPU X5650 @ 2.67GHz GPU: Tesla M2050 Mobile device: HTC sensation Slide 70 DVC encoder on mobile 14 s for foreman sequence, GOP8, intra mode on. 4 s for foreman sequence, GOP8, intra mode off. Slide 71 Test sequences : QCIF, 15Hz, all frames GOP Size 2, 4 and 8 Only luminance component is used Test materials SoccerForemanCoastguardHall Monitor Motion: High Low Slide 72 Speed up Slide 73 Bitrate Slide 74 RD curve Foreman with LRSS,GOP8 Slide 75 RD curve Hall monitor with LRSS,GOP8 Slide 76 Ref motion Adjust the estimated syndromes bit depends on motion between KEY frames. It will not cause a lot computing time and network loading. Slide 77 Slide 78 outline Motivation and introduction Mobile video trans Traditional video codec Distributed video codec DVC architecture Channel coding Ldpca The video communication system on Mobile device Speed up DVC decoding time in feedback channel Experiment Result Conclusion Slide 79 conclusion A practical DVC coding system is built on mobile devices. DVC encoder and H.264 decoder keep the mobiles light weight. We propose two method to reduce transmission time spent in feedback channel. Estimate the syndromes size per WZ frame Estimate the syndromes size per bitplane Slide 80 + Slide 81 4 DC AC1AC2 AC3 AC4AC5AC6 AC7AC8AC9 AC10AC11AC12AC13AC14 Bitplane Number Slide 82 Slide 83 Thank you