CrystalMobile: multimedia CrystalMobile: multimedia framework for mobile framework for mobile

platformsplatforms

Kim A. BondarenkoCrystal Reality LLC, CSA

kim@crystalreality.com

Computer Technology DepartmentSaint-Petersburg State University of

Information Technology, Mechanics and Optics,

RussiaRussia

www.crystalmobile.com

IntroductionIntroduction

• Application of video technologies on Application of video technologies on mobile platforms is very wide. mobile platforms is very wide.

- entertainment and adult industry- entertainment and adult industry

- information communications- information communications

- collecting of private videos- collecting of private videos

- etc.- etc.

ProblemsProblems

• Mobile platforms have not enough Mobile platforms have not enough memory space for full-quality videos memory space for full-quality videos and GPRS channels have no required and GPRS channels have no required bandwidth capacity.bandwidth capacity.

• CPUs are too slow for application of CPUs are too slow for application of complex algorithms.complex algorithms.

Why to compress video?Why to compress video?

• Uncompressed video occupies huge amounts Uncompressed video occupies huge amounts of data. For example, for storing of one of data. For example, for storing of one minute quality video with 320x240 resolution minute quality video with 320x240 resolution with 15 frames per second it is needed with 15 frames per second it is needed 320x240x3x15x60 = 200 MBytes of data.320x240x3x15x60 = 200 MBytes of data.

• Powerful CPU platforms allowed the Powerful CPU platforms allowed the implementation of complex algorithms implementation of complex algorithms giving 100-500 compression ratio.giving 100-500 compression ratio.

Common ideas of video Common ideas of video compressioncompression

• Superfluous information.Superfluous information.

• Neglect of the insignificant details for Neglect of the insignificant details for human-eye.human-eye.

Superfluous details that are Superfluous details that are considered in CM Videoconsidered in CM Video

• The low space correlation of picture. The low space correlation of picture. In any image, near lying pixels have In any image, near lying pixels have little difference between them.little difference between them.

• The low time correlation of picture. The low time correlation of picture. In any video clip there are a lot of In any video clip there are a lot of close frames with little difference close frames with little difference between them.between them.

Low space correlationLow space correlation

Normal Normal PicturePicture

Half Half resolutionresolution

Low time correlationLow time correlation

Normal PictureNormal Picture Difference between Difference between two following framestwo following frames

The details that are The details that are insignificant for human-eyeinsignificant for human-eye

• The human eye almost does not analyze The human eye almost does not analyze the color, but very good differentiate the color, but very good differentiate brightness of the image.brightness of the image.

• Insignificant noisy movements of image Insignificant noisy movements of image parts are mostly invisible for human-eye.parts are mostly invisible for human-eye.

• The disturbances on noisy image part are The disturbances on noisy image part are weakly differentiated.weakly differentiated.

Chroma vs colorChroma vs color

1/8 of color1/8 of color 1/8 of 1/8 of chromachroma

Insignificant movementsInsignificant movements

Normal pictureNormal picture The picture with The picture with random distortionrandom distortion

The disturbances on noiseThe disturbances on noise

Normal Normal PicturePicture Disturbances on noiseDisturbances on noise

(red)(red)

YUV ColorspaceYUV Colorspace

• During the translation into YUV-color every pixel During the translation into YUV-color every pixel (RGB vector) is multiplied by 3x3 matrix. (RGB vector) is multiplied by 3x3 matrix.

• As the result, the Y – brightness channel and two As the result, the Y – brightness channel and two U and V color compositions are determined. U and V color compositions are determined.

• CME uses confidant transformations with integers CME uses confidant transformations with integers – it gives significant performance improvements – it gives significant performance improvements on most platforms. on most platforms.

• Meanwhile, the resolution of U and V channels is Meanwhile, the resolution of U and V channels is divided by two along each axis. As human eye divided by two along each axis. As human eye does not differentiate colors as good as does not differentiate colors as good as brightness, it’s possible to reduce the detalization brightness, it’s possible to reduce the detalization of color planes twice.of color planes twice.

Low time correlation of the Low time correlation of the videovideo

• B, I and P frames.B, I and P frames.

• The definition of context change and The definition of context change and the division of video streams into the division of video streams into blocks (‘sandwiches’). The encoding blocks (‘sandwiches’). The encoding is done by sandwiches of 1-3 frames.is done by sandwiches of 1-3 frames.

B, I and P framesB, I and P frames

Frame processing. Frame processing. Macroblocks.Macroblocks.• Every frame in sandwich is divided Every frame in sandwich is divided

into 16x16 blocks (macroblock). into 16x16 blocks (macroblock). • Every macroblock consists of 4 Every macroblock consists of 4

brightness blocks of 8x8 and two brightness blocks of 8x8 and two color blocks 8x8, since the color color blocks 8x8, since the color resolution of the frame is divided by resolution of the frame is divided by two. two.

• Most processing methods operate on Most processing methods operate on macroblocks.macroblocks.

Motion compensationMotion compensation

• Motion compensation is intended for Motion compensation is intended for prediction of the motion on the prediction of the motion on the picture. It uses low time correlation picture. It uses low time correlation between close frames.between close frames.

• CME Video provides motion CME Video provides motion compensation of 16x16 blocks with compensation of 16x16 blocks with bi-linear interpolation.bi-linear interpolation.

Motion compensationMotion compensation

Frame from the streamFrame from the stream Motion vectorsMotion vectors

I-framesI-frames

• Encoder analyses whole frame to find Encoder analyses whole frame to find images cannot be predicted from the images cannot be predicted from the history.history. These frames should be stored These frames should be stored as independent data blocks, I-frames.as independent data blocks, I-frames.

• Positioning is Positioning is preciseprecise to I-frames, so I- to I-frames, so I-frames frames periodicityperiodicity should be at least should be at least one I-frame for 30 seconds.one I-frame for 30 seconds.

Wavelets (DWT)Wavelets (DWT)

• The images are passed the 2D wavelet The images are passed the 2D wavelet transformation. Every frame is passed by transformation. Every frame is passed by bicubic/bilinear wavelet transformation. The main bicubic/bilinear wavelet transformation. The main advantage of the format is about wavelet advantage of the format is about wavelet transformation is applied to separated images in transformation is applied to separated images in the picture.the picture.

• The smoothing is done by bicubic/bilinear The smoothing is done by bicubic/bilinear interpolation, there are no effects of DCT. The interpolation, there are no effects of DCT. The pictures on the next slide show the difference pictures on the next slide show the difference between the losses using DCT (left) and bicubic between the losses using DCT (left) and bicubic wavelet composition(right) using the same bitrate.wavelet composition(right) using the same bitrate.

DCT vs WaveletsDCT vs Wavelets

DCTDCT WaveletsWavelets

QuantizationQuantization

• The sandwich is passed through the The sandwich is passed through the process of quantization – every point is process of quantization – every point is rapidly divided by the definite number rapidly divided by the definite number from vector of quantization. Every from vector of quantization. Every element of vector of quantization is element of vector of quantization is correspondent to some frequency of correspondent to some frequency of wavelet composition. On this step, wavelet composition. On this step, the the most loss is occurringmost loss is occurring

Storing the coefficientsStoring the coefficients

• Each frame is passed to zero-quantors Each frame is passed to zero-quantors extraction by applying quadro-tree processing. extraction by applying quadro-tree processing. After quantization most number of small After quantization most number of small coefficients is zeroed, that is why quadro-tree coefficients is zeroed, that is why quadro-tree processing is very efficient on this step – it’s processing is very efficient on this step – it’s the the main part of compressionmain part of compression

• After group encoding, 16 data blocks of each After group encoding, 16 data blocks of each frame are compressed using Huffman method.frame are compressed using Huffman method.

The current status of CME The current status of CME EngineEngine

• Video codec of CME Engine is done. Video codec of CME Engine is done. High-performance audio codec is High-performance audio codec is under development.under development.

• The current implementation uses The current implementation uses fixed-point ANSI C without any fixed-point ANSI C without any proprietary libraries. proprietary libraries.

Symbian 6.1 platformSymbian 6.1 platform

• CME Engine perfectly runs on Symbian CME Engine perfectly runs on Symbian 6.1 platform (Nokia Series 60 phones). 6.1 platform (Nokia Series 60 phones). There is a room for tuning video There is a room for tuning video parameters, but overall playback is parameters, but overall playback is good.good.

• Release version of player software for Release version of player software for Symbian platform is ready and was Symbian platform is ready and was offered to public on 01.10.2003.offered to public on 01.10.2003.

Player is working on Player is working on Nokia3650Nokia3650

• Encoding on PCEncoding on PC

• Realtime playback Realtime playback on Nokia Series60 on Nokia Series60 familyfamily

• High quality of the High quality of the videovideo

Results & Results & comparisoncomparisonss

• CM Video has better quality than H.263 CM Video has better quality than H.263 and MPEG1 with the same bit rate. and MPEG1 with the same bit rate.

• CM Video technical parameters:CM Video technical parameters:

- 176x144 10fps on Nokia3650/7650- 176x144 10fps on Nokia3650/7650

- 1 Mb per minute bitstream for good - 1 Mb per minute bitstream for good quality.quality.

Standard video formats & Standard video formats & playersplayers

• MPEG1. Unusable for mobiles.MPEG1. Unusable for mobiles.

• H.263 Video Recorder for Nokia from H.263 Video Recorder for Nokia from Hantro Oy and Emuzed.Hantro Oy and Emuzed.

• Real Video. Real Player for Nokia Real Video. Real Player for Nokia from Real Networks.from Real Networks.

Crystal Reality LLCCrystal Reality LLC www.crystalreality.com www.crystalreality.com

• Founded in March 2003Founded in March 2003• 900.000 downloads of Crystal Player 900.000 downloads of Crystal Player

ProfessionalProfessional• Very strong user community in Europe, Very strong user community in Europe,

Russia and USARussia and USA• Crystal Mobile Engine was developed during Crystal Mobile Engine was developed during

July 2003 – Sept 2003July 2003 – Sept 2003• 3 fulltime and 2 part time developers are now 3 fulltime and 2 part time developers are now

employedemployed• Develops mobile technologies for the futureDevelops mobile technologies for the future