Android Media Player Development

Suyash GuptaAndroid Developer Talentica Software

Agenda

1.Introduction

2.Characteristics of Video streams

3.Android Media Player Development

4.Advanced Media Player Development

• Video - Most Popular and common way of entertainment.

• Boom in mobile market in past 5 years.

• 3g services revolutionized the telecommunication industry.

1. Introduction

1) Video - Journey So Far

● Video technology was first developed for (CRT) televisions.● Video recorders were sold for $50,000 in 1956, and videotapes

cost $300 per one-hour reel. ● Sony introduced VCR in 1971.● Then came CD's in 1997.● Followed by blue ray disc in 2006.● Computers to capture, store, edit and transmit video clips.● Smartphones users grew to a increase to a limit of 80%.● Video watching became very user friendly.

Note: In Real-time entertainment the major share is made by YouTube, Netflix and Pandora.

2. Characteristics of Video Streams

a) Frame rate

b) Interlaced VS progressive

c) Aspect ratio

d) Color depth

e) Digital video

f) Video compression method (digital only)

g) Containers for Codecs

a) Frame Rate

The number of still pictures per unit of time of video.

a) Frame Rate

• To identify each frame individually min 10-12 FPS.

• Early silent films had a frame rate from 14 to 24 FPS but it was perceived as jerky motion.

• There are three main frame rate standards in the TV and digital cinema business: 24p, 25p, and 30p.

• 48p is a progressive format and is currently being trailed in the film industry.(The Hobbit)

b) Video Can Be Interlaced/Progressive

Interlaced video:

● Every frame has horizontal lines.● Each frame is divided into 2 fields.● Odd field and even field.● Interlacing is a technique of doubling the FR within the

limitations of a narrow bandwidth.● Image Detail is sacrificed.

b) Interlaced & Progressive Frames

Odd Field Even Field

b) Interlaced Video Fields

b) Interlaced Video Frame

● Progressive is a method for displaying, storing or transmitting moving images.

● Each Frame resembles an image.

● Progressive scan is used in most CRTs used as computer Monitors.

● It is also becoming increasingly common in high-end Television equipment, which is often capable of performing de-interlacing so that Interlaced video can still be viewed.

b) Progressive Video

b) Progressive Video Frame

Advantages of progressive scan include:

1. Better quality of video.

2. No flickering of narrow horizontal patterns.

3. Simpler video processing equipment.

4. Easier compression than interlaced.

b) Progressive Video

c) Aspect ratio

● Aspect ratio describes the dimensions of video screens.● Traditional television screen is 4:3.● High definition televisions(720p and 1080p) 16:9.● AR 1.85:1 and 35 mm US and UK widescreen standard for

theatrical film.

c) Some Common Aspect Ratios

d) Color Depth

The number of bits used to indicate the color of a single pixel, in a video frame buffer.

Bit is a storage value of a pixel.

1-bit color (2^1 = 2 colors): monochrome, black and white

2-bit color (2^2 = 4 colors): CGA, gray-scale early NeXTstation, color Macintoshes ect.

3-bit color (2^3 = 8 colors)

4-bit color (2^4 = 16 colors):

d) Color Depth

6-bit color (2^6 = 64 colors):

8-bit color (2^8 = 256 colors): most early color Unix workstations, VGA at low resolution, video games.

16-bit color (2^16 = 32768 colors): some Silicon Graphics systems

True color supports 24-bit for three RGB colors.

d) Color Depths Examples

2 bits (4 colors)

4 bits (16 colors)

1 bit (2 colors)

d) Color Depths

24 bits (16,777,216 colors, "truecolor")

8 bits (256 colors)

e) Digital Video

● Analog video stores video signal on magnetic tape.● When copies are made of an Analog signal, degradation is

introduced due to loss of video signal in the copying.● Whereas in digital data every bit has a binary value.● So copying is 100% precise.

Analog Solution to Video Recording Digital Video Storage Medium

e) Size of Raw Digital Video

Digital Video in its raw form can be huge.

An example video can have a duration of (3600sec), a frame size of 640x480 (WxH) at a color depth of 24bits and a frame rate of 25fps.

● pixels per frame = 640 * 480 = 307,200

● bits per frame = 307,200 * 24 = 7,372,800 = 7.37Mbits

● bit rate (BR) = 7.37 * 25 = 184.25Mbits/sec

● video size (VS) = 184Mbits/sec * 3600sec = 662,400Mbits = 82,800Mbytes = 82.8Gbytes

e) Issues

Leads to two interrelated problems:

1) Large storage requirements.

2) High Bandwidth requirement during playback and recording.

f) Video Compression

1. Conversion of data to a format that requires fewer bits.

2. The inverse process is known as decompression.

3. Decompression may or may not yield an exact copy of the original data.

f) Video Codec

A codec is a compression algorithm, used to reduce the size of a stream. The compression is usually lossy.•Lossless compression: It is a class of data compression algorithms that allows the original data to be perfectly reconstructed from the compressed data.

•“Lossy" compression is a data encoding method that compresses data by discarding (losing) some of it.

80% compression

90% compression

96% compression

f) Comparison of Lossy Compression

MPEG-1(1993)(Moving Pictures Expert Group)[Old, supported by everything (at least up to 352x240)]

MPEG-2 codecsA good quality of compression. This is what is used for DVD.

MPEG-4 (1998)AVC (DivX codec)A good quality compression with maintaining the quality of video.

H.263/MPEG-4 Part 2 codecsDivX Pro Codec: Compresses video down to very small file sizes- excellent for web use but loses quality.

f) Different Codecs

That's why not used for the television purpose

Xvid: Free/open-source implementation of MPEG-4 ASP, originally based on the OpenDivX project. Similar as DivX Pro Codec.

FFmpeg MPEG-4: Open-source libavcodec codec library

Used by default for decoding or encoding in many open-source video players such as MXPlayer, VLC, MPlayer and GStreamer.

WMV (Windows Media Video): Microsoft's family of proprietary video codec designs including WMV 7, WMV 8, and WMV 9.

Google (On2) codecs:

VP6, VP6-E, VP6-S, VP7, VP8 VP9 : Proprietary high definition video compression formats and codecs developed by On2 Technologies

g) Containers for Codecs

The container describes the structure of the file: where the various pieces are stored and how they are made.

I) AVI (standard Microsoft Win container)

(mostly used in digital cameras)

II) MOV (standard QuickTime container)

Carries the widest range of codecs

III) MPEG-2 TS (acronym of Transport Stream, standard container for digital broadcasting)

g) Containers for Codecs

IV) MP4 (standard container for the MPEG-4 multimedia portfolio)

V) RealMedia (standard container for RealVideo and RealAudio)

VI) 3gp (used by many mobile phones)

3. Android Media Player

a) Android supported formats

b)Media player API and Sample Code

c)Limitations of Media Player

Type Format / Codec

Encoder Decoder Details Supported File Type(s) / Container Formats

Video H.263 • • • 3GPP (.3gp)• MPEG-4 (.mp4)

H.264 AVC •(Android 3.0+)

• Baseline Profile (BP)

• 3GPP (.3gp)• MPEG-4 (.mp4)• MPEG-TS (.ts, AAC audio only, not seekable, Android 3.0+)

MPEG-4 SP • 3GPP (.3gp)

VP8 •(Android 4.3+)

•(Android 2.3.3+)

Streamable only in Android 4.0 and above

• WebM (.webm)• Matroska (.mkv, Android 4.0+)

3. a) Android supported formats

3. b) Media Player API

RecordPlayback

Audio / Video

android.media.MediaPlayer android.media.MediaRecorder

b) Playback: MediaPlayer Basics

Play Media From:

• Resource Folder• File System Path• URL

Basically

• Create new instance of MediaPlayer• Call prepare()• Call start()

b) Playback View

Playing a video File

<?xml version="1.0" encoding="utf-8"?><RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android" android:id="@+id/RelativeLayout1" android:layout_width="match_parent" android:layout_height="fill_parent" android:orientation="horizontal" >

<SurfaceView android:id="@+id/surfaceView1" android:layout_width="match_parent" android:layout_height="wrap_content" android:layout_above="@+id/seekBar1" android:layout_alignParentTop="true" android:layout_weight="0.67" /> </RelativeLayout>

b) Layout View

Create Using:

• URI• Resource ID

States:

• Start• Prepare• Pause• Release• Reset• SeekTo• Stop• Error

Get:

• Position• Duration• isPlaying

Set:Output DisplayLooping

• Data Source• Wake Mode

Listeners:• Buffering Update• Completion• Error• When Prepared

Inherits fromjava.lang.Object

b) MediaPlayer Class

public class MainActivity extends Activity implements SurfaceHolder.Callback,OnPreparedListener, OnErrorListener, OnInfoListener {

MediaPlayer mediaPlayer;SurfaceHolder surfaceHolder;SurfaceView playerSurfaceView;String videoSrc = "rtsp://v6.cache1.c.youtube.com/CjYLENy73wIaLQkDsLHya4-

Z9hMYDSANFEIJbXYtZ29vZ2xlSARSBXdhdGNoYKX4k4uBjbOiUQw=/0/0/0/video.3gp";//String dataSrc = "http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4";//int resId = R.raw.sample;//.mp4//int String dataSrc = "/sdcard/music/sample_music.mp3";

@Override

protected void onCreate(Bundle savedInstanceState) {super.onCreate(savedInstanceState);setContentView(R.layout.activity_main2);playerSurfaceView = (SurfaceView) findViewById(R.id.playersurface);surfaceHolder = playerSurfaceView.getHolder();surfaceHolder.addCallback(this);

}

b) Activity Code

@Overridepublic void surfaceCreated(SurfaceHolder arg0) {

try {mediaPlayer = new MediaPlayer();mediaPlayer.setDisplay(surfaceHolder);mediaPlayer.setDataSource(videoSrc);mediaPlayer.setOnPreparedListener(this);mediaPlayer.setOnErrorListener(this);mediaPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);mediaPlayer.prepare();} catch (IllegalArgumentException e) {

e.printStackTrace();} catch (SecurityException e) {

e.printStackTrace();} catch (IllegalStateException e) {

e.printStackTrace();} catch (IOException e) {

e.printStackTrace();}

}

b) Activity Code

@Overridepublic void onPrepared(MediaPlayer mp) {

mediaPlayer.start();}

@Overridepublic boolean onError(MediaPlayer mp, int what, int extra) {

Log.e("MainActivity", "what="+what+" extra="+extra);return false;}

@Overridepublic boolean onInfo(MediaPlayer mp, int what, int extra) {

Log.i("MainActivity", "what="+what+" extra="+extra);return false;}

b) Activity Code

Media Player State Diagram

c) Limitations of Media Player

• Any time media player changes state, check before selecting any action or you may fall into IllegalStateException.

• Error when the orientation of screen is change, even if you manage the tag screenOrientation=“orientation” in manifest.xml

• When activity goes in background and MediaPlayer is playing, even if you have handled onPause and onResume in some devices it goes to error state.

• There are 30 to 34 types of possible error starting from -1 to -34.For e.g. Error: the video container is not valid for progressive playback (-32);Error due to resource being in wrong state to handle request (-14)Error due to general data processing(-18)

c) Limitations of Media Player

• public static void playMusic(int id) { // Not a good practiceMediaPlayer mediaPlayer = MediaPlayer.create(context, id); mediaPlayer.setLooping(true); mediaPlayer.start();

} // Here the media player will be removed by GC even if it is in play mode.

• The error value (1, -2147483648), the '1' value corresponds to the constant in MediaPlayer.MEDIA_ERROR_UNKNOWN. (This might happen)-2147483648 corresponds to hexadecimal 0x80000000 which is defined as UNKNOWN_ERROR

4. Advanced Media Player Development

• Customization Features

• Benefits of Customization

• Drawbacks

• FFMPEG Library

We can Customize the Media Player and achieve :

We can enhance security.a) Playing video in chunksb) Prevent video from getting stolen.

c) Encrypt the InputStream with Cipher.

• Real Time manipulation.(Play ads in between)• Adaptive Streaming.(Configurable bandwidth)• Multi Format Playing.(discussed later)

a) Customization Features

• Your video content is secure. • Easily applicable to other java platforms too for e.g. J2ME• Saved video can be played only by our media player.• Configurable bandwidth according to network speed.• Since playback is chunk by chunk you can show video ads in

between them too.• Event handling during playback is possible for e.g. at what point

the user stops the video or how much percentage watched.

b) Benefits of Customization

• Decoding of Android media player is bad, may fall into lot of errors and you might not get possible solution anywhere.(Limited number of file formats)

• The total chunk playback duration and the clip playback duration should match or the player will again throw errors like ChunkNotFound (Custom exception).

c) Drawbacks

d) FFMPEG Library

• Almost every android phone user is friendly with MX Player which is a pretty famous player.

• MX Player is build using FFMpeg library which we will discuss later.

• It is user-friendly, plays almost all formats and possesses great build-in features.

• Plays almost every movie files including .3gp .avi .divx .f4v .flv .mkv .mp4 .mpeg .mov .vob .wmv .webm .xvid and many more

• Hardware decoding (with h/w+ decoder)• Processor optimization

• High speed rendering for ARM® NEON™ compliant processors.

• Provides processor specific Codecs including Tegra 2 processor.

• Move forward/backward (Optional)• Aspect ratio selection

d)Features of MX Player

Following conditions are required to run included shell script:•Linux or Cygwin•Latest Google NDK

Download ffmpeg source code1.In config-ffmpeg.sh, change "NDK" variable to point NDK path.2.Add this line NDK=$HOME/Desktop/adt/android-ndk-r9 3.In build.sh, change "MAKE" variable to point ndk-build script of NDK directory.4.Run the shell scripts by command ./build.sh5.The build can take a while to finish depending on your computer speed.6.Make ffmpeg Libraries available for Your Projects

How to build FFmpeg for Android

References

Video Characteristics:http://en.wikipedia.org/wiki/Video

Android Developer Site:http://developer.android.com/reference/android/media/MediaPlayer.html

FFMPEG Build:http://www.roman10.net/how-to-build-ffmpeg-with-ndk-r9/

Thank You!