openmax al, opensl es - khronos group · 2014-04-08 · © copyright khronos group, 2011 - page 10...
TRANSCRIPT
© Copyright Khronos Group, 2011 - Page 1
OpenMAX AL, OpenSL ES Native Multimedia in Android
Erik Noreke Chair of OpenMAX AL and OpenSL ES Working Groups
© Copyright Khronos Group, 2011 - Page 2
Why Create Khronos Multimedia APIs?
Khronos APIs
•Conformance Tests
•Portable
•IP Protection
•Profiles & Components
•Predictable set of functionality
Open Source Framework
•Free to change
•Every implementation unique
•GPL license
•Plugins
•Functionality dependent on
implementation
© Copyright Khronos Group, 2011 - Page 3
&
Application Access Layer
© Copyright Khronos Group, 2011 - Page 4
The Need for Application Access
• Hardware and low level access is powerful but
complex
- More power than most application developers
require
• Most application developers just want to
playback and record media
- Specifying where the content comes from and is
rendered to
- Manipulate a few playback controls
- Have simple configurability
© Copyright Khronos Group, 2011 - Page 5
The Need for Application Access
• That are what OpenMAX AL and OpenSL ES
provide…
- Simple high-level multimedia APIs for playback
and recording use cases
- With hardware acceleration capabilities built in
• …With Cross-platform portability
- No need to rewrite an application for every
platform
© Copyright Khronos Group, 2011 - Page 6
OpenMAX AL Focused on Streaming Media
• Media playback and recording
- Playback and recording of everything from video files to
radio to camera input
• Full range of video effects and controls – including
playback rate, post processing, and image
manipulation
- Apply advanced effects to captured images and video
before sending to friends
© Copyright Khronos Group, 2011 - Page 7
OpenMAX AL Focused on Streaming Media
• OpenMAX AL gives easy access to
streaming media
- Allowing time to be spent on developing
applications, not using frameworks
• Designed as a complete set of
multimedia functionality
- Unlike any other streaming media API
© Copyright Khronos Group, 2011 - Page 8
OpenSL ES – Powerful Mobile Audio
• Full range of effects and controls,
including advanced 3D effects such as
Doppler and virtualization
- Experience rich, enhanced sound from
locations other than the handset, even
moving, for the ultimate multimedia
experience
• Create theater quality audio
experience
- In a mobile device!
© Copyright Khronos Group, 2011 - Page 9
OpenSL ES – Powerful Mobile Audio
• Full 3D audio functionality enhances
any gaming experience
- Perfect companion to OpenGL ES
• Designed for implementation by either a
hardware or software solution
- Take full advantage of system capabilities
© Copyright Khronos Group, 2011 - Page 10
Object Oriented Media Access
• Simplified object-oriented streaming media
- Built to enable common use cases
- But also has access to advanced functionality
• Media Objects enable PLAY and RECORD of content
- Media Objects input and output to Devices
- Perform some operation on an input and emit the result as output
• Objects have control interfaces
- Play, Record, Seek, Rate
Player Object URI
Memory
Content pipe
Media Input
Camera (AL)
URI
Memory
Content pipe
Audio Mix
Display Window (AL)
DSrc DSnk
© Copyright Khronos Group, 2011 - Page 11
OpenMAX AL & OpenSL ES
© Copyright Khronos Group, 2011 - Page 12
OpenSL ES and OpenMAX AL • Working groups collaborated to define the common API functionality
• Independent, Compatible, Consistent and Distinct
3D Audio
Audio Effects
Advanced MIDI
Audio Playback
Audio Recording
Basic MIDI
Video playback
Camera
Video recording
Image capture & display Buffer
queues
Radio and RDS
© Copyright Khronos Group, 2011 - Page 13
Why Profiles?
• Functionality divided into basic groups
- Most common groupings
• Profiles reduce fragmentation
- Not all devices support all functionality
- Only support the profiles you need
• Applications can query available profiles
- To adapt to the current device
- Facilitates porting of applications
© Copyright Khronos Group, 2011 - Page 14
Why Profiles?
• Profiles can be combined
- To best suit any product
• Optional features
- Some features are optional in all profiles
- APIs are consistent when feature is available
© Copyright Khronos Group, 2011 - Page 15
OpenSL ES and OpenMAX AL • Independent
- But designed to work together
- No dependency between the APIs – either one can exist by itself
• Compatible
- Working groups collaborated to make sure the APIs work together well
© Copyright Khronos Group, 2011 - Page 16
OpenSL ES and OpenMAX AL • Consistent
- Identical API architecture
- Identical APIs for same functionality
• Distinct
- OpenMAX AL represents basic multimedia functionality (audio, video, and image)
- OpenSL ES represents advanced audio functionality
© Copyright Khronos Group, 2011 - Page 17
Complete Multimedia Use-Case
Multimedia Processing
OpenGL ES GL_TEXURE_EXTERNAL_OES texture
3D Audio Rendering
Application on CPU
Video stream to GPU
EGLStream
Much more flexibility than just “show a video”
EGLSurface OpenGL ES
Audio stream to DSP
© Copyright Khronos Group, 2011 - Page 18
Android NDK for Multimedia • NDK – Native Development Kit
• Introduced in
Android 2.3 Gingerbread
Mobile 3D
Advanced Audio
© Copyright Khronos Group, 2011 - Page 19
Why the NDK?
• Better control of fuctionality and timing
• Access to hardware acceleration
• Multimedia is the primary reason for the NDK
• Increases application complexity
• Requires compilation for each platform family
- ARMv5TE
- ARMv7-A
- x86 instructions
© Copyright Khronos Group, 2011 - Page 20
Focus is on Standardized APIs
• Earlier internal APIs not stable enough…
• Avoid fragmentation
• Well tested and documented
• Focus on functionality
- Not on implementation
• Access to hardware acceleration
© Copyright Khronos Group, 2011 - Page 21
Native Multimedia APIs in Android 2.3
• OpenGL ES 2.0
• EGL 1.4
• OpenSL ES 1.0.1
- Partial implementation
- Almost complete Phone + Music profiles
- Buffer queues from 1.1 as Android extension
- Google version compliant as far as implemented
- Other vendors offer fully compliant solution
Mobile 3D
Advanced Audio Graphics Context Management
© Copyright Khronos Group, 2011 - Page 22
Ice Cream Sandwich – Android 4.0 • Same OS for both phones
and tablets
• Better multitasking
• Better support for
hardware accelerated
graphics
• DRM
• Partial OpenMAX AL 1.0.1
implementation
- Streaming use-case only
- Does not interoperate with
OpenSL ES
Streaming Media
© Copyright Khronos Group, 2011 - Page 23
Android & OpenMAX IL
• Android uses OpenMAX IL in their
Multimedia framework
- Codecs only
- No graph building
• Main Multimedia framework is still
Stagefright
- Introduced in Éclair
• Allows for both software
implemented and hardware
accelerated codecs
• Not part of NDK
- Not intended to be accessible from
applications
© Copyright Khronos Group, 2011 - Page 24
What is Google saying?
© Copyright Khronos Group, 2011 - Page 25
What is Google saying?
© Copyright Khronos Group, 2011 - Page 26
What is Google saying?
© Copyright Khronos Group, 2011 - Page 27
Where is Android Headed?
• More Devices
- TVs?
- Netbooks?
- Chrome OS?
• What does the future hold?
- Compliant OpenSL ES?
- Better OpenMAX AL support?
- OpenMAX AL EGL Stream Extension?
- Support for OpenCL?
- Content Pipes?
© Copyright Khronos Group, 2011 - Page 28
Thank You