camera 2.0 in android 4.2

| © 2013 Aptina Imaging Corporation1

© 2013 Aptina Imaging Corporation. All rights reserved. Products are warranted only to meet Aptina’s production data sheet specifications. Information, products, and/or specifications are subject to change without notice. All information is provided on an “AS IS” basis without warranties of any kind. Dates are estimates only. Drawings not to scale. Aptina and the Aptina logo are trademarks of Aptina Imaging Corporation. All other trademarks are the property of their respective owners.

Camera 2.0 The New Camera Hardware Interface in Android 4.2

Balwinder Kaur, Senior Member, Technical Staff, Aptina Imaging

Ashutosh Gupta, Principal Software Engineer, Aptina Imaging

Android Builder’s Summit, San Francisco, CA

2.18.2013


Agenda• Camera Use Cases

‣ Prominent Use Cases

‣ Limitations of existing APIs

‣ Overview of android.hardware.Camera

• Android Framework : Camera Service

‣ Native Camera service

‣ New - Under the hood – Camera 2.0

• It’s all about the Camera hardware !

‣ Camera Hardware Abstraction Layer

‣ Camera Device Driver

• Challenges in Camera HAL development

• Emerging Trends

• Q&A


Section I

Android Camera APIs


Prominent Camera Use Cases

• Main Use Cases

‣ Live Preview of Camera Stream

• Live Preview + copy of the Frame returned to the application

‣ Capture a frame

‣ Video Recording of a Camera Stream

• Secondary Use Cases

‣ Configuring the Camera

‣ Snapshot during video recording

‣ Event Callbacks: Shutter Clicked, AutoFocus Achieved

• Information Related Use cases• Receiving more than an image back . e.g. face detection data

• Meta Data of an Image


Limitations in Camera API

• No support for Burst Mode Photography

• No support for Panoramic Shots

• Very limited support for frame metadata

• No per-frame control of the camera or the image processing pipeline

• No access to control subsystems within the camera e.g. Focus, Flash, Image Sensor


Overview of android.hardware.Camera

6 Classes

• Camera

• Camera.CameraInfo

• Camera.Parameters

• Camera.Size

• Camera.Face

• Camera.Area

8 Callback Interfaces

• Camera.AutoFocusCallback

• Camera.ErrorCallback

• Camera.FaceDetectionListener

• Camera.OnZoomChangeListener

• Camera.PictureCallback

• Camera.PreviewCallback

• Camera.ShutterCallback

• Camera.AutoFocusMoveCallback


Camera class

Contains all the methods for the Camera Lifecycle

• Open & Release

• Access to the Camera Controls

• Preview

‣ Direct Live Preview to the display or a texture

‣ Get Preview Frame in a Callback

• Capture

‣ Callbacks: Shutter, JPEG, RAW, “Postview”

• Lock & Unlock

• Actions: startAutoFocus, startSmoothZoom & startFaceDetection


Camera.Parameters

Class for Camera Controls

① Mandatory Feature Set

‣ getSupportedPreviewSizes + set/get

② Optional Feature Set

‣ isVideoStabilizationSupported + set/get

③ Custom Feature Set

‣ Camera.Parameters class provides a “dumb” pipe to the hardware for custom controls

‣ set (“your_param_string”, value); get(“your_param_string”);

Auto White Balance, Scene Modes, Focus Modes, Preview Sizes, Picture Sizes, Thumbnail Sizes, Preview Format, Picture Format, Anti-Banding


… the rest of the Camera Classes

• Camera.CameraInfo

‣ For each camera, front or back facing, orientation of the camera image

• Camera.Size

‣ width and height of the image

• Camera.Face

‣ face-id, co-ordinates for left eye, right eye, mouth, outer bounds of the face

• Camera.Area

‣ Rectangular bounds with a weight

‣ Metering Regions for 3A : Auto Focus, Auto White Balance, Auto Exposure


Camera 2 Internals

Released as part of Android 4.2

Camera 1 => android.hardware.Camera

Camera 2 => android.hardware.ProCamera

New Camera HAL implementation based on Camera 2.0 – Samsung exynos5 based

No application that actually uses Camera 2.0Disclaimer: All reference to Camera2 in this slide deck is based on reverse engineering AOSP code !


Section II

Android Framework - Camera


High Level Architecture


Source: Android Anatomy and Physiology, Google IO 2008

Android High Level Architecture


Source: Android Anatomy and Physiology, Google IO 2008

Hardware Abstraction Layer

Camera


Camera Subsystem

Application

Application framework

Camera Service

Camera HALImplementation

Camera Device Driver

Hardware Independent

Hardware Dependent

Camera Hardware

HAL = Hardware Abstraction Layer


Process View

App AppApp

Binder IPCICamera

Media server

Camera Service

Back Facing Camera

Hardware Object

Front Facing Camera

Hardware Object

Kernel Driver Kernel Driver

System Call System Call

Binder IPCISurface

Surfaceflinger


Inside the Camera App

Media server

Camera Servicelibcameraservice.so

Application Code

android framework code Camera.java

JNI

android_hardware_Camera.cpp

IBinder Interfaces


JNI Layer


android_hardware_Camera

• Allocates Memory from the Java memory heap for JPEG images.

• If a Copy of the Preview Frame is requested by the app, then the copy from native to java buffers is done here.

• Creates a persistent context for callbacks from native code to Java (JNICameraContext)

• Holds references to the Java Camera, Face and Area objects.


Camera Service

- with Camera 2.0


Camera Service

Media server

Camera Servicelibcameraservice.so

Camera HALimplementation

Application Code


JNI


ICameraClient | ICameraService | ICamera

camera.h


Camera Service with Camera 2

Media server

Camera Service libcameraservice.so

Camera HALimplementation

Application Code


JNI


ICameraClient | ICameraService | ICamera

camera.h

Camera2 HALimplementation

camera2.h

Camera2Client

Camera2Device


Camera Service

• Runs in the media server process

• It is a shared library libcameraservice.so

• Queries System for Camera HAL Api version - accordingly creates a CameraClient or Camera2Client for the application

• For Camera 1, libcameraservice does:Permission check android.permission.CAMERA

Ensures only one Client connects to a Camera Hardware Object

Ensures each Process connects to a single Camera Hardware Object

Redirects callbacks back to the app layer

Accessed over IBinder Interface

Number of Cameras Available

CameraInfo Details


Camera2Client• Implements android.hardware.camera API on top of camera device HAL

version 2

• Has processors that run in their own threads

Streaming Processor

JPEG Processor

Callback Processor

ZSL Processor

Frame Processor

Capture Sequencer

Preview Stream

Recording Stream

Callback Stream

Capture Stream

ZSL Stream

Manages Preview & Recording Streams

Still Image capture output processing

Processes Callback

ZSL Queue Processing

Metadata processing for output frames

Manages Capture Sequences for ZSLregular and Burst modes


Camera2Device

• Manages Input and Output Streams between the Camera HAL and the Service

• Functions

‣ Retrieves Static Metadata Information of the Camera

‣ Sends requests for Capture and Streaming

‣ Manages input and output streams including re-processing streams and metadata buffers

‣ StreamAdaptor between ANativeWindow Interface and the Camera HAL stream ops

• ANativeWindows is Android Native Window interface

‣ Sends Notifications (Error, Shutter, AutoFocus, AutoExposure and AWB)


Metadata• Camera2 has a lot of emphasis on retrieving metadata from the camera

and making it available to the application

• Two kinds of Metadata

‣ Static & Frame-Based

‣ Static does not change and is available without opening the camera device (Camera Info Class)

‣ *_INFO is the static metadata

ANDROID_LENS, ANDROID_LENS_INFO, ANDROID_SENSOR, ANDROID_SENSOR_INFO, ANDROID_FLASH, ANDROID_FLASH_INFO, ANDROID_HOT_PIXEL, ANDROID_HOT_PIXEL_INFO, ANDROID_DEMOSAIC, ANDROID_DEMOSAIC_INFO, ANDROID_NOISE, ANDROID_NOISE_INFO, ANDROID_SHADING, ANDROID_SHADING_INFO, ANDROID_GEOMETRIC, ANDROID_GEOMETRIC_INFO, ANDROID_COLOR, ANDROID_COLOR_INFO, ANDROID_TONEMAP, ANDROID_TONEMAP_INFO, ANDROID_EDGE, ANDROID_EDGE_INFO, ANDROID_SCALER, ANDROID_SCALER_INFO, ANDROID_JPEG, ANDROID_JPEG_INFO, ANDROID_STATS, ANDROID_STATS_INFO, ANDROID_CONTROL, ANDROID_CONTROL_INFO, ANDROID_QUIRKS_INFO etc.

• Extensible - there is provision for Vendor Specific Tags


Android Open Source Project (AOSP) Structure

• Android Framework

‣ Java: frameworks/base/core/java/android/hardware

‣ JNI: frameworks/base/core/jni

• Camera Service

‣ frameworks/av/services/camera/libcameraservice/

• IBinder Interfaces

‣ frameworks/av/include/camera/ICamera.h etc.

• IBinder Implementation

‣ frameworks/av/camera/ICamera.cpp etc.

• Camera HAL Interface

‣ hardware/libhardware/include/hardware/camera2.h etc.

• Camera HAL Implementation

‣ hardware/<vendor>/camera (typically)

• Camera Metadata

‣ system/media/camera/include/system/camera_metadata_tags.h


Section III

Its all about the Camera Hardware !


Camera Hardware Abstraction Layer

Review of a Typical Implementation


Camera Stack – Camera HAL


(HAL)

Vendor Specific HAL Implementation

Camera Driver

Image Sensor Image Sensor Processer

SurfaceFlinger /Overlay Buffers User

Kernel

Hardware

Upper Camera Stack …


Android CameraHAL Library

• The Camera Hardware Abstraction Layer (HAL) is a library that is specific to the camera hardware platform

‣ Written by hardware vendors (Qualcomm, Samsung, TI, others)

• CameraHAL maps Android Camera Service calls to driver functions

‣ Ice Cream Sandwich (ICS) uses camera.h

‣ Jelly Bean (and above) use camera2.h

• CameraHAL low level interface communicates with the kernel level driver

‣ It can support interfaces including Video for Linux 2 (V4L2) or OpenMax (OMX)

‣ Communicates with the driver through file I/O calls (open, close, input/output controls (IOCTL), etc)


Sample ICS CameraHALFunctional Diagram

CameraHAL

Memory Manager

Camera Driver/dev/videoX

Camera Service I/F

Display Surface Manager

EventNotification Manager

Camera Manager

Kernel

User

Source: TI OMAP4 git.omapzoom.org


Sample JB CameraHALFunctional Diagram

CameraHAL Operations Handler

Stream Manager

Camera Driver/dev/videoX

Camera Service I/F

Metadata Handler

Reprocess Stream Management

Kernel

User

Source: Samsung Exynos 5


JB CameraHAL Block Diagram Discussion

• Block Functions

‣ CameraHAL

• Initialization, thread creation, function dispatch

‣ Stream Manager

• Handle stream events, coordinate buffer and stream usage, manage state machine

‣ Metadata Handler

• Acquire per shot metadata, convert to Android format

‣ Reprocess Stream Management

• Setup and manage reprocess streams


JB CameraHAL Changes

• For CameraHAL 2.0, much of the CameraHAL 1.0 functionality is moved to libcameraservice

• CameraHAL 2.0 targeted at HW specific functionality

• New CameraHAL functionality currently not passed up to applications

• Image metadata has more importance

• Reprocessing introduced – Process an already captured image stream

• Stream based rather than stream function based

‣ CameraHAL 1.0 has methods for specific stream types (e.g., start_preview(), take_picture())

‣ CameraHAL 2.0 has generalized stream methods (e.g., allocate_stream())


Camera Device Driver


Camera Stack – Camera Driver


(HAL)

Vendor Specific HAL Implementation

Camera Driver

Image Sensor Image Sensor Processer

SurfaceFlinger /Overlay Buffers User

Kernel

Hardware

Upper Camera Stack …


Android Kernel Camera Driver

• The kernel driver presents a standard interface for different types of camera hardware

‣ Camera hardware specific attributes are handled by the low level kernel driver

‣ Image Sensor Processor (ISP) vs. SOC (smart) sensor - differences are handled at the driver level

• For Android, Video for Linux 2 (V4L2) is used in many implementations

‣ V4L2 has been in existence for many years

‣ OpenMax (OMX) is also used for a low level driver interface by some vendors.


V4L2 Kernel Driver Block Diagram

V4L2 Driver Interface- IOCTL support/dispatch- V4L2 driver infrastructure

Controlling Interface- Support for different device configurations- Control device flow

Buffer/Memory Management- Memory allocation (if needed)- Buffer management- Buffer queue/de-queue

Camera HW Management- One of these blocks for each

camera type- Device discovery- Device initialization- Power management- Set/get device specific parameter- Enable/disable image streaming


V4L2 Kernel Driver Resources• Memory

‣ Memory can be either driver-allocated or user-provided

‣ The image transfers from the camera to memory through hardware Direct Memory Access (DMA)

‣ Hardware memory management may be used to avoid contiguous memory requirement

• Interrupts

‣ Camera ports support for interrupts on events such as frame start, finish, focus events, etc.

• Camera Control: I2C/SPI

‣ I2C (Inter-Integrated Control) is used for writing or reading camera registers

‣ SPI (Serial Peripheral Interface) is a faster alternative to I2C

• Control Signals/GPIO

‣ All controlled by the low level driver

• Power

‣ Sensor power management is critical to embedded device operation

‣ Sensors support standby mode where settings are maintained while power usage is reduced


V4L2 Driver Buffer Management• One or more buffers are supported

• User buffers or kernel-allocated buffers are supported

• Buffers are treated the same for preview, capture, video (output resolution does not matter)

• Buffers are queued to a circular list

• Buffer filling starts when the V4L2 Stream_On command is executed

• Once filled, the CameraHAL de-queues a buffer, processes the buffer, then re-queues the buffer

• The Stream_Off command causes all buffer to be released

Buff_0

Buff_1

Buff_2

Buff_3

Buff_x


Typical V4L2 Preview Sequence (1)

• V4L2 preview start up sequence is given below

V4L2 Call Driver Events Hardware Events

VIDIOC_S_FMT – set format

Set image format and size Set both resolution and output pixel format

VIDIOC_G_PARM – get parameter

Get a camera driver or hardware parameter

Read camera parameter

VIDIOC_S_PARM – set parameter

Set a camera driver or hardware parameter

Write camera parameter

VIDIOC_CROPCAP – get cropping capabilities

Return camera cropping capabilities

None

VIDIOC_S_CROP – set cropping

Set cropping rectangle Set camera cropping rectangle

VIDIOC_REQBUFS – request camera buffer

Request buffer support from the driver (user vs. kernel)

None

Loop: VIDIOC_QUERYBUF – query buffer caps

For kernel allocated buffers, return buffer characteristics

None

V4L2_MMAP – map buffers to user space

For kernel allocated buffers, memory map to user space

None


Typical V4L2 Preview Sequence (2)

• V4L2 preview start up sequence (cont)

• V4L preview shut-down sequence


Loop: VIDIOC_QBUF – queue buffers

Queue buffers in the circular queue

none

VIDIOC_STREAM_ON – start streaming

Start image capture state Enable image output


VIDIOC_STREAM_OFF – stop streaming

Stop streaming, deallocate buffer

Disable image output


V4L2 Media Controller Architecture

• Designed to support dynamically re-connectable hardware blocks.

• Allows for greater programmer control

• Introduces the notion of entities, pads and links


V4L2 Media Controller Architecture(Cont.)


Section IV

Challenges in Camera HAL Development


Challenges in Camera HAL Development

• Memory Management

• Various Implementation of Camera Driver

• Color Format Conversion

• Buffer Synchronization between Camera & Display

• Support for Advanced Features


Are all Camera HALs equal ?

• Supported Features depend on hardware capabilities

• Supported Features depend on the implementation in the Camera HAL

• Performance

• Reliability

• Extensions to the standard Android Feature Set

No


A Peek into the Future


Emerging Trends

• Computer Vision Applications go mainstream

‣ APIs on Object Tracking, Gesture Recognition become more common place

• Computation Photography application

‣ Developers get fine grained control of flash and camera

• High Dynamic Range

‣ Ability to capture larger exposure range

• 3D Imaging

‣ Use of 2 cameras to generate a 3D image


What’s coming ?


Q&A


References

• http://developer.android.com/

• http://www.codeaurora.org

• http://omappedia.org

• http://source.android.com

• http://stackoverflow.com/questions/10775942/android-sdk-get-raw-preview-camera-image-without-displaying-it

• http://www.cjontechnology.com/blog/?p=14

• http://graphics.stanford.edu/projects/camera-2.0/

http://developer.android.com/

http://www.codeaurora.org/

http://omappedia.org/

http://source.android.com/

http://stackoverflow.com/questions/10775942/android-sdk-get-raw-preview-camera-image-without-displaying-it




http://www.cjontechnology.com/blog/?p=14

http://www.cjontechnology.com/blog/?p=14

http://graphics.stanford.edu/projects/camera-2.0/

http://graphics.stanford.edu/projects/camera-2.0/


About

Aptina enables imaging everywhere.

A leading innovator of CMOS imaging technology, Aptina delivers excellent pixel performance, sensor functionality and camera system capability to a world going visual.

Aptina Camera Software Stack (Access) is Aptina’s implementation of the Android Camera HAL with some custom extensions.

Contact Information

Balwinder Kaur [email protected]

Ashutosh Gupta [email protected]

mailto:[email protected]

mailto:[email protected]

camera 2.0 in android 4.2

Technology

camera image camera

camera classes camera

face camera

parameters camera

area camera

size camera

camerainfo camera

camera snapshot