mobilebench consortium 2013

Copyright© MobileBench Consortium, 2013

MobileBenchMobileBench ConsortiumConsortium

MobileBench

Standard performance metric for Mobile

2013.07


Lack of credible Benchmark tools

• The biggest motivation of developing a new benchmark tool was the lack of credibility, fairness, and accuracy among conventional benchmark tools in the market.

No detailed information for device No detailed information for device comparisoncomparison

Small Test items and coverageSmall Test items and coverage Hard Hard to get feedbackto get feedback Difficult to interpret the resultsDifficult to interpret the results Wide fluctuationWide fluctuation User experiences reflected insufficientlyUser experiences reflected insufficiently

1 2 3 4


Goal for MobileBench consortium

FairnessFairnessEnsure the fairness through opening source-code

ConsistencyConsistencyMinimize the standard deviation of results

+ User Experience+ User Experiencethe performance evaluation based-on user scenario

+ Flexibility+ FlexibilitySupport various evaluation functions for Analysis

DistinctionDistinctionMaximize device performance


MobileBenchTM Strength (1/5)

Avg: 0.7

Avg: 4.4

Avg: 12.0

Avg: 0.6

ConsistencyMinimize the standard deviation of results



DistinctionMaximize device performance

108%

73%

17%12%

Spec.



FlexibilitySupport various evaluation functions for analysis



MobileBenchMobileBenchTMTM

ResultResult

User Experiencethe performance evaluation based-on user scenario

Other ToolOther ToolResultResult

Difference ofstorage

performance

MobileBenchMobileBenchTMTM --UXUXResultResult



System MonitoringMonitor the behavior of each device in the background

Processor Usage & Freq.

Memory Usage

Storage I/O

MFLOPS/MIPS/Memory/StorageMFLOPS/MIPS/Memory/Storage Java-scriptJava-script Web-loading / HTML5Web-loading / HTML5 UI-stressUI-stress

* Time resolution 1sec * Time resolution 1sec


MobileBench Suite

MobileBench MobileBench-UX

MobileBenchMobileBench SuiteSuite

Mobile ApplicationMobile Application Device Benchmark on Device Benchmark on PlatformPlatform Device Benchmark on Device Benchmark on KernelKernel Synthetic WorkloadSynthetic Workload System MonitorSystem Monitor

Mobile App. + Mobile App. + PC AgentPC Agent Mobile Device Control via USBMobile Device Control via USB Real Workload/ApplicationReal Workload/Application ScenarioScenario--basedbased System MonitorSystem Monitor


MobileBench Features

• MobileBench is a total benchmark solution for mobile devices. It can measure the performance of each H/W component as well as web-related and UI performance.

1. System Performance:- CPU/Memory/Storage

2. Web browsing:- Javascript, Web Loading,

HTML5 compatibility

3. User Experience:- U/I Stress, Battery Eater

4. Special Feature:- System Monitor, Email

Reporting

MobileBenchTM Features


MobileBench Features

Features AP Memory Storage Function in Advanced Mode

Processor Core #, Time control

Memory Thread #, Request size

Storage Chunk size, Pre-condition, Native access

Java Script None

Web Loading None

HTML5 Compatible None

Graphic 3D Graphic (TBD)

MobileBenchTM


MobileBench-UX Features

• MobileBench-UX is a PC software to measure UX performance controlling mobile devices with PC via USB connection. It can simulate all the different user scenario, changing the combination of scenario items. Since all the test items are what users actually do every day, this tool can actually represent and score the real user experience.

One Click!

UX based bench

Multitasking bench

Multimedia bench

Automatic reporting

CameraVideo Playback

Video Creating

Application Installation

Application Loading

Media Scanning

I/O Operation

Boot Test


MobileBench-UX Features

Category Item AP Memory Storage Function

Utility

Reboot Rebooting time

Media Scanning Android media scan service time

External File I/O PC to phone file copy time

MTP Test Android media transfer protocol

App. Install Application install time

Multi-media

Video Playback-SMovie frame drop/delay time- Background storage file copy

Video Playback-MMovie frame drop/delay time - Background memory access

Video Creating MPEG-1 video encoding time

Video Trans-coding Movie, MPEG-1 video encoding time

Camera Taking pictures and saving time

Multi-tasking

App. Switching 20-dummy application switching time

Multi-Popup Dual-movie play

Web Browsing Web page loading & zoom in-out

MobileBenchTM-UX


MobileBench Consortium

• Organization Structure

Board of Director(Chair & Directors)

Special Committees Work Committees

CFO

SecretaryPresident

Tool Maintainer

Board of Director

WorkCommittee

Spec. releaseApproval

Spec.Approval

Specification& Tools

Developer Yes Yes Yes Yes Yes

Adopter No Yes No Yes Yes

Honorary* No Yes No No Yes

Non-member No No No NoNo

(Lite version only)

* Non-Profit organization, educational institutes may be participated as “Honorary” membership with limited right for FREE

• Membership types


Consortium Roadmap

• First General Assembly is set for 26th of September 2013 at Shenzhen, China• Open for anyone for 1st General Assembly and launching event of

MobileBench suit 2013 for FREE

2013 2014

2Q 3Q 4Q 1Q

Jun.

Consortium

Development

Sep. Mar.

Milestone

2nd Consortium Meeting

Milestone setup for MobileBench 2014

Sep.Jun. Mar.MobileBench 2013 rev.1.0

1st MobileBenchGeneral Assembly

Release of MobileBench suit 2013

1st BoD @ Beijing

Release of beta MobileBench Suit


MobileBench Consoritium

Register in advance to secure your spot for 1st MobileBench Consoritum General Assembly for FREESeptember 26th 2013, Shenzhen, China(25th for Press & Board of Director meeting)

Hotels and more information will be informed later


Thank youThank you

mobilebench consortium 2013

Technology

device performance

ux performance

time resolution

system performance

performance evaluation

mobilebenchtm strength

web loading

ui performance