qualcomm snapdragon pdf - the 2017 qualcomm® 4g/5g summit | qualcomm ·...

1

Sam Jha Director, Product Management Qualcomm Technologies, Inc.

Creating better mobile user experiences with Qualcomm Technologies’ advanced technologies

2

Immersive experiencesThe experiences worth having, remembering, and reliving

Draw you in…Take you to

another place…Keep you present in the moment…

3

Immersion and intelligence are the new “User Experiences”

Experiences become more realistic, engaging, and satisfying

Connecting your devices at home and work, making them more rewarding

On-device applications that learn, object, face and scene

detection

Smooth, interactive, cognitive user interfaces

Augmented reality-based experiences

New virtual reality applications

Realistic gaming experiences

Theater-quality movies and live sports

4

Better Display & Camera

Better Audio & Voice

Qualcomm® Snapdragon™ processors – enhancing user experienceFocusing on broader dimensions of user experience

Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.

High resolution audioSampling rate and precision3D surround sound and clarity

Pixel quantity and qualityResolution and frame rate Color accuracy, contrast, brightness

“True to life” soundReach out and Touch with your voice

Contextual interactionsIntelligent and personalized

Better content creationSensor, ISP, Resolution, fps, Computational camera, DSC quality

Natural user interfacesResponsive, accurate, and seamless

Visual quality

Sound quality

Intuitive interactions

Immersion&

Intelligence

5

Snapdragon – enhancing visual qualityFocusing on broader dimensions

High resolution audioSampling rate and precision

“True to life” sound 3D surround sound and clarity



Visual quality

Sound quality


Immersion&

Intelligence

Pixel quantity and qualityResolution and frame rate Color accuracy, contrast, brightness

Better content creationSensor, ISP, Resolution, fps, Computational camera, DSC quality

6

Visual quality is much more than resolution and frame rate Color accuracy, contrast, and brightness are also critical elements

Pixel quantity Pixel quality

Resolution

Increased definition and sharpness

Frame rate

Reduced blurring and latency

Color accuracy

More realistic colors through an expanded color gamut, depth, and temperature

Contrast and brightness

Increased detail through a larger dynamic range and lighting enhancements

7

Power management is very important on any displayColor accuracy, contrast, and brightness are also critical elements

Qualcomm EcoPix is a product of Qualcomm Technologies.

Sunlight viewabilityQualcomm®

EcoPix™ features

Assertive Display/ SVI

Better viewing under bright light

Variable RefreshImproved battery life

CABL/ FOSS

Power savings for 30 ~ 50% on LCD or AMOLED

Display Stream Compression

High refresh

CABL/FOSS OFF CABL/FOSS ON

Same Image Quality at lower Backlight or Intensity

Low refresh

Reduce power and cost with VESA DSC

DD

ICW

ith RA

M

LCD

Controller

No C

ompression

FHD panelw/o 4Gbpsw/ 2Gbps

MSM

w/o 16 MB; w/ 8 MB

8

Display related complaints

AMOLED display power is high

Difficult to view in bright conditions

Natural imagery looks unnatural on some large color gamut displays

Display Jank under high system load

Mechanical design challenges for QHD displays

VESA DSC compression

Hardware composition (8 layers)

Color Gamut mapping

Assertive DisplayFOSS

Qualcomm Technologies’ solution

9

Taking an end-to-end approach to improve visual qualityThe journey of light from the mobile camera to a correct pixel on the display is challenging

End-to-end solutionsColor management, artifact removal, optimized click-to-shoot time

Component tuning | Image processing | System optimization

Mobile size constraints create challenges:•  Poor light gathering•  Increased noise

Lens and sensorAddresses camera and display

challenges while enhancing images

SoCPhysical limitations of LCDs and LEDs:•  Partial color gamut•  Limited contrast and brightness

Display

SoC

10

Qualcomm Spectra™ camera ISP

Qualcomm Spectrs is a product of Qualcomm Technologies QTI research..

•  Improved image quality, with more natural skin tones•  Pre & Post processing on HVX for new UX•  Improved power efficiency and higher throughput

Benefits

What’s new

•  Improved image quality •  Enhanced de-mosaicing for new ~1.0u image sensors•  New hardware for radial noise reduction & color artifact

correction•  Camera sensor pre-processing streaming to DSP

•  Improved hybrid autofocus framework (PDAF/Contrast/Laser)•  GPGPU Optical Zoom•  Universal Bandwidth Compression (UBWC)•  Serial C-PHY interface

WithoutQualcomm Spectra ISP

Using Qualcomm Spectra ISP

11

Radial noise reduction

QTI research.

•  New Diffusion Bayer Noise Filter•  Removes Radial Noise from Image

Radial Noise, makes woodsurfaces look unnatural

12

Color artifact correction

QTI research.

Competition SDRawInput

Pink spots

No pink spots

13

35%

28%

28%

28%

27%

Camera quality influences over half of future smartphone selectionsOwners want capabilities that ensure quality pictures

Qualcomm survey with n=1065 smartphone ownersQ16. How important will the camera quality be in influencing the selection of your next smartphone? 5pt – extremely, very, moderately, slightly, not at all. Total N=1065Q5. Which of the following camera features would influence your next smartphone purchase? . N=1065

56%believe camera quality is very/

extremely important in influencing

their next smartphone selection

Top 5 Camera CapabilitiesInfluencing SP Purchase

1

2

3

4

5

Auto-Focus

Fast Auto-Focus

Low-Light Shooting Mode

Image Stabilization

Digital Zoom

14

Smartphones are the most used cameras – even for DSLR & P&S owners

Qualcomm survey with n=1065 smartphone owners

77%

76%

75%

72%

53%

46%

Easy to share

Convenience

Easy to use

Portability

Software/apps

On device editing

Top Smartphone Camera Advantages

72%Smartphone owners take

smartphone pictures at least weekly

1

Device UsageAve. % of pictures taken with device

61%

67%27%

29%

94%

DSLR Owner

P&S Owner

SP Only Owner2

3

4

5

6

15

Qualcomm Technologies, Inc. has a solution for the top issues users facePoor clarity, low light, zoom and focus are the most mentioned pain points

Q6. When taking pictures on your smartphone camera, which if any problems do you have? Total N=1567Qualcomm OptiZoom and Qualcomm ChromaFlash are products of Qualcomm Technologies, Inc.

Blurry picturesBad in low light Limited zoom Flash ruinsPoor/slow focus Overall slow toopen/zoom/take

36% 35% 28% 25% 22% 19%

Top SP Camera Pain PointsWhile taking pictures

SDLLVSDLLP

Qualcomm® Chromaflash™ camera feature

Qualcomm® OptiZoom™ camera

feature

Dual Camera AFPDAF

ZSLFast Start

Touch to TrackDual Camera AF

PDAF


16

Re-focus

Segmentation with various background replacement

Qualcomm Technologies’ dual camera

Bokeh with variable aperture

Instant AF

17

Low light video & photo: Let there be light

•  An algorithm for adaptively brightening areas of video/images that would otherwise appear too dark

BenefitsWhat’s new•  Improves the quality of scenes that are

generally bright, but include dark regions•  Fast and efficient noise-reduction for areas that

were under-exposed in the original frame

Low Light Video Snapdragon Low Light Video

18

Produce stable video despite moving hands/camera

Without EIS 2.0 With EIS 2.0

•  Electronic Image Stabilization (EIS) is an image stabilization feature for video•  The feature takes the sensor input and augments the Digital Image Stabilization (DIS). Resultant Hybrid solution

is substantially better than DIS

−  EIS 2.0 compensates for 3 axis shake −  Removes rolling shutter distortion

19

•  Flash images usually suffer from over-exposedness and bluish tint (unnatural color).•  Our ChromaFlash technology overcomes these problems algorithmically and produces and image that preserves

color, texture, and brightness.

Capture a real life image with ChromaFlash

No-flash image Flash image ChromaFlash Image

20

Dual camera features with even single camera - Qualcomm® UbiFocus™ camera feature

Qualcomm UbiFocus is a product of Qualcomm Technologies, Inc.

Out of focus Out of focus

Input Image 1 Input Image 2 UbiFocus

In focus In focus In focus

This is a post processing feature which takes five captured images as input

How it works:

Each image has its own desired pre-determined lens position as a capture requirement

The algorithm combines five images and output one fused image

The algorithm also generates a “map” that can be used for dynamic focusing

UbiFocus is a camera feature that generates an all-in-focus image from a mobile camera with a limited depth of field. Additionally, this feature allows users to selectively refocus their image in post-processing.

21

Snapdragon – enhancing sound qualityFocusing on broader dimensions

Pixel quantityResolution and frame rate


Pixel qualityColor accuracy, contrast,

and brightness


Visual quality

Sound quality


Immersion&

Intelligence


“True to life” sound3D surround sound and clarityReach out and Touch with your voice

22

Enriching sound by focusing on broader dimensionsProviding truly immersive audio experiences optimized for how humans hear

Sampling rateIncreased sampling rates to match human hearing

3D surround soundAccurate 3D capture and playback of audio

PrecisionIncreased bits-per-sample for increased audio fidelity

High resolution audio “True to life” sound

Clear audioZoom and focus on the important sound while filtering out the noise

23

Using multiple dimensions to capture vivid 3D sound

Understand environmentAutomatically adjust settings based on environment detection and voice identification

Isolate sounds of interestZoom, focus, and track the sounds of interest while separating out other sounds

Capture realistic soundCapture high-resolution, 3D surround sound through object-based or scene-based audio

Object-based sound captureScene-based sound capture

24

Zoom and focus on the important sound while filtering out the noise

Automatically tracks the dominant speaker•  Audio source tracking in a 360° plane−  180° tracking with 2-mics

Useful outside of voice calls

•  Gaming (player 1 vs. player 2)

Source localization & trackingConfigurable listen and quiet zones•  Useful for camcorder

−  Automatically only capture audio from in front of the device, and not the user holding the device

−  Use Audio Zoom to increase the volume of the captured audio as you zoom in

Sound focus & Audio Zoom

1

24

3Zoom-In Area

25

End-to-end solutionsHigh-res audio, noise suppression, and 3D surround sound

Component tuning | Audio processing | System optimization

Size and cost constraints result in:•  Limited number of mics•  Increased noise

MicAddresses mic and speaker challenges

while enhancing audio

SoCSize and cost constraints result in:•  Limited number of speakers•  Reduced quality and fidelity

Speaker

Optimizing sound quality across the systemThe journey of audio from microphone to speaker is challenging

SoC

26

76%68% 66%

60%

49%

36%28%

21%

12%8% 8% 7%

Text Call Browse web

Email SocialNetwork

Listen to Music

Take Pictures

Watch TV/Videos

UseMaps

Record Videos

Mobile Bank

Read eBooks

Voice still the primary function of Smartphones!Nearly three quarters of Smartphone owners make calls daily

Qualcomm survey with n=1065 smartphone ownersQ3. For your current smartphone, how often do you do each of the following activities? 8pt – Several times/day, once/day, several times/week, once/week, 2-3 times/month, once/month, less than once/month, never

Smartphone Activity FrequencySeveral times/Once a day

Base: Total SP O N=1065

27

People still complain about voice on mobile devicesQualcomm technologies to address the challenges

Linear Analog Codec/ HD

Voice/ Full HD Voice

HD Voice/ Full HD Voice

Full HD Voice ECNS Dual/ Multi Mic NS

FENS ANC

Modem Centric Voice Solution/

VoLTE

EVS Channel Aware Mode

SRVCC, VCC, IMS PRO

eAMR, BeAMR, SRVCC


Unnatural voice (lack of listening

comfort)

Voice clarity/

intelligibility

Sense of presence

Echo Background or ambient

noise

Cross talking

Call (set up) delay

Call drops Non-uniform voice quality

28

Always wideband voice (BeAMR) – Qualcomm Technologies customized codec

Received – Side Bandwidth Extension

GSM/WCDMA/TD-SCDMA/FDD/TTD-LTENone-TrFO, Legacy CS RAB & Device Interop

PSTN

AWV (BeAMR)HD Voice User Experience

Cellular Network Wide BandNarrow Band

Validation•  Blind Enhanced AMR (BeAMR) is a receiver bandwidth extension (BWE) of the Enhanced AMR (eAMR)•  BeAMR completes our HD Voice “Always HD” marketing initiatives•  Minimizes impact of lack of AMR-WB coverage to mobile users at all times including in-network non-TrFO M2M, out-of-network M2M, VoLTE SRVCC,

IRAT and even L2M calls

•  Can be a key turn-key voice differentiation feature for operators and OEM vendors by providing uniform HD Voice quality across the entire network

Technology and product development•  BeAMR aka eAMR2.0 belongs to the same codec concept class as eAMR and uses the same codebase

•  BeAMR operates in voice packet domain as compared with typical Blind Bandwidth Extension which generally operates in PCM domain•  BeAMR is commercially available for both CS and VoLTE calls on MSM8994 and onwards.

29

Standard vs. Qualcomm Technologies customized speech codecs Relative voice quality comparison

HDOn is a product of Qualcomm Technologies, Inc.

AMR (12.2)

AMR-WB (12.65)

BeAMR (12.2)

eAMR (12.2)

EVS (13.2)

Standard Codecs HDOn™ codecs

Rela

tive

Subj

ectiv

e V

oice

Qua

lity

•  High CAPEX deployment cost•  Major radio & network upgrades•  Extensive IoT & field testing

•  Minimal CAPEX deployment cost•  Only handset upgrades required•  Interop to legacy RAB & network

WB

NB

SWB

WB

30

Crystal clear voice with Fluence™ noise cancellation technology

Fluence is a product of Qualcomm Technologies, Inc.

Dual-microphone noise cancellation•  High stationary and non-stationary

noise suppression•  Lower avg. vocoder bitrate in

noisy environmentsReceive voice enhancement•  Render far end reference signal more

intelligible in noisy environments•  Increase playback volume in noisy

environments

Wind noise rejection•  Substantially reduces wind noise interference•  Maintains full intelligible communication

in outdoor winds

Far-end noise suppression•  Suppress stationary and non-stationary

noise coming from far-end and enhance DL voice quality

•  Level of noise suppression can be adjusted on the fly

WideVoice•  Creates wideband-like

extended speech•  Blind Bandwidth Extension applied

to all narrowband speechEZHearing (Slow Talk)•  Slows down speech without

affecting perceived characteristics •  Increases speech intelligibility

for seniors and foreigners

31

Hear HD quality voice with Fluence HD™Fluence Solution – part of Fluence HD package

Snapdragon Voice Activation (SVA)•  Use voice to start a device’s voice

natural user interface–  No need to press a button, Just say

“Hey Snapdragon” to start the voice NUI

•  Voice Activation detects the keyword and starts OS’s voice NUI

•  Designed to only respond to the owner of the device

Adaptive Active Noise Cancellation•  Adaptive handset ANC cancels out

background sound using the earpiece

•  Improves voice intelligibility while in a loud environment

•  AANC offers robustness across different handset positioning

•  Low Power, Low cost implementation

“Always Wideband Voice” – Anytime, Any device, Any network•  HD voice on legacy narrow band

•  Customized end-to-end differentiated turnkey codec technologies which seamlessly can transform legacy narrowband network to HD Voice

•  You “always” hear HD on Snapdragon

32

Ultimate voice experience with Fluence Pro

Robust echo cancellation & noise suppression solution for tablets & smartphones•  Voice calls, VoIP calls, Voice recognition,

Video conferencing etc.•  Handheld, desktop, round table

speakerphone use cases•  Portrait and landscape mode usages

•  Supports 3 & 4 mic configurations

-  Flexible mic placement for NS (broadside or front-back)

Generic speakerphone support•  Automatically tracks speaker movements

and provides consistent voice quality in dynamic scenarios

•  Consistent non-stationary noise reduction that is robust to holding pattern variations-  No degradation of non-stationary noise

reduction and retains similar performance as vertical holding

Surround sound recording•  5.1 audio capture for full HD Camcorder

experienceSource position tracking Sound Focus and Audio Zoom•  Details on next page

33

Snapdragon – enhancing device interactionFocusing on broader dimensions


Pixel quantityResolution and frame rate

“True to life” sound 3D surround sound and clarity


Pixel qualityColor accuracy, contrast,

and brightness


Visual quality

Sound quality


Immersion&

Intelligence

34

Intuitive interactions immerse us in the moment

Natural user interfaces•  Seamless•  Responsive•  Accurate

Contextual interactions•  Intelligent•  Personalized

Providing the appropriate interfaces based on the user, device, and

application

Providing the right level of immersion based on context

35

Natural user interfaces for intuitive interactions

Adaptive, multimodal, user interfaces

Voice recognition Use natural language

processing

Motion & gesture recognitionUse computer vision, motion sensors, or touch

Face recognitionUse computer vision to recognize facial expressions

Personalized interfacesLearn and know user preferences

based on machine learning

Eye tracking Use computer vision to measure point of gaze

Bringing life to objectsGive user interfaces to headless objects

36

Providing the right level of immersion based on contextTrusting your device to take appropriate actions

Contextual awareness•  Scene recognition•  Sensors•  Learned preferences

Personalized experiences•  Reduced friction•  Personal assistant

Intelligent notifications•  Right information at the right time•  Interrupt immersion appropriately

37

Enabling new user experiences with Snapdragon

38

Devices, machines and things are becoming more intuitive Through new levels of on-device intelligence and cognitive capabilities

Reasoning Action Perception Act intuitively based on context

and user preferences Learn, anticipate, infer context

through machine learning Hear, see, and observe like humans

through sensors, computer vision

39

Offering a broad range of on-device cognitive capabilities

Qualcomm Snapdragon Sense and Qualcomm Snapdragon Smart Product are products of Qualcomm Technologie’s, Inc.

Intuitive security•  Qualcomm® Snapdragon™ Smart Protect•  Qualcomm® Snapdragon Sense™ ID biometrics

Visual perception•  Scene Detect & Gallery

Classification•  Snapdragon Text Activation

Voice and audio understanding

•  Qualcomm® Snapdragon Sense™ ID Voice

Always-on awareness•  Smart Connectivity

Immersive multimedia•  Immersive Audio

On-device machine learning

Computer vision

Ultra-low powerSensor processing

Cognitive technologies

40

Supporting cognitive technologies on the device is key for many use cases

Enhanced reliability

Immediate response

Efficient use of network bandwidth

Security and user privacy

41

Smart audioImmersive audio

•  Immersive 3D surround sound

Benefits

What’s new

•  Perceived multi-channel surround through directional speaker virtualization

42

Smart visualsScene detect & gallery classification

•  Near real-time image classification•  No data sent off device

Benefits

What’s new

• Machine-learning automatically and intelligently recognize scenes

•  Recognition done on-device, quickly and privately

43

Smart visualsSnapdragon text activation

•  Intelligently engage with text as if it were already a link in your phone

Benefits

What’s new

•  Recognize and Act on text in the real world•  Support for Korean, English, Chinese with

more languages planned

Call NumberCompose E-mailOpen URL

Save ContactSearch WebNavigate

44

•  Advanced water rejection and wet multi-finger tracking•  Ultra low power mode•  Improved touch responsiveness•  Context awareness and sensor fusion•  Improved UX and advanced algorithms through

Snapdragon host processing

What’s new

•  Passive stylus and gloved touch•  Superior cost to performance ratio•  Improved time to market

Benefits

Qualcomm® improveTouch™ technologyCreating the next generation touch experience

Qualcomm improveTouch technology, a feature of Qualcomm Snapdragon processors, is a product of Qualcomm Technologies, Inc.

45

Snapdragon Smart Protect technologyReal-time, on-device behavioral analysis platform

Qualcomm Zeroth is a product of Qualcomm Technologies, Inc.

•  Machine learning-based behavioral analysis•  Based on the Qualcomm® Zeroth™ cognitive

computing platform•  Almost instant detection and classification of

previously unknown malware

What’s new

Benefits

•  Helps protect private, valuable data•  Seamless user experience with negligible overhead•  Strengthens signature-based anti-malware apps•  Automatically adapts to each device

Traditional signature-based

detection

Snapdragon Smart Protect

behavioral-based detection

Comprehensive Protection

Safe content

Transformed malware

Unknown malware

Known malware Abnormal behavior

Photos taken while display is off

Abnormal behavior

SMS sent without user interaction

46

Adaptive smart connectivitySwitch between best connection intelligently and automatically with Connectivity Engine (CnE)

Signal Quality

LTE

Wi-Fi

Network Load vs. Capacity

Internet Reachability

Captive Portal

Internet

AGREE

Terms

Backhaul Throughput

Internet

100 Mbps 5 Mbps

Many variables impact which connection is “best”

3G/LTE

Wi-Fi

Operator Policy

3G/LTE HW

Wi-Fi HW

Internet-connected App

CnE

47

Wrap up

48

•  One of the most advanced connectivity features in a mobile SoC including Cat 12/13 LTE and 802.11ad Wi-Fi supports high performance mobile computing

•  Zeroth – the on-device machine learning platform powering cognitive applications to simplify and enhance user experiences

•  Images & camcorder footage consistently delighting end users with quality, crispness, color and simplicity to capture−  Intelligently focused on the subject, regardless of setting or situation

−  Optimized settings and simplified operation for improved footage and image capture quality

−  Highly efficient components working together to achieve higher performance per milliwatt, through tight hardware and software integration while delivering feature innovation

−  Immersive mobile movie experiences

•  High performance display processing subsystem delivers peak display graphics while reducing power and thermals−  4K wireless display performance rivaling modern living room experiences

−  Console quality graphics

•  Platform with strong support for advanced applications like Tango as well as other AR and VR integrations

Snapdragon – mastering Smartphone user experienceCombining immersion and intelligence to provide OEM’s a more creative platform solution

49

Efficient heterogeneous computing architecture

Within device constraints

Development time / Sleek form factor / Power and thermal efficiency / Cost

Qualcomm Technologies, Inc. is uniquely positioned to deliver superior immersive experiencesProviding an efficient, comprehensive solution in the Snapdragon 820

Visual quality

Immersive experiences

Sound quality


Commercialization

Consistent, accurate color

In-focus images

Low-light video & photography

3D surround sound

Noise removal

Enlarged sweet spot

Seamless, responsive, smooth user interfaces

Intelligent, contextual interactions

Via Snapdragon™ solutions

Via ecosystem enablement

Comprehensive solutions across tiers

Custom designed processing engines

Snapdragon development platforms

Ecosystem collaboration

App developer tools

50

For more information, visit us at: www.qualcomm.com & www.qualcomm.com/blog

Nothing in these materials is an offer to sell any of the components or devices referenced herein.

©2013-2015 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Snapdragon, ChromaFlash and Fluence are trademarks of Qualcomm Incorporated, registered in the United States and other countries, used with permission. EcoPix, Qualcomm Spectra, OptiZoom, UbiFocus, HDOn, Snapdragon Sense, improveTouch and Zeroth are trademarks of Qualcomm Incorporated. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable.

Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.

Thank youFollow us on:

qualcomm snapdragon pdf - the 2017 qualcomm® 4g/5g summit | qualcomm ·...

Documents