the connected world and speech technologies - ortra - afeka conference (07-07... · at&t...

The Connected World and Speech Technologies: It’s About “Effortless”

© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.

Jay Wilpon

Executive Director, Natural Language Processing and Multimodal Interactions Research

IEEE Fellow, AT&T Fellow

Jay Wilpon – 7/7/2014

© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property. 2

“In the Beginning…” Genesis 1:1



3 Front view of the spoken digit recognizer (J. Suzuki & K. Nakata, Radio Research Labs, Japan, 1961)


“It would be too simple to say that work in speech recognition is carried out simply because one can get money for it. . . .The attraction is perhaps similar to the attraction of schemes for

turning water into gasoline, extracting gold from the sea, curing cancer, or going to the moon”


John R. Pierce, Executive Vice President at AT&T Bell Labs (1969)



Good Morning America – 1987


Virtual Assistants – Siri / Google Now


Start at 37 seconds


http://www.youtube.com/watch?v=z4LvJOFTlGo







AT&T Evolution of Speech Industry Evolution

1990 2000

Simple Commands

• Command & Control

(“Press or say”)

“Please say…”

“Collect call please”

“The New Frontier”

2010

Invention of Industrial Leading Technologies & Services “The New Frontier”

Beyond 1980

Pioneered speech service industry

Intelligent Multimodal Services

• Home automation

• Connected car

• Cloud API

• Big data analytics

• Emerging devices, wearables

• M2M

• Video and television

“You are running late for your meeting and you need gas”

“Thanks, can you find a good restaurant for Sunday brunch”

Seamless communications, information access, commerce –

Connected Life

Large-vocabulary

Speech Services

• Operator services

• Enterprise services

Natural Language

Services

• “How May I Help You”

Customer care


• Watson Speech mashup

Resulted in billions of dollars in savings

Paradigm shift in user experience and automation

“AT&T, how may I help you?”

“I have a problem with my bill”



Devices

App Developers

Industries Connected Car

Home Automation

Customer Service

Healthcare

Finance

Consumer Applications

Machine Learning

Dialog Manager

Video Language Processing

Text to Speech

Speech Recognition

Voice Biometrics

3rd Party Developers

Solution Providers Systems

Integrators

Large Enterprises

Speech and multi-modal technologies universally accessible to all devices and industries through a wide ecosystem of partners

Connected World Ecosystem

Cloud Providers

Security



Multimodal Virtual Assistants

• Cloud-based virtual assistant manifests itself appropriately based on the device and environment

- Personalized - Adaptive - Connected

• Natural and effective

- Spoken dialog - Language

understanding - Multimodal interaction

Web

Office

Social Media

Retail

Emerging Devices

Connected Car

Digital Home

Personal Data

Effortless access to information, services, transactions – anywhere, anytime


The Science Behind the Service

Speech and Language

Technology Suite

Natural Language

Processing

Speech Recognition

Dialog

Text to Speech

Language Generation

Biometrics

Speech Translation

Machine Learning


Speech Industry is Approaching a Cross Road

Market Trends Market Challenges

Hello

Hola

• Growth in connected devices (smartphones, tablets,

connected cars, wearables, home automation)

• Increased adoption of and access to cloud computing

• Desire to enhance and differentiate the consumer

experience (virtual assistants, multimodal solutions)

• Demand for business intelligence uses increasing

(speech, video, text)

Low customer adoption due to poor user experience

and performance

Low accuracy “in the real world”

Complexity due to variety of languages and devices

Competing / complementary input technologies

(touchscreen, keyboard, multi-touch and visual

gestures)

Slow development/adaption of applications


Its all about “Effortless”


• Scientists / Engineers

• Software vendors

• Platform vendors

• Application developers

• Customers


Speech/Language Industry Ecosystem - Today

• Multiple code bases • Multiple version • Distribution • Integration • Testing

Expensive, Inefficient and Slow

Software Vendors

• Hardware management • Integration • Optimization • Testing • Version code • Prof. Services • Distribution • (Oh, did I add testing?)

Expensive and Slow

Platform Vendors

• Software maintenance • Platform dependence • Testing • Optimization • Version code • Prof. Services

Slow

App Developers Invention

• Ideas • Experiments • Trial • Ideas • Experiments • Trial • …

“Inspiration time”


Speech/Language Industry Ecosystem - Today



Slow





Its all about “Effortless” – Invert the Food Chain


• Customers – Intuitive, useful, secure, personalized, “anywhere, anytime”, device agnostic

• Application developers – technologies easy to integrate, scalable tools, simple interfaces, limited need for professional services

• Platform providers – technologies easy to integrate, interfaces simple and stable, tools

• Software Vendors – platform independence, scalable engines, tools, simple and stable interfaces

• Scientists / Engines – “it works”, fast to market, platform semi-independence, easy to update

Getting to ”Effortless” – Start with the Customer



Fast




Innovative, Fast, Device Agnostic


Cloud Computing

Bandwidth

Network Connectivity


Speech/Language Industry Ecosystem - Moving

19

Cloud Engines APIs and libraries

• “One” customer • “Single” code base • Low maintenance • Simple testing • Virtual ‘on-demand’ machines • Optimized • Personalization • Pricing << Software licensing

Fast

Innovative, Fast, Device Agnostic

Invention




Fast

App Developers


Aggressive Market Segmentation


Google Apple Microsoft Nuance Amazon AT&T

Device/OS dependent

Speech APIs

Speech APIs

Device/OS independent

Google Now

Cortana Siri Kindle Nina mVA

Android Windows iOS Android “All” “All”


Getting to ”Effortless” – Independence

21

Cloud Engines APIs and Libraries

Android Windows iOS Linux Other

Application Developers



Getting to ”Effortless” – Scientists and Engineers


Are we in the … ?

“… Golden Age of Engineering for Speech Technologies”

“… Age of Invention for Language Technologies”

Data is driving innovation…


Technical Challenges to Move Industry Forward

1990 2000

Simple Commands

• Command & Control

(“Press or say”)

“Please say…”

“Collect call please”

“The New Frontier”

2010

Invention of Industrial Leading Technologies & Services “The New Frontier”

Beyond 1980

Pioneered speech service industry

Intelligent Multimodal Services

• Home automation

• Connected car

• Cloud API

• Big data analytics

• Emerging devices, wearables

• M2M

• Video and television

“You are running late for your meeting and you need gas”

“Thanks, can you find a good restaurant for Sunday brunch”

Seamless communications, information access, commerce –

Connected Life

Large-vocabulary

Speech Services

• Operator services


Natural Language

Services

• “How May I Help You”

Customer care


• Speech mashup

Resulted in billions of dollars in savings

Paradigm shift in user experience and automation

“AT&T, how may I help you?”

“I have a problem with my bill”



Seamless communications, information access and commerce – anywhere & anytime

• Phones/tablets/PCs: unified access to communications, information and transaction

• Enterprise customer care moving from IVR to mobile web

• E-commerce/retail sales

• Connected car & home

• Entertainment: easy access to content everywhere

• Wearable devices: new industry, hands free control

Market Opportunities

Multimodal Virtual Assistants

Customer Care

Entertainment

Enterprise Verticals

Wearables Home Automation

Connected Car


Seamless communications, information access and commerce – anywhere & anytime

• It has to work: Everywhere, be intuitive, be available whenever needed

• Always on: Handsfree, ROBUST, speech detection, sensors everywhere

• Multimodality: Connected devices need multimodal technologies

• Context-awareness: Pragmatics of the world around each user

• Conversational interaction: Effortless user experience require cooperative, personalized & multimodal human-machine dialog

• Secure and Personalized: Knows “its me” and learns about “me” – adaptive

• Proactive: Reasoning, M2M, not just Q/A

• Incrementality: Users expect fluid/engaging interaction

• Rapid application authoring: Enterprises demand easy ‘cold start’ for new domains and rapid support for ongoing improvements (e.g., tools); unsupervised (semi-supervised) training

• Internationalization: multilingual, translation

• Data, data, data: Vast amounts (content, personal, dynamic) – ingest, model, adapt, learn

Challenges Addressed by Scientists/Engineers

Multimodal Virtual Assistants – Getting to Effortless



AT&T Watson Multimodal Intelligent Virtual Assistant


AT&T Watson Multimodal Intelligent Virtual Assistant Planning a “night out on the town” – events, movies and restaurants

Multimodal input and output – “effortless”

– Supports recognition, understanding, generation and

gesture

– Integration of ASR, NLP and Dialog maximizes performance

– Natural conversation supports contextually-aware NLU and

reference resolution

– Supports Targeted Clarification: error handling through

machine learning and natural language generation

Built for rapid authoring and adaptation

– Dynamic hierarchical language models (DHLM) enable ‘cold

start’ for new domains and rapid adaptation

– Bootstrap new domains with generic NLP pipeline

– Watson’s Semantic Abstraction Layer (SAL) facilitates easy

expansion to new domains and new content

Watson VA Features mVA Multimodal Concierge

19


Driving to Intelligence

Data

Natural Language

Processing

Speech Recognition

Dialog

Text to Speech

Language Generation

Biometrics

Speech Translation

Machine Learning


Natural Language Processing (NLP)

NLP

Analysis

Intent classification – semantics

Named entities

Part-of-speech

Syntactic parse

Coreference labels

Text normalization

Sentiments

Topic detection

Information extraction

Emotion detection

Anomaly detection

Summarization

Sentence planning

System intent

Semantic frames

NLP

Generation


Intelligence – Information extraction

This authentic Venezuelan food is delicious homemade and great on price and service

Input sentence

Classify

Identify triggers

Identify related targets

Summarize

Sentiment/No Sentiment

This authentic Venezuelan food is delicious homemade and great on price and service

This authentic Venezuelan food is

delicious homemade and great on

price and service

Aspect/attribute Opinion Venezuelan food authentic

delicious homemade

Venezuelan food: Price great

Venezuelan food: Service

Sentiment analysis

My phone is not calling or receiving messages This girl told me Verizon is down

Problem/No Problem

My phone is not calling or receiving messages This girl told me Verizon is down

My phone is not calling or receiving

messages This girl told me Verizon is

down

Problem object Problem My Phone not calling or receiving

Verizon down

Problem Detection Basic Steps



“In the Beginning…” Genesis 1:1


Rethink Possible



the connected world and speech technologies - ortra - afeka conference (07-07... · at&t...

Documents