using common sense reasoning to create intelligent mobile applications software agents group mit...

Using Common Sense Reasoning to Create Intelligent Mobile Applications

Software Agents GroupMIT Media Lab

The PC vs. the Smart Phone

Current efforts to develop software for smart phones are based on taking PC applications

and making them smaller

The PC vs. the Smart Phone

The PC has an email application The smart phone has a smaller one

Unlike the PC, text entry on smart phones is very difficult

The PC vs. the Smart Phone

Because of the limited IO, the smart phone needs to be better at understanding the user

then the PC ever was

The PC vs. the Smart Phone

Unlike the PC which stays on its desk, people carry their smart phones with them everywhere

The PC vs. the Smart Phone

Because of this, the smart phone needs to be better at understanding the world then the PC

ever was

The PC vs. the Smart Phone

To understand the user and understand the world, smart phone applications need something

that computers have always lacked, common sense reasoning

This is a problem we are trying to solve

Introduction to Common Sense Reasoning

Over the past 3 years 12,000 people have entered common sense facts to a Web site called Open Mind

Introduction to Common Sense Reasoning

Open Mind now contains over 650,000 common sense facts about the world

Introduction to Common Sense Reasoning

Using this knowledge base we have created a semantic network of concepts and relationships called OMCSNet

Using Common Sense Reasoning in Mobile Applications

We are currently using OMCSNet in two cell phone based applications

Using Common Sense Reasoning in Mobile Applications

A language translation aid called GloBuddy 2

Using Common Sense Reasoning in Mobile Applications

And a text entry system capable of contextually relevant word completion

GloBuddy 2

GloBuddy 2

When Traveling in a foreign country, people often rely on phrase books. However:

-They only cover a very limited number of common situations

-Even a common situation will quickly deviate from the book’s script

GloBuddy 2

Translation Software and dictionaries currently exist for PDA devices

But users must write out the entire phrase they wish to translate, and look up each word individually

GloBuddy 2 7 / 14

GloBuddy 2 will translate phrases for the user into English

But GloBuddy can go beyond simple translation and understand the context of the user’s situation

GloBuddy 2

Instead of writing the entire phrase they wish to translate into a foreign language, GloBuddy asks the user to write the ideas they would like to talk

about.

GloBuddy then uses common sense reasoning to build a localized and contextually relevant

vocabulary of terms for the user

GloBuddy 2

1. GloBuddy asks the user to describe the ideas they would like to talk about

2. GloBuddy then uses OpenMind and OMCSNet to build a localized vocabulary of terms

3. Contextually relevant words and phrases are displayed to the user for use in conversation

GloBuddy 2

To return to our example in the introduction:

GloBuddy 2

Because GloBuddy can intelligently expand on the user’s translation request:

The amount of text entry is significantly reduced

And Conversations are more fluid

GloBuddy 2

There is a traditional trade off between a system’s breadth of information, and its depth of reasoning:

Breadth of Information

Dep

th o

f R

eason

ing

Phrase Book

Dictionary

Human Translator

GloBuddy 2

GloBuddy 2 is able to avoid this tradeoff due to its extensive knowledge of common sense facts

Breadth of Information

Dep

th o

f R

eason

ing

Phrase Book

Dictionary

Human Translator

GloBuddy 2

Word Completion

Using Common Sense Reasoning in Mobile Applications

Current approaches to reducing text entry on Smartphones are limited

Using Common Sense Reasoning in Mobile Applications

Using common sense reasoning, we can perform intelligent word completion based on the context of

the message

Using Common Sense Reasoning in Mobile Applications

Some technical notes:

• Unlike GloBuddy, OMCSNet has to be stored locally

• This approach is best used in addition to classic word completion, rather then replacing it

http://agents.media.mit.edu/projects/mobile