spatial reasoning in a multi-modal user guide for a complex machine nadejda soudzilovskaia, rafael...
Post on 21-Dec-2015
216 views
TRANSCRIPT
Spatial reasoning in a multi-modal user guide for a complex machine
Nadejda Soudzilovskaia, Rafael Bidarra, Frederik W. Jansen
Delft University of Technology,Faculty of Information Technology and Systems
Problem introduction Multimodal interaction
– Multimodal = speech + graphics + text• Computer speech needs feedback of text and graphics• Speech introduces geometric references,
(e.g.the object in front of the building)
• Solving of these references needs combination of geometric and functional reasoning
?
Application: Interactive fax user guide Speech & Reasoning & Graphics Task: develop dialogue technologies
Application
Single questions & answers about the fax Questions: Speech Answers: Graphics & Speech
How can I make a copy of a document?
• Put the document into the paper source tray
• Press copy button
Fax encyclopedia
Pre-programmed answers– develop set of possible user questions– develop answer + “movie” for every question
Possible solutions
Not an intelligent system. Only pre-cooked question and answers are possible
Possible solutions
Answers derived by reasoning
– knowledge based system containing:
• data about fax functionality
• data about fax construction (3D model)
– Reasoning system based on:
• semantic
• geometry
Answers derived by reasoning
– knowledge based system containing:
• data about fax functionality
• data about fax construction (3D model)
– Reasoning system based on:
• semantics
• geometry
System architecture
Dialogue manager
Speech recognizer
Parser
Geometrybased
reasoning
Semanticbased
reasoning
Graphical answer
generation
Textual answer
generation
?
Graphics
– Solve geometric references
– Derive descriptions of a component location
– Show components
Tasks:
Geometry based reasoning
Tasks:
– Navigation though the virtual world
How can I make a copy of a document?
• Put the document into the paper source tray
• Press copy button
Geometry based reasoning Show components
Where is the power slot?
Power slot is on theback side of the fax
Geometry based reasoning Deriving location descriptions
What is the button left of the start button?
This is a copy button
Geometry based reasoning Solve geometric references
Data about fax construction
– hierarchy graph
– feature graph
– geometry graph
Geometry based reasoning
Top face
Rib
Bottom face
CornerCorner
vertexedgeedge
face face
Telefax
Corpus Telephone Powerplug
Telephonereceiver
Operational panel
LCD screen
Start button
Keypad
....
....
Problem: search for a suitable reference object Proposal: use of hierarchy Result: reduction of reasoning complexity
Techniques
Telefax
Corpus Telephone Powerplug
Telephonereceiver
Operational panel
LCD screen
Start button
Keypad
....
....
Analyze of dimensionality in view of children location
Techniques
Objects that do not have children nodes:• Leaf objects
Objects that have children nodes:
• 3D objects: fax corpus, telephone
• 2D objects: operational panel, slot panel
• 1D objects: cables
• Set objects: group of slots, group of switches
Use of applicability zone in for a geometric reference
• when reason about a position on a surface of another object the spatial reference differ for 3D object and 2D object
• use of fuzzy logic to process zones of applicability
Techniques
Where is the power slot?
Power slot is on theback side of the fax
Examples Deriving location descriptions
1 priority Location on the surface
1.1 on the corner 1.2 on the right / left side 1.3 on the top / bottom face 1.4 on the front / back face
• Search for the object parent
• Select meaningful descriptions
2 priority Location relatively to neighbor objects
2.1 between (two other objects) 2.2 in front of , under, above 2.3 to the left of 2.4 to the left of 2.5 behind, under 2.6. near
Developed answer : The printer slot is on the back face of the fax, to the left of the printer slot
Deriving location descriptions
What is the button left of the start button?
This is a copy button
Examples Solve geometric references
1. Search through the same hierarchy level
1.1 Search for a parent group of operational buttons
1.2 Analyze location left of the start button
1.3. Choose suitable objects
2. Go one level higher in the hierarchy
2.1 Search for a parent of the parentoperational panel
2.2 Analyze location left of the start button
2.3 Select objects
3. Go one level higher……...
Solve geometric references
Summary
Role of spatial reasoning techniques in the application of multi-modal user guides
A fax machine user guide system Untilisng 3D model of a fax Use of hierarchy Use of dimentionality Use of areference applicability zone
Conclusions
Geometry based reasoning is a new functionality for such a dialogue system
3D model of the fax-machine is used as a source of data for reasoning
Several techniques for the geometry-based reasoning were developed and implemented using Java3D and OpenGL languages