spatial reasoning in a multi-modal user guide for a complex machine
DESCRIPTION
Delft University of Technology, Faculty of Information Technology and Systems. Spatial reasoning in a multi-modal user guide for a complex machine. Nadejda Soudzilovskaia, Rafael Bidarra, Frederik W. Jansen. Contents. Problem introduction Application Geometry based reasoning Techniques - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/1.jpg)
Spatial reasoning in a multi-modal user guide for a complex machine
Nadejda Soudzilovskaia, Rafael Bidarra, Frederik W. Jansen
Delft University of Technology,Faculty of Information Technology and Systems
![Page 2: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/2.jpg)
Contents
Problem introduction Application Geometry based reasoning Techniques Examples Questions
![Page 3: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/3.jpg)
Problem introduction Multimodal interaction
– Multimodal = speech + graphics + text• Computer speech needs feedback of text and graphics• Speech introduces geometric references,
(e.g.the object in front of the building)
• Solving of these references needs combination of geometric and functional reasoning
![Page 4: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/4.jpg)
?
Application: Interactive fax user guide Speech & Reasoning & Graphics Task: develop dialogue technologies
Application
![Page 5: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/5.jpg)
Single questions & answers about the fax Questions: Speech Answers: Graphics & Speech
How can I make a copy of a document?
• Put the document into the paper source tray
• Press copy button
Fax encyclopedia
![Page 6: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/6.jpg)
Pre-programmed answers– develop set of possible user questions– develop answer + “movie” for every question
Possible solutions
Not an intelligent system. Only pre-cooked question and answers are possible
![Page 7: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/7.jpg)
Possible solutions
Answers derived by reasoning
– knowledge based system containing:
• data about fax functionality
• data about fax construction (3D model)
– Reasoning system based on:
• semantic
• geometry
Answers derived by reasoning
– knowledge based system containing:
• data about fax functionality
• data about fax construction (3D model)
– Reasoning system based on:
• semantics
• geometry
![Page 8: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/8.jpg)
System architecture
Dialogue manager
Speech recognizer
Parser
Geometrybased
reasoning
Semanticbased
reasoning
Graphical answer
generation
Textual answer
generation
?
![Page 9: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/9.jpg)
Graphics
– Solve geometric references
– Derive descriptions of a component location
– Show components
Tasks:
Geometry based reasoning
Tasks:
– Navigation though the virtual world
![Page 10: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/10.jpg)
How can I make a copy of a document?
• Put the document into the paper source tray
• Press copy button
Geometry based reasoning Show components
![Page 11: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/11.jpg)
Geometry based reasoning
– choose appropriate scale– choose appropriate view point
Show components
![Page 12: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/12.jpg)
Where is the power slot?
Power slot is on theback side of the fax
Geometry based reasoning Deriving location descriptions
![Page 13: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/13.jpg)
What is the button left of the start button?
This is a copy button
Geometry based reasoning Solve geometric references
![Page 14: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/14.jpg)
Data about fax construction
– hierarchy graph
– feature graph
– geometry graph
Geometry based reasoning
Top face
Rib
Bottom face
CornerCorner
vertexedgeedge
face face
Telefax
Corpus Telephone Powerplug
Telephonereceiver
Operational panel
LCD screen
Start button
Keypad
....
....
![Page 15: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/15.jpg)
Problem: search for a suitable reference object Proposal: use of hierarchy Result: reduction of reasoning complexity
Techniques
Telefax
Corpus Telephone Powerplug
Telephonereceiver
Operational panel
LCD screen
Start button
Keypad
....
....
![Page 16: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/16.jpg)
Analyze of dimensionality in view of children location
Techniques
Objects that do not have children nodes:• Leaf objects
Objects that have children nodes:
• 3D objects: fax corpus, telephone
• 2D objects: operational panel, slot panel
• 1D objects: cables
• Set objects: group of slots, group of switches
![Page 17: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/17.jpg)
Use of applicability zone in for a geometric reference
• when reason about a position on a surface of another object the spatial reference differ for 3D object and 2D object
• use of fuzzy logic to process zones of applicability
Techniques
![Page 18: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/18.jpg)
Where is the power slot?
Power slot is on theback side of the fax
Examples Deriving location descriptions
![Page 19: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/19.jpg)
1 priority Location on the surface
1.1 on the corner 1.2 on the right / left side 1.3 on the top / bottom face 1.4 on the front / back face
• Search for the object parent
• Select meaningful descriptions
2 priority Location relatively to neighbor objects
2.1 between (two other objects) 2.2 in front of , under, above 2.3 to the left of 2.4 to the left of 2.5 behind, under 2.6. near
Developed answer : The printer slot is on the back face of the fax, to the left of the printer slot
Deriving location descriptions
![Page 20: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/20.jpg)
What is the button left of the start button?
This is a copy button
Examples Solve geometric references
![Page 21: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/21.jpg)
1. Search through the same hierarchy level
1.1 Search for a parent group of operational buttons
1.2 Analyze location left of the start button
1.3. Choose suitable objects
2. Go one level higher in the hierarchy
2.1 Search for a parent of the parentoperational panel
2.2 Analyze location left of the start button
2.3 Select objects
3. Go one level higher……...
Solve geometric references
![Page 22: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/22.jpg)
Summary
Role of spatial reasoning techniques in the application of multi-modal user guides
A fax machine user guide system Untilisng 3D model of a fax Use of hierarchy Use of dimentionality Use of areference applicability zone
![Page 23: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/23.jpg)
Conclusions
Geometry based reasoning is a new functionality for such a dialogue system
3D model of the fax-machine is used as a source of data for reasoning
Several techniques for the geometry-based reasoning were developed and implemented using Java3D and OpenGL languages
![Page 24: Spatial reasoning in a multi-modal user guide for a complex machine](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568150a9550346895dbeba4d/html5/thumbnails/24.jpg)
Questions
?