1 speech, ink, and slides: the interaction of content channels richard anderson crystal hoyer craig...
Post on 21-Dec-2015
217 views
TRANSCRIPT
![Page 1: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/1.jpg)
1
Speech, Ink, and Slides: The Interaction of Content Channels
Richard AndersonCrystal HoyerCraig PrinceJonathan SuFred VideonSteve Wolfman
![Page 2: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/2.jpg)
2
Background
Content channels simply refers to the various sources of information in some context (e.g. audio, slides, digital ink, video, etc.)
Our focus is on the use of digital ink in the classroom setting
We want to capture/playback/analyze these channels intelligently
![Page 3: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/3.jpg)
3
Why do we want to analyze content channels?
We want to make it easier to interact with electronic materials Better search and navigation of
presentations Accessibility for the
hearing/learning/visually impaired Generating text transcripts Recognizing high level behaviors
![Page 4: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/4.jpg)
4
Distance Learning Classes
![Page 5: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/5.jpg)
5
Classroom Presenter
General tool for giving presentations on the Tablet PC
Many similar systems – our findings applicable to all such systems
Enables writing directly on the slides Tablet PC enables high-quality digital ink Used in over 100 courses so far Allows us to collect real usage data
![Page 6: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/6.jpg)
6
Questions We Wanted to Explore
High Level Question: What is the potential for automatic analysis of archived content?
Other Questions: How well can digital ink be recognized by itself? How closely are different content channels tied
together? Speech and Ink? Ink and Slide Content?
Can we identify high level behaviors by analyzing the content channels?
![Page 7: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/7.jpg)
7
Research Methodology
1. We wanted to understand what real presentation data is like
2. We collected several 100’s of hrs. of recorded lectures from distance learning classes
3. Analyzed the data in various ways to help answer our guiding questions.
• Note: All examples given here are from real presentations!
![Page 8: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/8.jpg)
8
Outline
Motivation Handwriting Recognition Joint Writing and Speech Recognition Attentional Mark Identification Activity Inference: Recognizing
Corrections
![Page 9: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/9.jpg)
9
Handwriting Recognition
Classroom lectures on Tablet PC offer interesting challenges for handwriting recognition Somewhat Awkward
• Small Surface to Write On• Bad Angle to the Tablet PC
Hastily Written• Concentrating on Speaking• Excited / Nervous
![Page 10: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/10.jpg)
10
Recognition Examples
The Good:
The Bad:
The Ugly:
![Page 11: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/11.jpg)
11
Recognition Procedure
Studied isolated words/phrases written on slides
Removed all non-textual ink Fed through the Microsoft Handwriting
Recognizer No training done!
![Page 12: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/12.jpg)
12
Handwriting Recog. Results
Exact Alternate Close None
Prof. A 16 (88%) 1 (6%) 0 (0%) 1 (6%)
Prof. B 146 (59%) 26 (10%) 6 (2%) 71 (29%)
Prof. C 18 (42%) 5 (11%) 1 (3%) 19 (44%)
Prof. D 262 (61%) 45 (11%) 9 (2%) 111 (26%)
Prof. E 408 (79%) 46 (9%) 2 <(1%) 58 (11%)
Total 850 (68%) 123 (10%) 18 (1%) 260 (21%)
![Page 13: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/13.jpg)
13
Outline
Motivation Handwriting Recognition Joint Writing and Speech Recognition Attentional Mark Identification Activity Inference: Recognizing
Corrections
![Page 14: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/14.jpg)
14
Joint Writing and Speech Recognition
Co-expression of ink and speech Is digital ink spoken as it is written?
Yes, but how often? How “closely” to the written text?
Can speech be used to disambiguate handwriting?
Can handwriting be used to disambiguate speech? (incl. deictic references)
![Page 15: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/15.jpg)
15
Examples
Difficult for Speech and Ink Recognition
Difficult Written Abbreviations
Speech/Ink Used to Disambiguate Ink/Speech
![Page 16: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/16.jpg)
16
Experiment
Examined instances of isolated word writing Selected word writing episodes at random
but uniformly from the various instructors Generated transcripts manually from the
audio Checked whether the instructor spoke the
exact word written Measured the time between the written and
spoken word
![Page 17: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/17.jpg)
17
Speech/Text Co-occurrence Results
Exact Approx None Simul 0-2s > 2s
A 1 (100%) 0 (0%) 0 (0%) 1 (100%) 0 (0%) 0 (0%)
B 9 (75%) 3 (25%) 0 (0%) 12 (100%) 0 (0%) 0 (0%)
C 9 (82%) 2 (18%) 0 (0%) 10 (91%) 1 (9%) 0 (0%)
D 12 (86%) 2 (14%) 0 (0%) 10 (71%) 4 (29%) 0 (0%)
E 9 (56%) 7 (44%) 0 (0%) 7 (44%) 4 (25%) 5 (31%)
Total 40 (74%) 14 (26%) 0 (0%) 40 (74%) 9 (17%) 5 (9%)
![Page 18: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/18.jpg)
18
Outline
Motivation Handwriting Recognition Joint Writing and Speech Recognition Attentional Mark Identification Activity Inference: Recognizing
Corrections
![Page 19: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/19.jpg)
19
Attentional Mark Identification
Attentional Marks are… First step is to Identify a stroke as a
mark Tying Attentional Marks to slide
content is important Attentional Ink provides a concrete link
between speech and slide content!
![Page 20: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/20.jpg)
20
Example
![Page 21: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/21.jpg)
21
Method
Segmentation Few strokes Close spatial and temporal proximity
Mark Recognition Created hand tuned classifiers for:
Circles, Lines, Bullets/Ticks Matched with slide content
![Page 22: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/22.jpg)
22
Experiment
1. Identified and Classified Attention Marks by Hand
Two different people per slide Identified type of mark as well as slide
content mark referred to
2. Identified Attention Marks Automatically
3. Compared Resulting Identification
![Page 23: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/23.jpg)
23
Content Matching Issues
Hard to determine exactly what content a mark refers to
![Page 24: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/24.jpg)
24
Content Matching Cont.
Granularity of content parsing can be an issue
![Page 25: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/25.jpg)
25
Attentional Ink Recognition Accuracy
Exact Exact to Punctuation
Close Non-Match
Circles 70 (66%) 13 (12%) 6 (6%) 17 (16%) 106
Underlines 207 (61%) 22 (6%) 44 (13%) 66 (20%) 339
Bullets 52 (60%) 0 (0%) 0 (0%) 35 (40%) 87
329 (62%) 35 (7%) 50 (9%) 118 (22%) 532
![Page 26: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/26.jpg)
26
Outline
Motivation Handwriting Recognition Joint Writing and Speech Recognition Attentional Mark Identification Activity Inference: Recognizing
Corrections
![Page 27: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/27.jpg)
27
Recongizing Corrections Why?
Want to answer the broad question: - “Can we recognize patterns of activity by analyzing the ink and speech channels?”
Useful for Presenters- Occurs frequently (about 1-3 per lecture)
But Non-trivial
![Page 28: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/28.jpg)
28
Recognizing Corrections
Identified Six Types of Corrections
![Page 29: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/29.jpg)
29
Example Results
![Page 30: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/30.jpg)
30
Wrap-up
We wanted to understand the nature of real data to direct our focus when building tools for automatic analysis
Our studies provided the necessary understanding to accomplish this
![Page 31: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/31.jpg)
31
Wrap-up (Cont.)
Specific Results: Basic handwriting recognition is
surprisingly good Very strong co-occurrence of written and
spoken words We were able to identify attentional
marks and the content associated with them
Activity Recognition: There are certain high-level activities that we can identify
![Page 32: 1 Speech, Ink, and Slides: The Interaction of Content Channels Richard Anderson Crystal Hoyer Craig Prince Jonathan Su Fred Videon Steve Wolfman](https://reader035.vdocuments.mx/reader035/viewer/2022062715/56649d6d5503460f94a4d96f/html5/thumbnails/32.jpg)
32
Questions?
Classroom Presenter Websitehttp://www.cs.washington.edu/education/dl/presenter/