![Page 1: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/1.jpg)
Speech and Language Processing for Adaptive Training
Diane Litman
Professor, Computer Science Department Senior Scientist, Learning Research & Development Center
University of Pittsburgh
![Page 2: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/2.jpg)
Outline The State of the Art: A Brief Survey Adaptation to Student Uncertainty in Tutorial
Dialogue: A Case Study– ITSPOKE: System and Corpora– Uncertainty Detection – System Adaptation– Experimental Evaluation
Summing Up
![Page 3: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/3.jpg)
What is Natural Language Processing?
• “The goal of this new field is to get computers to perform useful tasks involving human language, tasks like enabling human-machine communication, improving human-human communication, or simply doing useful processing of text or speech.”
[Jurafsky and Martin 2008]
• Many names and facets– Speech and Language Processing– Human Language Technology– Computational Linguistics
![Page 4: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/4.jpg)
Relevance for Adaptive Training Knowledge of Language is often needed to
– trigger adaptation– personalize training, using the enormous amount of
machine-readable text and audio that is now available
Conversational Agents are becoming an important form of human-computer interaction
![Page 5: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/5.jpg)
Knowledge of Language
• Phonetics and Phonology: speech sounds• Morphology: words and their internal composition• Syntax: the structuring of words into larger units• Semantics: the meaning of words and larger units• Pragmatics: interpretation in situational context• Discourse: interpretation in context of previous
utterances
![Page 6: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/6.jpg)
Computational Models (and Associated Algorithms)
State Machines Formal Rule Systems / Grammars Logic-Based Formalisms Models of Uncertainty
![Page 7: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/7.jpg)
A Brief Survey of Applications
NLP Applications to Education
![Page 8: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/8.jpg)
A Brief Survey of Applications
NLP Applications to Education
Learning Language(reading, writing,
speaking)
Tutors
Scoring
![Page 9: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/9.jpg)
A Brief Survey of Applications
NLP Applications to Education
Learning Language(reading, writing,
speaking)
Using Language (to teach everything else)
Tutors
Scoring
ConversationalTutors / Peers
CSCL
![Page 10: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/10.jpg)
A Brief Survey of Applications
NLP Applications to Education
Learning Language(reading, writing,
speaking)
Using Language (to teach everything else)
Tutors
Scoring
Readability
Processing Language
ConversationalTutors / Peers
CSCLDiscourse
CodingLecture
Retrieval
Questioning& Answering
![Page 11: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/11.jpg)
Outline The State of the Art: A Brief Survey Adaptation to Student Uncertainty in Tutorial
Dialogue: A Case Study– ITSPOKE: System and Corpora– Uncertainty Detection – System Adaptation– Experimental Evaluation
Summing Up
![Page 12: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/12.jpg)
Tutorial Dialogue Systems Why is one-on-one tutoring so effective?
“...there is something about discourse and natural language (as opposed to sophisticated pedagogical strategies) that explains the effectiveness of unaccomplished human [tutors].”
[Graesser, Person et al. 2001]
Currently only humans use full-fledged natural language dialogue
![Page 13: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/13.jpg)
Spoken Tutorial Dialogue Systems Most human tutoring involves face-to-face
spoken interaction, while most computer dialogue tutors are text-based
Can the effectiveness of dialogue tutorial systems be further increased by using spoken interactions?
![Page 14: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/14.jpg)
Potential Benefits of Spoken Dialogue: I
Conversation provides a learning environment that promotes student activity
Self-explanation correlates with learning and occurs more in speech
![Page 15: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/15.jpg)
Potential Benefits of Spoken Dialogue: II
Speech contains prosodic information, providing new sources of information about the student for adaptation
A correct but uncertain student turn– ITSPOKE: How does his velocity compare to that of
his keys?– STUDENT: his velocity is constant
![Page 16: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/16.jpg)
Potential Benefits of Spoken Dialogue: III
Spoken computational environments may foster social relationships that may enhance learning
![Page 17: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/17.jpg)
Potential Benefits of Spoken Dialogue : IV
Some applications inherently involve spoken language – Conversational skill training
Others require hands-free interaction– e.g., NASA
![Page 18: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/18.jpg)
Outline The State of the Art: A Brief Survey Adaptation to Student Uncertainty in Tutorial
Dialogue: A Case Study– ITSPOKE: System and Corpora– Uncertainty Detection – System Adaptation– Experimental Evaluation
Summing Up
![Page 19: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/19.jpg)
• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech
![Page 20: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/20.jpg)
• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech
![Page 21: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/21.jpg)
• Back-end is Why2-Atlas system [VanLehn, Jordan, Rose et al. 2002]• Sphinx2 speech recognition and Cepstral text-to-speech
![Page 22: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/22.jpg)
Three Types of Tutoring Corpora Human Tutoring
– 14 students / 128 dialogues (physics problems)– 5948 student turns, 5505 tutor turns
Computer Tutoring– 77 students / 385 dialogues – both synthesized and pre-recorded tutor voices
Wizard /Computer Tutoring– 81 students / 405 dialogues– human performs speech recognition, semantic analysis– computer performs dialogue management
![Page 23: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/23.jpg)
Experimental Procedure College students without physics
– Read a small background document– Took a multiple-choice Pretest – Worked 5-10 problems (dialogues) with tutor– Took an isomorphic Posttest
Goal was to optimize Learning Gain– e.g., Posttest – Pretest
![Page 24: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/24.jpg)
Outline The State of the Art: A Brief Survey Adaptation to Student Uncertainty in Tutorial
Dialogue: A Case Study– ITSPOKE: System and Corpora– Uncertainty Detection – System Adaptation– Experimental Evaluation
Summing Up
![Page 25: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/25.jpg)
Standard Empirical Detection Methodology Manual annotation of user states that will trigger
system adaptation– Naturally-occurring spoken dialogue data
Prediction via machine learning– Use speech and language processing to automatically
extract features from user turns– Use extracted features and annotations to learn a model
for predicting user state(s) in new data– Significant reduction of baseline error
![Page 26: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/26.jpg)
Example Features What a user says
– words (speech recognition), stems (morphology)– part-of-speech, syntactic constituents (parsing)– correctness (semantic analysis)– dialogue moves (pragmatics and discourse)
How a user says it– acoustic-prosodic analysis
![Page 27: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/27.jpg)
Extracting Pitch Features
![Page 28: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/28.jpg)
Extracting Energy Features
![Page 29: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/29.jpg)
Temporal Features Duration = end time - begin time Tempo (speaking rate) = #syllables/duration
![Page 30: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/30.jpg)
Detecting Neg/Pos/Neu in ITSPOKE
40
45
50
55
60
65
70
sp asr lex sp+asr sp+lex
+id-idmaj
- Baseline Accuracy via Majority Class Prediction
![Page 31: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/31.jpg)
Detecting Neg/Pos/Neu in ITSPOKE
40
45
50
55
60
65
70
sp asr lex sp+asr sp+lex
+id-idmaj
-Use of prosodic (sp), recognized (asr) and/or actual (lex) lexical features outperforms baseline
![Page 32: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/32.jpg)
Detecting Neg/Pos/Neu in ITSPOKE
40
45
50
55
60
65
70
sp asr lex sp+asr sp+lex
+id-idmaj
-As with other applications, highest predictive accuracies are obtained by combining multiple feature types
![Page 33: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/33.jpg)
Outline The State of the Art: A Brief Survey Adaptation to Student Uncertainty in Tutorial
Dialogue: A Case Study– ITSPOKE: System and Corpora– Uncertainty Detection – System Adaptation– Experimental Evaluation
Summing Up
![Page 34: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/34.jpg)
System Adaptation: How to Respond?
Our initial focus: responding to student uncertainty– Most frequent user state in our data – Focus of other studies– .62 Kappa
Approaches to adaptive system design– Theory-based– Data-driven
![Page 35: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/35.jpg)
Theory-Based Adaptation:Uncertainty as Learning Opportunity Uncertainty represents one type of learning impasse,
and is also associated with cognitive disequilibrium– An impasse motivates a student to take an active role in
constructing a better understanding of the principle. [VanLehn et al. 2003]
– A state of failed expectations causing deliberation aimed at restoring equilibrium. [Craig et al. 2004]
Hypothesis: The system should adapt to uncertainty in the same way it responds to other impasses (e.g, incorrectness)
![Page 36: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/36.jpg)
Data-Driven Adaptation: How Do Human Tutors Respond?
An empirical method for designing dialogue systems adaptive to student state– extraction of “dialogue bigrams” from annotated
human tutoring corpora – χ2 analysis to identify dependent bigrams– generalizable to any domain with corpora labeled for
user state and system response
![Page 37: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/37.jpg)
Example Human Tutoring Excerpt
S: So the- when you throw it up the acceleration will stay the same? [Uncertain]
T: Acceleration uh will always be the same because there is- that is being caused by force of gravity which is not
changing. [Restatement, Expansion]
S: mm-k. [Neutral]
T: Acceleration is– it is in- what is the direction uh of this acceleration- acceleration due to gravity?
[Short Answer Question]
S: It’s- the direction- it’s downward. [Certain]
T: Yes, it’s vertically down. [Positive Feedback, Restatement]
![Page 38: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/38.jpg)
Findings Statistically significant dependencies exist
between students’ state of certainty and the responses of an expert human tutor– After uncertain, tutor Bottoms Out and avoids
expansions – After certain, tutor Restates– After mixed, tutor Hints– After any non-neutral, tutor increases Feedback
Dependencies suggest adaptive strategies for implementation in computer tutoring systems
![Page 39: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/39.jpg)
Outline The State of the Art: A Brief Survey Adaptation to Student Uncertainty in Tutorial
Dialogue: A Case Study– ITSPOKE: System and Corpora– Uncertainty Detection – System Adaptation– Experimental Evaluation
Summing Up
![Page 40: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/40.jpg)
Experimental Design: 4 Conditions
Manipulate tutor responses to student uncertainty and investigate impact on learning and efficiency
Experimental-Basic: treat all uncertain turns as incorrect (theory)
Experimental-Empirical: for uncertain or incorrect turns, provide original content but vary dialogue act (human tutor analysis)
Control-Norm: ignore uncertainty (as in original system)
Control-Random: ignore uncertainty, but treat a percentage of random correct answers as incorrect (to control for additional tutoring)
![Page 41: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/41.jpg)
TUTOR: Now let’s talk about the net force exerted on the truck. By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?
STUDENT: The force of the car hitting it? [uncertain+correct]
TUTOR (Control-Norm): Good [Feedback] … [moves on] TUTOR (Experimental-Basic): Fine. [Feedback] We can
derive the net force on the truck by summing the individual forces on it, just like we did for the car. First, what horizontal force is exerted on the truck during the collision? [Remediation Subdialogue]
– Same tutor response if student had been incorrect
Treatments in Different Conditions
![Page 42: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/42.jpg)
Platform: Adaptive WOZ-TUT System
Modified version of ITSPOKE– Dialogue manager adapts to uncertainty
» system responses based on combined uncertainty and correctness
– Full automation replaced by some Wizard of Oz (WOZ) components
» human wizard recognizes student speech» human also annotates uncertainty and correctness» provides upper-bound speech and NLP performance
![Page 43: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/43.jpg)
WOZ-TUT Screenshot
![Page 44: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/44.jpg)
Experimental Procedure 20-21 subjects in each condition
– Native English speakers with no college physics
– Procedure: 1) read background material, 2) took pretest, 3) worked training problem with WOZ-TUT, 4) took user Brief Survey, 5) took posttest
![Page 45: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/45.jpg)
Experimental Results Two-way ANOVA
indicated students learned (F(1,77) = 271.214, p = 0.000, MSe = 0.009)
Amount depended on condition (F(3,77) = 3.275, p = 0.025, MSe = 0.009)
One-way ANOVA with post-hoc Tukey tests determined which conditions learned more
![Page 46: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/46.jpg)
Experimental Results Two-way ANOVA
indicated students learned (F(1,77) = 271.214, p = 0.000, MSe = 0.009)
Amount depended on condition (F(3,77) = 3.275, p = 0.025, MSe = 0.009)
One-way ANOVA with post-hoc Tukey tests determined which conditions learned more
![Page 47: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/47.jpg)
In Addition… Learning Efficiency also improved
– Two Efficiency Measures» (Normalized Learning Gains) / (Total Student Turns)» (Normalized Learning Gains) / (Total Time in Minutes)
– Experimental-Basic > Control-Norm (p < .05)
Current Directions– New evaluation of Experimental-Basic
» fully-automated ITSPOKE– New methods for designing Experimental-Empirical
» educational data mining using reinforcement learning– Other student states
![Page 48: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/48.jpg)
Outline The State of the Art: A Brief Survey Adaptation to Student Uncertainty in Tutorial
Dialogue: A Case Study– ITSPOKE: System and Corpora– Uncertainty Detection – System Adaptation– Experimental Evaluation
Summing Up
![Page 49: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/49.jpg)
Summing Up: I Spoken Dialogue Systems for Adaptive Training
– Natural language dialogue is a key aspect of human one-on-one training
– Using presently available technology, successful conversational computer training environments are now being built
– Evidence that more adaptive versions of such systems will further enhance performance
![Page 50: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/50.jpg)
Summing Up: II
Adaptive Training in turn provides many other opportunities and challenges for researchers in Speech and Natural Language Processing
![Page 51: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/51.jpg)
Acknowledgements ITSPOKE group past and present
– Hua Ai, Min Chi, Joanna Drummond, Kate Forbes-Riley, Alison Huettner, Michael Lipschultz, Beatriz Maeireizo-Tokeshi, Greg Nicholas, Amruta Purandare, Mihai Rotaru, Scott Silliman, Joel Tetreault, Art Ward
– Columbia Collaborators: Julia Hirschberg, Jackson Liscombe, Jennifer Venditti
NLP@Pitt– Jan Wiebe, Rebecca Hwa, Wendy Chapman, Paul Hoffmann, Behrang
Mohit, Carol Nichols, Swapna Somasundaran, Theresa Wilson, Chenhai Xi Why2-Atlas and Human Tutoring groups
– Kurt Vanlehn, Pamela Jordan, Uma Pappuswamy, Carolyn Rose– Micki Chi, Scotty Craig, Bob Hausmann, Margueritte Roy
![Page 52: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/52.jpg)
Thank You! Questions?
Further Information– http://www.cs.pitt.edu/~litman/itspoke.html
![Page 53: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/53.jpg)
The End
![Page 54: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/54.jpg)
Overview: Towards Adaptive Spoken Dialogue Systems
UserState
Detection Promising across user states and applications, e.g.:
Craig et al., 2006 Litman & Forbes-Riley, 2006 Lee & Narayanan, 2005 Vidrascu & Devillers, 2005 Batliner et al., 2003
Adaptation Sparse, can be difficult to show adaptation improves performance Some used basic adaptations and showed likeability increases For other performance metrics, basic adaptations not clear a priori
System AdaptationHealth Assessment Stress Empathy [Liu & Picard 2005]
Gaming Frustration Apology [Klein et al. 2002]
Application
Tutoring ??????? ???????
![Page 55: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/55.jpg)
Detecting and Responding to Student States
Opportunity – Adaptive spoken dialogue system technology can
improve student learning and other measures of performance [Aist et al. 2002; Pon-Barry et al. 2006]
Challenges– What to detect?– How to respond?– Evaluation?
![Page 56: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/56.jpg)
Example Student States in ITSPOKE
ITSPOKE: What else do you need to know to find the box‘s acceleration?
Student: the direction [UNCERTAIN] ITSPOKE : If you see a body accelerate, what caused that
acceleration?Student: force [CERTAIN] ITSPOKE : Good job. Say there is only one force acting on the box.
How is this force, the box's mass, and its acceleration related?Student: velocity [UNCERTAIN] ITSPOKE : Could you please repeat that?Student: velocity [ANNOYED]
![Page 57: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/57.jpg)
Prediction via Machine Learning Multiple feature types per student turn, e.g.
– Acoustic-prosodic– Lexical– Identifiers– System and student performance
Sample research questions– Relative utility of feature types– Impact of speech recognition – Speaker and task dependence– Impact of learning algorithm, amount of training data
![Page 58: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/58.jpg)
Detecting Neg/Pos/Neu in ITSPOKE
40
45
50
55
60
65
70
sp asr lex sp+asr sp+lex
+id-idmaj
- However, relative feature utility differs in tutoring (e.g., for speech features: temporal > energy > pitch)
![Page 59: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/59.jpg)
Example from ITSPOKE
ITSPOKE: What is the net force acting on the truck equal to?
Student: I don’t know [Uncertain, Frustrated]
ITSPOKE: Since net force is defined as the vector sum of all the forces exerted on an object, let's first
find the forces acting on the truck. Try to name them.
Student: The impact force, the normal force, and gravity[Certain, Neutral]
ITSPOKE: Yep.
![Page 60: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/60.jpg)
Approaches to Evaluation “Correlational” Studies, e.g.
– Student uncertainty positively correlates with learning [Craig et al. 2004]
– Adding uncertainty and frustration metrics to regression models increases model fit [Forbes-Riley et al. 2008]
“Causal” Studies, e.g.– Adding human-provided emotional scaffolding to a reading tutor
increases student persistence [Aist et al. 2002] – Experimentally manipulate tutor responses to student uncertainty
and investigate impact on learning [Pon-Barry et al. 2006]
![Page 61: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/61.jpg)
Treatments in Different Conditions TUTOR: Now let’s talk about the net force exerted on the
truck. By the same reasoning that we used for the car, what’s the overall net force on the truck equal to?
STUDENT: The force of the car hitting it? [uncertain+correct]
TUTOR (Control-Norm): Good [Feedback] … [moves on] TUTOR (Experimental-Empirical): That’s exactly right,
but you seem unsure, so let’s sum up. [Feedback] The net force on the truck is equal to the impact force on it… [New Bottom Out]– New tutor responses for incorrect +/- uncertainty answers as well
![Page 62: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/62.jpg)
In Closing Synergy between Intelligent Tutoring and
Spoken Dialogue Systems can provide– Better scientific understanding of how dialogue
facilitates learning– Long-term benefit for scaling spoken dialogue
systems to new and complex domains
![Page 63: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/63.jpg)
What is Tutoring?
• “A one-on-one dialogue between a teacher and a student for the purpose of helping the student
learn something.” [Evens and Michael 2006] • Human Tutoring Excerpt [Thanks to Natalie Person and Lindsay Sears,
Rhodes College]
![Page 64: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/64.jpg)
Intelligent Tutoring Systems Students who receive one-on-one instruction
perform as well as the top two percent of students who receive traditional classroom instruction [Bloom 1984]
Unfortunately, providing every student with a personal human tutor is infeasible– Develop computer tutors instead
![Page 65: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/65.jpg)
Monitoring Student State (motivation)
Tutor26: Let me ask my question in a more general way. What happens to the velocity of a body when there is no force acting on it?
Student27: dammit (ASR: it is)Tutor28 : Could you please repeat that?Student29 : same (ASR: i same)Tutor30 : Right. And what happens to the HORIZONTAL velocity of a body when
there is no HORIZONTAL force acting on it?Student31: zero (ASR: the zero)Tutor32 : Well... OK, let's consider how forces in the vertical direction affect the
motion of a body in a horizontal direction in a different scenario <…omitted…>Student33: oh gravity you already said this (ASR: beats gravity you're exert less)Tutor34: Fine. Are there any other forces acting on the apple as it falls?Student35: no why are you doing this again (ASR: no y and to it yes)Tutor36: Correct. The only force acting on the apple as it falls to the earth is the
earth's gravitational force. In which direction does gravitational force act?Student37: downward you computer (ASR: downward you computer)
![Page 66: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/66.jpg)
What to Annotate? Information-Access and Customer Care Systems
– Negative: Angry, Annoyed, Frustrated, Tired– Positive/Neutral: Amused, Cheerful, Delighted, Happy, Serious
[Ang et al. 2002; Shafran et al. 2003; Lee and Narayanan 2005; Liscombe et al. 2005]
![Page 67: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/67.jpg)
What to Annotate? Information-Access and Customer Care Systems
– Negative: Angry, Annoyed, Frustrated, Tired– Positive/Neutral: Amused, Cheerful, Delighted, Happy, Serious
[Ang et al. 2002; Shafran et al. 2003; Lee and Narayanan 2005; Liscombe et al. 2005]
Tutorial Dialogue Systems – Negative: Angry, Annoyed, Frustrated, Bored, Confused,
Uncertain, Contempt, Disgusted, Sad– Positive/Neutral: Certain, Curious, Enthusiastic, Eureka
[Litman and Forbes-Riley 2006, D’Mello et al. 2006]
![Page 68: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/68.jpg)
Theory-Based Adaptation• In tutoring, not all negatively-valenced states are bad!
– While frustration/anger/annoyance is often frustrating…– Frustration can also be an opportunity to learn
• Example from AutoTutor– neutral flow confusion frustration neutral
[Thanks to Sidney D‘Mello and Arthur Graesser, University of Memphis]
![Page 69: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/69.jpg)
Bigram Dependency Analysis
EXPECTED Tutor IncludePos
Tutor OmitsPos
neutral 439.46 2329.54
certain 175.21 928.79
uncertain 129.51 686.49
mixed 36.82 195.18
OBSERVEDTutor
IncludesPos
Tutor OmitsPos
neutral 252 2517
certain 273 832
uncertain 185 631
mixed 71 161
χ2 = 225.92 (critical χ2 value at p = .001 is 16.27)- “Student Certainness – Tutor Positive Feedback” Bigrams
![Page 70: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/70.jpg)
Bigram Dependency Analysis (cont.)
EXPECTEDIncludes
PosOmits
Pos
neutral 439.46 2329.54
OBSERVEDIncludes
PosOmits
Pos
neutral 252 2517
- Less Tutor Positive Feedback after Student Neutral turns
![Page 71: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/71.jpg)
Bigram Dependency Analysis (cont.)
EXPECTEDIncludes
PosOmits
Pos
neutral 439.46 2329.54
certain 175.21 928.79
uncertain 129.51 686.49
mixed 36.82 195.18
OBSERVEDIncludes
PosOmits
Pos
neutral 252 2517
certain 273 832
uncertain 185 631
mixed 71 161
- Less Tutor Positive Feedback after Student Neutral turns- More Tutor Positive Feedback after “Emotional” turns
![Page 72: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/72.jpg)
Adaptation to Student Uncertainty: A First Evaluation
Most systems respond only to (in)correctness
Recall that literature suggests uncertain as well as incorrect student answers signal learning impasses
Experimentally manipulate tutor responses to student uncertainty and investigate impact on learning
![Page 73: Speech and Language Processing for Adaptive Training Diane Litman Professor, Computer Science Department Senior Scientist, Learning Research & Development](https://reader035.vdocuments.mx/reader035/viewer/2022062905/5a4d1acb7f8b9ab05996f8f4/html5/thumbnails/73.jpg)
Experimental Design: 4 Conditions
Experimental-Basic: treat all uncertain turns as incorrect Experimental-Empirical: for uncertain or incorrect turns
– provide original content, but vary dialogue act (human tutor analysis)
– provide additional feedback on uncertainty (beyond propositional content)
Control-Norm: ignore uncertainty (as in original system) Control-Random: ignore uncertainty, but treat a
percentage of random correct answers as incorrect (to control for additional tutoring)