the impact of automated measurement of text characteristics graesser-plenary... · potential...
TRANSCRIPT
![Page 1: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/1.jpg)
Art GraesserProfessor, Psychology & Institute for Intelligent Systems, University of Memphis
Honorary Research Fellow, University of Oxford
The Impact of Automated
Measurement of Text
Characteristics
Bill & Melinda
Gates Foundation
![Page 2: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/2.jpg)
Overview• A snapshot of conversational agents
in assessments of reading, writing,
listening, and speaking
• Models of reading that emphasize
discourse
• Automatic scoring of text and writing
with CohMetrix and other automated
systems
![Page 3: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/3.jpg)
Foundational Claims• There have been major advances in
computational linguistics and automated
discourse analyses during the last two
decades.
• Accuracy is impressive in computer analyses
of reading, writing, listening, and speaking.
• Conversational agents in social scenarios will
play an increasing role in these assessments.
![Page 4: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/4.jpg)
A snapshot of agents in assessments of
Reading
Writing
Speaking
Listening
![Page 5: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/5.jpg)
Conversational Agents
BEAT LeonardoPKD Android Casey
iSTART TLTS MRE
AutoTutor Adele STEVE
iMAP
SI Agent
Guru
Writing-Pal
![Page 6: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/6.jpg)
Memphis Agent Environments
PKD Android
Andrew Olney
DeepTutor
Vasile Rus
AutoTutor
Art Graesser
iMAP
Max Louwerse
Guru
Andrew Olney
Meta-Tutor
Roger Azevedo
AutoTutor-LITE
Xiangen Hu
iSTART
Danielle McNamara
iDRIVE
Barry Gholson
Writing-Pal
Danielle McNamara
HURA Advisor
Xiangen Hu
DeepTutor
Vasile Rus
![Page 7: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/7.jpg)
Trialogs
Expert Fellow Student
Human
Vicarious
![Page 8: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/8.jpg)
Trialogs in Learning
Low Ability Vicarious learning
Medium Ability Tutorial dialogue
High Ability Teachable agent
![Page 9: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/9.jpg)
Trialogs in Assessment
Low Ability Short responses to prompts
Inaccurate or irrelevant
Violation of social norms
Medium Ability
High Ability Lengthier turns
Accurate contributions
Social appropriateness
![Page 10: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/10.jpg)
10
![Page 11: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/11.jpg)
Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved.Confidential and Proprietary. Copyright © 2011 Educational Testing Service. All rights reserved.
Trialog (English Language Skills)
11
Agent Utterance
Lisa: Hey, Ron, you need to leave your water outside. I'm going
to go talk to my friends. I'll see you guys inside.
Ron: Why did she tell me I have to leave my water outside, Tim?
Human (Tim): I do not know.
Ron: Tim, why can't I drink water?
Human (Tim): The books may get wet.
Lisa: Why do you still have your water bottle, Ron? Look at rule
number 2. We cannot get in the library with food or drink.
![Page 12: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/12.jpg)
2005 ETS Invitational Conference, New York City, October 10-11, 2005
Tactical Language and Culture
Training System (Lewis Johnson)
![Page 13: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/13.jpg)
iSTART (Danielle McNamara) interactive Strategy Training for Active Reading and Thinking
![Page 14: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/14.jpg)
Writing Pal (Danielle McNamara)
![Page 15: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/15.jpg)
![Page 16: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/16.jpg)
Self-Regulated Learning with
MetaTutor (Roger Azevedo)
![Page 17: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/17.jpg)
Scientific reasoning
Electronic textbook
Periodic questions – answer and ask
Critique experiments and newspaper clips
Game context with aliens
Trialogs with tutor and student agents
Operation ARIES!
![Page 18: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/18.jpg)
People need to learn how to critically evaluate descriptions of research . . .
Th
Correlational
Design
Causal statement
![Page 19: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/19.jpg)
Core Concepts are presented
across the game
c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c c
Interactive
textCase studies Interrogation
---------------- Story elements ---------------
![Page 20: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/20.jpg)
Daphne Greenberg
Lee Branum-Martin
Robin Morris
Chris Oshima
Maureen Lovett
Jan Frijters
Art Graesser
Xiangen Hu
Mark Conley
Andrew Onley
![Page 21: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/21.jpg)
Intervention with AutoTutor and Trialogues(Graesser, D’Mello, Hu, Cai, Olney, & Morgan, 2012)
Conversational agents
Intelligent Tutoring System
Online through browser
Media include texts, diagrams, videos, quizzes, games, and social media
Adults communicate by typing, speaking, or pointing/clicking
![Page 22: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/22.jpg)
Text Selection and Repository• Critical to good instruction
• Related to both cognition and motivation
• Interesting• Relevant to adult lives• Multiple purposes• Not too easy or too difficult
• Different genres, media, and technologies
![Page 23: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/23.jpg)
Models of reading
that emphasize
discourse
![Page 24: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/24.jpg)
Models or Frameworks
in Discourse Processes Field
• Construction-integration model (Kintsch)
• Structure building framework (Gernsbacher)
• Causal structure (Trabasso, Van den Broek)
• Landscape model (Van den Broek)
• Constructionist theory (Graesser, Singer, Trabasso)
• Event indexing model (Zwaan, Magliano, Graesser)
• Memory-based resonance model (Myers, O’Brien)
• Embodied cognition (Glenberg, Zwaan)
![Page 25: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/25.jpg)
Multilevel framework of
discourse comprehension1. Words
2. Syntax
3. Textbase
Explicit ideas (propositions)
Referential cohesion
4. Situation model
Causal, intentional, temporal,
spatial, logical relationships
Connectives
5. Genre and rhetorical structure
6. Pragmatic communication
Graesser & McNamara
(2011). Topics in
Cognitive Science.
![Page 26: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/26.jpg)
Language Learning is Multidimensionaland Changes over Time
(Scarborough, 2003)
![Page 27: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/27.jpg)
Reading Framework Proposed by
Perfetti (1999)
![Page 28: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/28.jpg)
Goldman, Brown, Britt, Magliano,
Greenleaf, Lee, Griffin, Hastings,
Lawless, Pellegrino, Radinsky,
Raphael, Shanahan, Wiley
![Page 29: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/29.jpg)
Multilevel framework of discourse comprehension
1. Words
2. Syntax
3. Textbase
Explicit ideas (propositions)
Referential cohesion
4. Situation model
Causal, intentional, temporal, spatial logical relationships
Connectives
5. Genre and rhetorical structure
6. Pragmatic communication
Graesser & McNamara (2011). Topics in Cognitive Science.
![Page 30: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/30.jpg)
Potential Principles of Processing
• Bottom-up processing until reaching a level
where the reader is not proficient. Attempt to achieve deepest, most global level.
• Intermediate levels (3 & 4) have the highest
information novelty.Textbase and situation model demand resources.
• Top-down higher levels can circumvent the
need to process lower levels.A cost to comprehension at lower levels.
• Levels can compensate for deficits at other
levels.Particular compensation mechanisms require more research.
![Page 31: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/31.jpg)
Scenario 1
A child has trouble recognizing letters in
the alphabet so there is an obstacle in
lexical decoding at the word level (level 1).
The word deficit blocks him from
understanding any of the text at levels 2-6.
![Page 32: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/32.jpg)
Scenario 2
• Parents take their children to a new
Disney movie that have some adult
themes. The children notice the
parents laughing at different points in
the movie than they do.
• The children are making it successfully
through discourse levels 1-4, but not
levels 5 and 6.
![Page 33: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/33.jpg)
Scenario 3• An adult reads a health insurance document.
There are lengthy sentences with embedded
clauses, complex syntax, numerous quantifiers
(all, many, rarely), and many logical operators
(and, or, not, if).
• The adult signs the contract because she
understands its purpose and trusts the insurance
agency. Levels 5 and 6 circumvent the need to
understand levels 2-4 completely.
![Page 34: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/34.jpg)
Scenario 4
• Laboratory partners in an engineering course
read the directions to assemble a new
computer. They argue about how to hook up
the cables on the dual monitors.
• They have no problems with levels 1, 2, 3, 5,
and 6, but they do have a deficit at the
situation model level (level 4).
![Page 35: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/35.jpg)
Conclusion
A mature assessment of reading
and writing needs to be sensitive
to the different levels of language
and discourse.
![Page 36: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/36.jpg)
Automatic scoring of text and writing with Coh-Metrix and other automated systems
Graesser, McNamara, & Kulikowich (2011). Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40, 223-234.
Graesser, A. C., & McNamara, D. S. (2012). Automated analysis of essays and open-ended verbal responses. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA Handbook of Research Methods in Psychology, Vol 1: Foundations, Planning, Measures, and Psychometrics(pp. 307-325). Washington, DC: American Psychological Association.
![Page 37: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/37.jpg)
Text Difficulty Measures• Popular measures (correlate r = .89 to .94)
– Flesch-Kincaid (Klare, 1976)
– Degrees of Reading Power (Koslin, Zeno, Koslin, 1987)
– Lexile scores (Stenner, 2006, MetaMetrix)
– SourceRater (Kosten, 2011, Educational Testing Service)
– Reading Maturity Metric (Landauer, Foltz, 2011, Pearson)
• Typical factors
– Word Familiarity
Word frequency, # of letters, # of syllables
– Sentence Length
• Measures are more accurate if there are other levels
![Page 38: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/38.jpg)
Automated Computer
Assessments of Writing• Essay graders (accuracy = human experts)
– Intelligent Essay Assessor (Pearson Knowledge Technology)
– E-Rater, Criterion, CBAL (Educational Testing Service)
– Writing-Pal (McNamara, Crossley, similar to Coh-Metrix)
• Answers to questions– C-Rater (Educational Testing Service)
• Think aloud & self-explanations during reading– iSTART (McNamara)
– Reading Strategy Assessment Tool (Magliano, Millis)
• Contributions during conversation– AutoTutor dialogs and trialogs (Graesser)
– Operation ARA (Pearson Education)
![Page 39: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/39.jpg)
Language & Discourse
Analysis Tools• Analysis of Words
– WordNet, Framenet, MRC Database, Celex, LIWC
• Syntax– Penn Treebank, Charniak Parser
• Propositions – Propbank, logical form, entailment
• Coreference and cohesion – Coh-Metrix, SourceRater
• Essay Graders – Intelligent Essay Assessor, E-rater
• Genre analyzers– Biber, Coh-Metrix
• Dialogue– AutoTutor, iSTART, Operation ARA
![Page 40: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/40.jpg)
Tools to Represent World
Knowledge• Linguistic Inquiry and Word Count (LIWC)
(Pennebaker, Booth, & Francis, 2007)
• Latent semantic analysis (Landauer, McNamara, Dennis, & Kintsch, 2007)
• N-grams
(Jurafsky & Martin, 2008)
• Topics model and MEM (Griffiths, Steyvers, & Tenenbaum, 2007; Chung
& Pennebaker, 2010)
![Page 41: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/41.jpg)
Figurative Language Remains
a Challenge
• Metaphor, Simile
• Personification
• Metonymy
• Hyperbole, Understatement
• Irony, Sarcasm
• Jokes, Wit
• Indirect speech acts
![Page 42: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/42.jpg)
Cohesion– Repeated nouns and concepts across
sentences
That cat sat on a hat. The hat was black.
– Less anaphor
The hat was black. vs. It was black.
That is short. vs. That sentence is short.
– More connectives between sentences
because, although, however, first, then
– More headers and topic sentences
– Genre structure
![Page 43: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/43.jpg)
Plants (low and high cohesion)What Are the Needs of Plants?
Like all living things, plants have certain needs. Plants need sunlight, water, and air to live. Plants also need minerals(MIN·uhr·uhlz). A mineral is a naturally occurring substance that is neither plant nor animal.
The parts of plants help them to get or make what they need. All plants get water and minerals from the soil. The root is the part of the plant that grows underground. Roots help hold the plant in the ground. Roots also help take in water and minerals that the plant needs.
The stem is the part that supports the plant. It helps the plant stand upright. It carries minerals and water from the roots. It also carries food from the leaves to other parts of the plant.
[…]
What Plants Need
Plants have certain needs, just like all living
things have needs. For example, plants need
sunlight, water, and air to live. Plants also need
minerals (pronounced as MIN·uhr·uhlz). A
mineral is not a plant or an animal. Instead, a
mineral is a substance in the ground that occurs
naturally. There are three parts of plants that
help plants get what they need or help plants
make what they need.
The Three Parts of a Plant
The three parts of the plant are the roots, stems,
and leaves.
1. The Root
The root is the part of the plant that grows
underground. All plants get water and minerals
from the ground, which is sometimes called soil.
Roots help the plant take in water and minerals
that the plant needs from the soil. Roots also
help hold the plant in the ground.
2. The Stem
The stem is the part that supports the plant. The
stem helps the plant stand upright. It carries
minerals and water from the roots of the plant to
other parts of the plant. The stem also carries
food from the leaves to other parts of the plant.
![Page 44: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/44.jpg)
Coh-Metrix Goals
• Automatic analysis of texts on multiple
levels of the multilevel, multicomponent
frameworks
• A tool that can be used by researchers,
teachers, students, educational leaders,
and the public
![Page 45: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/45.jpg)
Coh-Metrix: A Natural Language Processing Tool
Analyzes texts on over 100 measures of
cohesion and language
Google Cohmetrix
![Page 46: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/46.jpg)
Example Coh-Metrix MeasuresWord Measures
• Number of syllables
• Part of speech (noun, verb…)
• Word frequency
• Concreteness, imagery
• Multiple meanings
• WordNet
Co-reference Cohesion
• Noun and argument overlap
• Stem overlap
– (lemmas: run, runs, runner)
• Latent semantic analysis (LSA)
• Lexical diversity (type-token ratio)
• Pronouns
Situation Model Cohesion
• Connectives & discourse markers
• Causal and intentional verbs
• Causal and intentional cohesion
• Repetition in tense and aspect
• Logical operators
– and, or, therefore, if, then, not
Syntax
• Structural complexity
• Modifiers per noun-phrase
• Words before main verb of
main clause
• Syntactic similarity between
sentences
![Page 47: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/47.jpg)
Preprocessing Syntax Analysis Lexical Analysis(Filters) (Tagger, Parser) (Lemmatizier, Stemmer)
LSA
WordNet
MRCCELEX
Wd Lists
Coh-Metrix
Database
Database
Info
Syntax
Features
Sentence
Complexity
Word
Difficulty
Lexical
Features
Spatial
Cohesion
Temporal
Cohesion
Causal
Cohesion
Referential
Cohesion
Text
Complexity
Components
![Page 48: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/48.jpg)
Copy and Paste text
Click on Analyze
Graph appears with verbal translation
Tea.cohmetrix.com
![Page 49: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/49.jpg)
Coh-Metrix Easability Components
• Narrativity. Narrative text tells a story, with characters, events, places, and
things that are familiar to the reader.
• Syntactic simplicity. Sentences with more complex syntax are more
difficult to process, whereas those with few words and simple, familiar structures are easier to process and understand.
• Word concreteness. Concrete words evoke mental images and are
more meaningful to the reader than abstract words.
• Referential cohesion. High cohesion text contains words and ideas
that overlap across sentences and the entire text, forming threads that connect the textbase together for the reader.
• Deep cohesion. Causal, intentional, and temporal connectives help the
reader to form a more coherent and deeper understanding of the text.
![Page 50: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/50.jpg)
0 20 40 60 80 100
Deep Cohesion
Referential Cohesion
Word Concreteness
Syntactic Simplicity
Narrativity
Percentiles
Maps and Globes
0 20 40 60 80 100
Deep Cohesion
Referential Cohesion
Word Concreteness
Syntactic Simplicity
Narrativity
Percentiles
How the Camel Got His Hump Back
Thousands of years ago, our
ancestors invented the map.
Ancient maps were crude but
very useful tools. They helped
people find food, clean water,
and the way back home--even
when home was a cave.
As civilizations grew, better
maps were needed.
Now this is the next tale, and it
tells how the Camel got his big
hump. In the beginning of
years, when the world was so
new and all, and the Animals
were just beginning to work for
Man, there was a Camel, and
he lived in the middle of a
Howling Desert…
![Page 51: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/51.jpg)
How did we arrive at our 5 major measures? (Graesser, McNamara, & Kulikowich, 2011, Educational Researcher)
• TASA - Touchstone Applied Science Associates • 37,520 texts
• Texts had mean of 288.6 words (SD = 25.4)
• Most of the texts in the language arts, science, and social
studies
• Represents texts a student would experience
throughout K12.
• Degrees of Reading Power (DRP) scores and
Genre classification
• Conducted a Principal Components Analysis
with 53 Coh-Metrix measures
![Page 52: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/52.jpg)
Red = Narrative, Green = Social Studies, Blue = Science
-1,5
-1
-0,5
0
0,5
1
1,5
2
Grades < 2 Grades 2-3 Grades 4-5 Grades 6-8 Grades 9-10 Grades 11-CCR
Narrativity Large differences
between narrative and
informational texts
Grade level approximated by DRP
![Page 53: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/53.jpg)
Red = Narrative, Green = Social Studies, Blue = Science
-1,5
-1
-0,5
0
0,5
1
1,5
2
Grades < 2 Grades 2-3 Grades 4-5 Grades 6-8 Grades 9-10 Grades 11-CCR
Syntactic SimplicitySyntax is simpler
for informational
texts
Grade level approximated by DRP
![Page 54: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/54.jpg)
Red = Narrative, Green = Social Studies, Blue = Science
-1,5
-1
-0,5
0
0,5
1
1,5
2
Grades < 2 Grades 2-3 Grades 4-5 Grades 6-8 Grades 9-10 Grades 11-CCR
Referential CohesionReferential cohesion
may compensate for
difficult content
Grade level approximated by DRP
![Page 55: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/55.jpg)
Red = Narrative, Green = Social Studies, Blue = Science
-1,5
-1
-0,5
0
0,5
1
1,5
2
Grades < 2 Grades 2-3 Grades 4-5 Grades 6-8 Grades 9-10 Grades 11-CCR
Deep CohesionScience is a bit
lower and slight
increase over
grades
Grade level approximated by DRP
![Page 56: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/56.jpg)
Red = Narrative, Green = Social Studies, Blue = Science
-1,5
-1
-0,5
0
0,5
1
1,5
2
Grades < 2 Grades 2-3 Grades 4-5 Grades 6-8 Grades 9-10 Grades 11-CCR
Word Concreteness An intriguing
curvilinear trend
Grade level approximated by DRP
![Page 57: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/57.jpg)
Measures of Text Difficulty Report(Nelson, Perfetti, Liben, & Liben, 2012)
![Page 58: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/58.jpg)
Measures of Text Difficulty Report(Nelson, Perfetti, Liben, & Liben, 2012)
![Page 59: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/59.jpg)
Formality of Language
Formal
• Expository
• High cohesion
• Complex syntax
• Abstract words
Informal
• Narrative
• Low cohesion
• Simple syntax
• Concrete words
![Page 60: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/60.jpg)
Policies of Text Assignment
• Pushing the envelope
• Building self-efficacy
• Balanced diet
• The role of topic interest
![Page 61: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/61.jpg)
Text Library
Search
Coh-MetrixProfile
Analysis & Recommendations
Sort Texts
![Page 62: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/62.jpg)
Easy Hard
Text Difficulty Heat Maps
![Page 63: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/63.jpg)
There was once a popular television show about an exceptional talking
horse named Mr. Ed.
You might surmise that some people actually believed, or, at the very
least, wanted to believe in the alleged talking horse.
Centuries ago, there were vast majorities of people who believed that
there was a horse that could answer questions; that it could even spell
words and do complex arithmetic!
This is the factual story about a horse known as Clever Hans, and the
"clever" part of his name came from his lofty intelligence.
Around the turn of the 20th century, a German school instructor named
Mr. von Osten exhibited his keenly intelligent horse who could not
convey information verbally, so instead, he did so by tapping his hoofs
on the ground in order to give his responses to questions.
There was once a popular television show about a talking horse named
Mr. Ed.
You might think that some people actually believed there really were
talking horses.
But no, even back then they knew that horses could not talk.
Over 100 years ago there were a lot of people who believed that there
was a horse that could answer questions.
The horse could even spell words and do arithmetic!
This is the true story about a horse known as Clever Hans.
The horse lived with his owner in Germany; Hans is a common name
(even for a horse) in Germany.
The “clever” part of his name came from his high intelligence.
Here is the story: Around the turn of the 20th century, a German school
teacher named Mr. von Osten showed off his very smart horse.
Of course, not even this smart horse could talk, so he communicated
by tapping his hoofs.
Easy Hard
Easy Text Version Difficult Text Version
Corresponding sentences differ
up to 3 levels between Text A and
Text B
![Page 64: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/64.jpg)
Definition of Engagement
Ztd = Z-score of text segment on difficulty
Zrt = Z-score of reading time
/ Zrt – Ztd / = Discrepancy score
Disengagement increases with the
discrepancy score.
What is the half-life of engagement for a
text?
![Page 65: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/65.jpg)
FKG Score
De-Coupling as a function of Flesch-
Kincaid score of sentence triplets
![Page 66: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/66.jpg)
Conclusions about Reading Time
and Decoupling
• Readers most engaged at the zone of
language they can handle
• Currently analyzing this decoupling at
different levels of the multilevel
framework
• Intelligent tutor could provide feedback
and activities at periodic points as they
read.
![Page 67: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/67.jpg)
Foundational Claims• There have been major advances in computational
linguistics and automated discourse analyses during the last two decades.
• Accuracy is impressive in computer analyses of reading, writing, listening, and speaking.
• Conversational agents in social scenarios will play an increasing role in these assessments.
![Page 68: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/68.jpg)
Who is at the Table?
• AI and computational
linguistics
• Cognitive and learning
sciences
• Language and
discourse processes
• Measurement and
assessment
![Page 69: The Impact of Automated Measurement of Text Characteristics Graesser-plenary... · Potential Principles of Processing ... understands its purpose and trusts the insurance agency](https://reader031.vdocuments.mx/reader031/viewer/2022030704/5af0968d7f8b9aa17b8f2ae3/html5/thumbnails/69.jpg)