computational analyses of the effects of a …...computational analyses of the effects of a...
TRANSCRIPT
COMPUTATIONAL ANALYSES OF THE EFFECTS OF A STRUCTURE STRATEGY ON COLLEGE-LEVEL SUMMARIES:
COHESION AND RHETORICAL STRUCTURE
, Winterthur Switzerland, 5-6 September 2019, https://writinganalytics.zhaw.ch
Tamara Sladoljev-Agejev, University of Zagreb
Jan Šnajder, University of Zagreb
Svjetlana Kolić-Vehovec, University of Rijeka
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICS https://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
GENERAL RESEARCH QUESTION
Can the effects of structure strategy training be
identified automatically in student summaries?
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
STRUCTURE STRATEGIES HELP CONTENT INTEGRATION
Structure strategies: e.g. notes/graphic organizers (e.g. Jiang, 2012), summaries (e.g. Kirkland & Saunders, 1991)
Structure strategies ‘enable students to ...
a. follow the logical structure of text to understand how an author organized and emphasized ideas;
b. increase their own learning and thinking (e.g.,comparing, finding causal relationships, looking for solutions to block causes of problems);
c. use these text structures to organize their own writing ...’ (Meyer & Ray, 2011, p. 128)
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICS,https://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
Note-making and summaries are coherence-building exercises
TEXT MACROSTRUCTUREKintsch & van Dijk (1978); Lorch (2001); Louwerse & Graesser (2005); global coherence (source author’s plan/intention, Hobbs, 1993, Grosz & Sidner, 1986)
READING-FOR-UNDERSTANDINGintegrating segments of a text into a coherent
whole (Sabatini et al., 2013)
READING-TO-WRITEDelaney (2008)
conveying information – ‘real-life skill’ (Folz, 2016)
van den Broek et al., 1995; Zwaan & Singer, 2003
COHERENCE
IDEAS
concepts/propositions
RELATIONSe.g. causality, listing,
comparison, problem-solution
OUR RESEARCH
based on teaching rhetorical structure strategy (RSS)
↓
identifying the rhetorical structure of a text (Moore & Wiemer-Hastings, 2003) to achieve deep comprehension(i.e. ideas & rhetorical relations)
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur. Switzerland, 5-6 September 2019
Quasi-experimental research design
PRE-TEST
C-E
I BEFORE READING (TEXT 1)
1. Biodata
2. Metacognitive self-assessment
3. EL2 proficiency
a) grammar
b) vocabulary (Text 1)
4. Prior knowledge (Text 1)
II READING & NOTE MAKING
III AFTER READING (TEXT 1)
5. Summary writing
RSS INTERVENTION
E
READING AND NOTE-MAKING (GRAPHIC ORGANIZERS)
1. Paragraph-level comprehension (key words and headings to paragraphs)
2. Text segmentation (paragraph grouping)
3. Establishing rhetorical relations witin and between paragraphs
4. Making notes with explicitly indicated rhetorical relations (graphic organizers)
POST-TEST
C-E
I BEFORE READING (TEXT 2)
1. Prior knowledge (Text 2)
2. Vocabulary (Text 2)
II READING & NOTE MAKING
III AFTER READING (TEXT 1)
6. Summary writing
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
Pretest/posttest: A three-step procedure
SOURCE TEXT
reading
RHETORICAL STRUCTURE (MACRO, GLOBAL)
making structured notes, i.e. graphic organizers)
SUMMARY
writing
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
Part of a broader research study
COHERENCE?
(assessed by human raters)
COHESION?
(assessed by human raters)
RHETORICAL STRUCTURE?
Šnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)
COH-METRIX INDICES?
Crossley & McNamara, 2009, 2010 & 2011; Graesser et al., 2004; McNamara & Graesser, 2012
EFFECTS OF RHETORICAL STRUCTURE STRATEGY
(RSS)
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
Research questions in this study
COHERENCE?
(assessed by human raters)
COHESION?
(assessed by human raters)
RHETORICAL STRUCTURE?
Šnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)
COH-METRIX INDICES?
Crossley & McNamara, 2009, 2010 & 2011; Graesser et al., 2004; McNamara & Graesser, 2012
EFFECTS OF RHETORICAL STRUCTURE STRATEGY
(RSS)
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICS, https://writinganalytics.zhaw.chWinterthur Switzerland, 5-6 September 2019
Instruction to participants
‘Write a summary in the note form which will clearly convey the ideas of the text to a third person. Then write the summary as connected (linear) text.’
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
DATASETTot. 225 text-present summaries (300+/-10% w.), Sladoljev-Agejev & Šnajder (2017)
113 first-year business/economics undergraduates (English-L2, mostly upper intermediate and advanced)
Pretest: C(N=55), E (N=58), source text - 901 w.(The Economist)
Posttest: C (N=55), E(N=58), source text - 981 w. (The Economist)
Raters: -coherence/cohesion scoring (coherence/cohesion breaks)
-scores independently assigned first, then discussed and agreed
-weighted kappa: Chr-0.69, Chs-0.83
RESEARCH QUESTION 1
Are there effects of RSS on summaries measured by
Coh-Metrix indices of coherence and cohesion?
a) Referential cohesion (CRF)?b) Semantic similarity (LSA)?c) Text connectives (CNC)?d) Situation model (SM)?
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
Coh-Metrix indices
CRF referential cohesion
LSA semantic similarity
CNC connectives
SM situation model-related indices
Source: http://cohmetrix.com/
PRE-TEST POST-TEST
COH-METRIX C E E-C C E E-C
CRFAO1 0,261 0,263 0,002 0,303 0,360 0,056*
CRFAOa 0,261 0,263 0,002 0,303 0,360 0,056*
CRFCWO1 0,048 0,047 <-0,001 0,049 0,064 0,015**
CRFCWOa 0,037 0,037 <0,001 0,037 0,044 0,007**
CRFANP1 0,149 0,157 0,008 0,249 0,242 -0,007
CRFANPa 0,032 0,029 -0,003 0,053 0,061 0,008
LSASS1 0,170 0,173 0,004 0,130 0,145 0,015*
LSASSp 0,155 0,158 0,003 0,103 0,137 0,035****
LSAGN 0,274 0,278 0,004 0,248 0,255 0,007
CNCAll 77,798 78,186 0,388 88,021 96,630 8,606**
CNCCaus 25,183 24,800 -0,383 20,194 25,710 5,516**
CNCLogic 28,642 24,869 -3,773* 32,549 39,034 6,485*
CNCADC 5,011 5,643 0,631 12,634 12,033 0,602
CNCTemp 16,375 16,181 -0,195 14,494 17,048 2,554
CNCTempx 20,066 18,830 -1,236 10,933 11,320 0,387
CNCAdd 42,592 43,350 0,758 56,598 62,376 5,778*
CNCPos 73,727 74,194 0,467 79,736 89,680 9,944**
CNCNeg 4,472 4,945 0,473 9,053 8,795 -0,259
SMINTEp 21,177 18,655 -2,523 23,986 17,896 -6,090***
SMCAUSr 0,309 0,286 -0,023 0,335 0,514 0,179**
SMINTEr 0,840 0,956 0,116 0,627 0,958 0,330***
SMCAUSlsa 0,095 0,090 -0,004 0,086 0,114 0,028****
SMTEMP 0,679 0,665 -0,014 0,670 0,728 0,058**
RQ1: RESULTS
• effects of RSS in 16/23 features
• E summaries: more cohesion
CRF more referential cohesion
LSA more semantic similarity
CNC more connectives
SM more relatedness
Welch’s t-test
Boldface: statistical significance *p<0,05, **p<0,01, ***p<0,001, ****p<0,0001
RQ1: RESULTSMore cohesion devices found in E summaries
CRF
• more overalapping arguments locally and globally
• more overlapping content words locally and globally
LSA
• more semantically similar sentences locally and globally
CNC
• more connectives (all, causal andlogical operators)
SM
• fewer events/actions, more explicitrelations
• more semantically overlapping verbs
• more tense/aspect repetitions
RESEARCH QUESTION 2
Are there RSS effects on the computationally analysed rhetorical structure of student summaries in comparison with expert-written summaries (Šnajder, Sladoljev-Agejev & Kolić-Vehovec, 2019)?
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
Rhetorical structure may be linked to coherencevan den Broek et al., 1995; Moore & Wiemer-Hastings, 2003; Zwaan & Singer, 2003
COHERENCE
IDEAS
concepts/propositions
RELATIONSe.g. causality, listing,
comparison, problem-solution
Computational analysis of rhetorical structure (RS) Šnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)
COHERENCE
Arg 1 – R – Arg2
Comparing rhetorical structuresŠnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)
RHETORICAL STRUCTURE OVERLAPOVERLAPPING RELATIONS + OVERLAPPING ARGUMENTS
discourse parsing + semantic similarity measures
student summary expert-written summary
AUTOMATED SUMMARY SCORINGŠnajder, Sladoljev-Agejev & Kolić-Vehovec (2019)
1. DISCOURSE PARSING (Prasad et al., 2008; Lin et al., 2014)
2. COMPARING RHETORICAL STRUCTURES (SS vs REFS)
a) Equivalent rhetorical relations? If yes, then b) and c)
b) Argument similarity between pairs of rhetorical relations(Mikolov et al., 2013)
c) Total overlap scores between pairs of summaries (Kuhn, 1955)
RQ2: RESULTS
Rhetorical structure of E summaries - higher overlap with expert summaries
Average computational scores of RS overlap
PRETEST Precision Recall F-measure
C (N=55) 0.270 ± 0.086 0.284 ± 0.107 0.255 ± 0.058
E (N=58) 0.270 ± 0.076 0.263 ± 0.108 0.243 ± 0.057
E - C <0.0004 -0.021 -0.012
POSTTEST Precision Recall F-measure
C (N=55) 0.378 ± 0.064 0.234 ± 0.114 0.270 ± 0.101
E (N=58) 0.368 ± 0.056 0.331 ± 0.055 0.343 ± 0.041
E - C -0.010 +0.097**** +0.072****
CONCLUSION
Effects of teaching RSS can be detected automatically.
The following RSS effects are revealed:
• more cohesion devices (e.g. more overlapping words, more connectives, fewer actions/events, more semantically similar sentences, ...)
• higher overlap with the rhetorical structure of expert-writtensummaries
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
REFERENCES
Crossley, S. A. & McNamara, D. S. (2009). Computational assessment of lexical differences in L1 and L2 writing. Journal of Second Language Writing, 18(2), 119-135.
Crossley, S. A. & McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. U S. Ohlsson & R. Catrambone (eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society, 984–989. Austin, TX: Cognitive Science Society.
Crossley, S. A. & McNamara, D. S. (2011). Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing. International Journal of Continuing Engineering Education and Life Long Learning, 21(2-3), 170-191
Delaney, Y. A. (2008). Investigating the reading-to-write construct. Journal of English for Academic Purposes, 7(3), 140-150.
Foltz, P. W. (2016). Advances in Automated Scoring of Writing for Performance Assessment. In Handbook of Research on Technology Tools for Real-World Skill Development, 659-678.
Graesser, A. C., McNamara, D. S., Louwerse, M. M. & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohe-sion and language. Behavior Research Methods, Instruments & Computers, 36(2), 193-202.
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
Grosz, B. J. & Sidner, C. L. (1986). Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3), 175-204.
Hobbs, J.R. (1993). Intention, information, and structure in discourse: A first draft. In Burning Issues in Discourse, NATO Advanced Research Workshop, 41-66.
Jiang, X. (2012). Effects of discourse structure graphic organizers of EFL reading comprehension. Reading in a Foreign Language, 24(1), 84.
Kintsch, W., & Van Dijk, T. A. (1978). Toward a model of text comprehension and production. Psychological Review, 85(5), 363.
Kirkland, M. R. & Saunders, M. A. P. (1991). Maximizing student performance in summary writing: Managing cognitive load. Tesol Quarterly, 25(1), 105-121.
Kuhn, H.W. (1955). The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83–97.
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland,, 5-6 September 2019
Lin, Z., Ng, H. T. & Kan, M. Y. (2014). A PDTB-styled end-to-end discourse parser. Natural Language Engineering, 20(2), 151-184.
Lorch Jr., R. F. (2001). Psychology of macrostructure in discourse comprehension. In N. J. Smelser and P. B. Baltes (eds.) International Encyclopedia of the Social & Behavioral Sciences, 9122–9125. Oxford: Pergamon.
Louwerse, M. M. & Graesser, A. C.,(2005). Macrostructure. In K. Brown (ed.), Encyclopedia of Language and Linguistics, 7, 2005.
McNamara, D. & Graesser, A. (2012). Coh-Metrix: An automated tool for theoreticaland applied natural language processing. In P. M. McCarthy and C. Boonthum(eds.), Applied Natural Language Processing and Content Analysis: Identification, Investigation, and Resolution,188-205. Hershey, PA: IGI Global.
Meyer, B. J. & Ray, M. N. (2011). Structure strategy interventions: Increasingreading comprehension of ex-pository text. International Electronic Journal ofElementary Education, 4(1), 127.
Mikolov,T., Sutskever, I., Chen, K., Corrado, G.S. & Dean, J. (2013). Distributedrepresentations of words and phrases and their compositionality. In Advances inNeural Information Processing Systems, 3111–3119.
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICShttps://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019
Moore, J. D., & Wiemer-Hastings, P. (2003). Discourse in computational linguistics andartificial intelligence. In Handbook of Discourse Processes, 438-484. Routledge.
Prasad, R., Dinesh, N., Lee, A., Miltsakaki,E., Robaldo, L., Joshi, A. K., Webber, B.L. (2008). The Penn Discourse Treebank.2.0. In LREC. Citeseer.
Sabatini, J., O'Reilly, T. & Deane, P. (2013). Preliminary reading literacy assessmentframework: Foundation and rationale for assessment and system design. ETS Research Report Series.
Sladoljev-Agejev, T. & Šnajder, J. (2017). Using Analytical Scoring Rubrics in the Automatic Assessment of College-Level Summary Writing Tasks in L2. Proceedings of the 18th International Joint Conference on Natural Language Processing - IJCNLP 2017, Taipei, Taiwan (Volume 2: Short papers), 181-186.
Šnajder, J., Sladoljev-Agejev, T. & Kolić-Vehovec, S. (2019). Analysing Rhetorical Structure as a Key Feature of Summary Coherence. Proceedings of the 14th Workshop on InnovativeUse of NLP for Building Educational Applications – BEA 2019, Firenze, Italy, 46-51.
Van den Broek, P., Risden, K. & Husebye-Hartmann, E. (1995). The role of readers’ standards for coherence in the generation of inferences during reading. In Sources ofCoherence in Reading, 353–373. Hills-dale, NJ: Lawrence Erlbaum.
Weigle, S. C. (2002). Assessing Writing. Stuttgart: Ernst Klett Sprachen.
Zwaan, R. A., & Singer, M. (2003). Text comprehension. In Handbook of Discourse Processes89-127. Routledge.
THE EIGHTH INTERNATIONAL CONFERENCE ON WRITING ANALYTICS https://writinganalytics.zhaw.ch
Winterthur, Switzerland, 5-6 September 2019