3rd learning analytics workshop
DESCRIPTION
This talk was given for the Open University's 3rd informal Learning Analytics workshop.TRANSCRIPT
LEARNING ANALYTICS FOR
ACADEMIC WRITING
Duygu Simsek
3rd Learning Analytics Workshop, The Open University, UK 15th May, 2014
people.kmi.open.ac.uk/simsek
simsekduygu_
Supervisors: Prof. Simon Buckingham Shum, Dr. Rebecca Ferguson, & Dr. Anna De Liddo
Imagine if a machine that could detect…
Human analyst-Highlight key contributions of the project outcomes
http://bit.ly/HumanVsXIP
Learning Analytics Summer Institute (LASI), UK, Informatics Forum, Edinburgh July 5, 2013
Imagine if a machine that could detect…
Human analyst-Highlight key contributions of the project outcomes
http://bit.ly/HumanVsXIP
Learning Analytics Summer Institute (LASI), UK, Informatics Forum, Edinburgh July 5, 2013
XIP
4
Research Aim
To investigate
whether computational techniques can automatically identify the attributes of good academic writing in higher education
if this proves possible, how best to feed back actionable analytics to support students and educators
whether this feedback has any demonstrable benefits
5
Where this research sits?
ACADEMIC WRITING
LEARNING ANALYTICS
COMPUTATIONALTEXT ANALYSIS
Rhetorical Parsers
DiscourseCentric
Learning Analytics
Meta-discoursein Student writing
6
Where this research sits?- Academic Writing
Key aim of academic writing is to convince
readers about the validity of the claims and arguments put forward through an effective narrative.
ACADEMIC WRITING
LEARNING ANALYTICS
COMPUTATIONALTEXT ANALYSIS
Rhetorical Parsers
DiscourseCentric
Learning Analytics
Meta-discoursein Student writing
7
Where this research sits?-Meta-discourse
This effective narrative is signalled through meta-discourse!
ACADEMIC WRITING
LEARNING ANALYTICS
COMPUTATIONALTEXT ANALYSIS
Rhetorical Parsers
DiscourseCentric
Learning Analytics
Meta-discoursein Student writing
8
Meta-discourse
Meta-discourse refers to the features of text that convey the author’s intended meaning and intention. It provides cues to the reader which explicitly express a viewpoint, argument and claim, and signals the writer's stance.
Fig. 1 Meta-discourse that convey summary statements
Cues
to S
um
mary
st
ate
ments
9
Examples of meta-discourse cues thatsignal academic/analytical rhetorical moves
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is universally accepted
NOVELTY:
New insights provide direct evidence……suggest a new approach…Results define a novel role ...
OPEN QUESTION:
Little is known …
… role … has been elusive
Current data is insufficient…
TENDENCY:
... emerging as a promising approach
Our understanding ... has grown exponentially ...
Growing recognition of the
importance ...
CONTRASTING IDEAS:
In contrast with previous hypotheses ...
... inconsistent with past findings ...
SIGNIFICANCE:
studies ... have provided important advances
... is crucial for ... understanding
valuable information ... from
SURPRISE:
We have recently observed ... surprisingly
We have identified ... unusual
The recent discovery ... suggests intriguing roles
SUMMARISING:
The goal of this study ...
Here, we show ...
Our results ... indicate
10
Where this research sits?- Meta-discourse
In order to assess students’ writing therefore, educators will be examining students’ use of meta-discourse which make their students’ thinking visible.
However, students find it challenging to learn to write in an academically sound way.
They need to learn how to make their thinking visible by recognising and deploying meta-discourse.
ACADEMIC WRITING
LEARNING ANALYTICS
COMPUTATIONALTEXT ANALYSIS
Rhetorical Parsers
DiscourseCentric
Learning Analytics
Meta-discoursein Student writing
11
Where this research sits?- Computational Text Analysis
Meta-discourse cues are automatically identifiable.
This PhD investigates whether it is possible to provide automatic meta-discourse analysis of student writing through the use of a particular rhetorical parser, XIP.
ACADEMIC WRITING
LEARNING ANALYTICS
COMPUTATIONALTEXT ANALYSIS
Rhetorical Parsers
(XIP)
DiscourseCentric
Learning Analytics
Meta-discoursein Student writing
12
Example of a rhetorical parser: Incremental Parser (XIP)
Ágnes Sándor - Xerox Research Centre Europe
Natural Language Processing (NLP) product which includes a rhetorical parser detecting meta-discourse in academic texts.
XIP extracts salient sentences based on their rhetorical functions: Background Knowledge Summarising Tendency Novelty Significance Surprise Open Question Contrasting Ideas
13
Student Writing Analysed by XIP
CONTRAST
SUMMARY
14
Rhetorical functions classified by XIP
BACKGROUND KNOWLEDGE:
Recent studies indicate …
the previously proposed …
… is universally accepted
NOVELTY:
New insights provide direct evidence……suggest a new approach…Results define a novel role ...
OPEN QUESTION:
Little is known …
… role … has been elusive
Current data is insufficient…
TENDENCY:
... emerging as a promising approach
Our understanding ... has grown exponentially ...
Growing recognition of the
importance ...
CONTRASTING IDEAS:
In contrast with previous hypotheses ...
... inconsistent with past findings ...
SIGNIFICANCE:
studies ... have provided important advances
... is crucial for ... understanding
valuable information ... from
SURPRISE:
We have recently observed ... surprisingly
We have identified ... unusual
The recent discovery ... suggests intriguing roles
SUMMARISING:
The goal of this study ...
Here, we show ...
Our results ... indicate
15
Fine for researchers or machines but it
is not learner/educator
friendly
XIP’s Output
16
Why XIP? – Key Features of Academic Writing?
RelevanceUnderstanding & KnowledgeStructure & OrganisationLinguistic AccuracyIllustrations ReferencingArgumentation
17
Why XIP?
There is a mapping between good and strong features of academic writing and the XIP’s rhetorical functions.
18
Where this research sits?- Learning Analytics Potential?
XIP is a parser with potential, if it can be embedded in a more
complete learning analytics (LA) approach.
ACADEMIC WRITING
LEARNING ANALYTICS
COMPUTATIONALTEXT ANALYSIS
Rhetorical Parsers
(XIP)
DiscourseCentric
Learning Analytics
Meta-discoursein Student writing
19
Where this research sits?- Discourse-centric Learning Analytics
How should a DCLA approach be validated?
ACADEMIC WRITING
LEARNING ANALYTICS
COMPUTATIONALTEXT ANALYSIS
Rhetorical Parsers
(XIP)
DiscourseCentric
Learning Analytics
(DCLA)
Meta-discoursein Student writing
20
Main Research Question
To what degree can computational text analysis and visual analytics be used to
support the academic writing of students in higher education?
21
To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good
academic writing within student writing, as judged by the grade, and by educators?
XIP
Evaluates Accuracy & Sufficiency
Any correlation between Grades &
XIP output?
XIP’s Highlights vs. Marker’s
RQ1
22
To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good
academic writing within student writing, as judged by the grade, and by educators?
RQ1
XIP Highlighted Student Writing
Any correlation between the final grade of writing & XIP
findings?Pearson for Total number of salient sentences vs. GradeGeneralised Multiple Regression How strongly each rhetorical sentence type influences the final grade
Grades
23
What is the overlap between XIP’s output and how tutors judge quality?
Tutor Highlighted Student WritingXIP Highlighted Student Writing
To what extent is the rhetorical parser XIP accurate and sufficient for identifying the attributes of good
academic writing within student writing, as judged by the grade, and by educators?
RQ1
24
In what ways should XIP output be delivered to end users (students and educators)?
XIP
Evaluates Accuracy & Sufficiency
Any correlation between Grades &
XIP output?
XIP’s Highlights vs. Marker’s
Output
RQ2
25
1st Year Pilot study
1 min. Intro. Video: http://goo.gl/WGIkDs5 mins. Demo: http://goo.gl/Km1di5
In what ways should XIP output be delivered to end users (students and educators)?
RQ2
26
To what extent do educators value the results of XIP’s analysis of an individual student or cohort’s work when the primary focus is on assessment?
XIP
Evaluates Accuracy & Sufficiency
Any correlation between Grades &
XIP output?
XIP’s Highlights vs. Marker’s
Output
What educators think
RQ3
27X
IP
Evaluates Accuracy & Sufficiency
Any correlation between Grades &
XIP output?
XIP’s Highlights vs. Marker’s
Output
What educators think
Semi-structured interviews
To what extent do educators value the results of XIP’s analysis of an individual student or cohort’s work when the primary focus is on assessment?
RQ3
28
To what extent do students value the results of XIP’s analysis as formative feedback on their writing?
XIP
Evaluates Accuracy & Sufficiency
Any correlation between Grades &
XIP output?
XIP’s Highlights vs. Marker’s
Output
What educators think
What students think
RQ4
System for OU students.
29
Summary
Academic writing: Challenging to recognise & deploy meta-discourse.
XIP: A promising language technology
Visual analytics: Built on top of XIP output
Learning Analytics Potential: What happens when we bring it into the realm of learning? How might a learning analytics approach help? Do educators recognise value in such analytics? Do learners value it as formative feedback on their writing?