formation and learning analytics?
DESCRIPTION
Formation by Design (FxD) Symposium Georgetown University, Washington, June 30 – July 2, 2014 https://futures.georgetown.edu/formationTRANSCRIPT
Formation and Learning Analytics? Simon Buckingham Shum Knowledge Media Institute, The Open University Visiting Fellow, Grad. School of Education, University of Bristol From August: Connected Intelligence Centre, University of Technology Sydney http://simon.buckinghamshum.net http://twitter.com/sbskmi
Formation by Design (FxD) Symposium Georgetown University, Washington, June 30 – July 2, 2014 https://futures.georgetown.edu/formation
1
Emergence Learning
It’s out of the labs and into products: every learning tool now has an “analytics dashboard” (a Google image search)
2
http
s://
twit
ter.
com
/Wis
wij
zer2
/sta
tus/
4140
5547
2451
5758
08
3
http
s://
twit
ter.
com
/Wis
wij
zer2
/sta
tus/
4140
5547
2451
5758
08
“Note: check the huge difference between
knowing and measuring…”
4
5 c.f. “Data is the new oil”
Scaleable Precise
Quan?fiable Reprocessable
For Morozov, analytics is where technological solutionism hits education:
6
“This flight from thinking and the urge to replace human judgments with timeless truths produced by algorithms is the underlying driving force of solutionism.”
If something like this is the pedagogical architecture for (trans)formational learning…
7
Structuring Knowledge Performing
Generating Learning Power
Purposing
If something like this is the pedagogical architecture for (trans)formational learning…
8
Structuring Knowledge Performing
Generating Learning Power
Purposing
…what roles can computational learning analytics play?
The worldview clash is what makes this
fascina?ng!
For discussion on how analytics shape educational worldviews
9 http://people.kmi.open.ac.uk/sbs/2014/06/edmedia2014-keynote
thought experiment: could we imagine…
liminal space analy7cs?
10
liminal space
= opportunity for transformation
or withdrawal 11
Liminal space = opportunity for transformation, or withdrawal
12
So while we initially strive to make our students
feel comfortable on campus, we then must help them balance their desire for security with the need to take risks and explore new ideas and possibilities. Rather than attempting to resolve the tension,
a college should help students find their place on this precarious threshold, in the liminal space between the familiar and strange, the old and the new.
Johansson & Felten, 2014
Liminal space = opportunity for transformation, or withdrawal
13
“It is when you have left the tried and true, but have not yet been able to replace it with anything else. [...] It is when you are
between your old comfort zone and any possible new answer. If you are not trained in how to hold anxiety, how to live with ambiguity, how to entrust and wait, you will run... anything to flee this terrible cloud of unknowing.”
Richard Rohr O.F.M. on the spirituality of liminal space
meaningful behavioural proxies?
some clues…
14
interpersonal engagement and social capital
analy7cs 15
Social Network Analysis (SNAPP)
16
What’s going on in these discussion forums?
Social Network Analysis to gain insight into peer-peer and peer-mentor dynamics (SNAPP tool)
http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation Bakharia A and Dawson S. (2011) SNAPP: a bird's-eye view of temporal participant interaction. Proceedings of the 1st International Conference on Learning Analytics and Knowledge. Banff, Alberta, Canada: ACM, 168-173.
Social Network Analysis (SNAPP)
18 http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
2 learners connect otherwise separate clusters
tutor only engaging with active students, ignoring disengaged ones on the edge
Visualizing and filtering online social ties: by topic and type of tie
Schreurs B., Teplovs C., Ferguson R., De Laat M. and Buckingham Shum S. (2013). Visualizing Social Learning Ties by Type and Topic: Rationale and Concept Demonstrator. Proc. 3rd Int. Conf. on Learning Analytics & Knowledge. Leuven, BE: ACM, 33-37. Open Access Eprint: http://oro.open.ac.uk/36891
Visualizing and filtering online social ties: by topic and type of tie
Visualizing and filtering online social ties: by topic and type of tie
Visualizing and filtering online social ties: by topic and type of tie
Visualizing and filtering online social ties: by topic and type of tie
disposi7onal learning analy7cs
24
Dispositional Learning Analytics Workshop
25
http://learningemergence.net/events/lasi-dla-wkshp http://learningemergence.net/2014/03/01/ assessing-learning-dispositions-academic-mindsets
Observation informal and formal
Self-Diagnostic informal and formal
Behavioural Analytics
Assessing Learning Dispositions/Mindsets
Future sweetspot... multiple lenses to provoke self-reflection
http://people.kmi.open.ac.uk/sbs/2014/02/assessing-learning-dispositions-academic-mindsets
Self-report through reflective blogging 9-10 yr old EnquiryBloggers • Bushfield School, Wolverton, UK
EnquiryBlogger Wordpress Multisite plugins http://learningemergence.net/tools/enquiryblogger
27
Masters level EnquiryBloggers Graduate School of Education, University of Bristol
EnquiryBlogger: blogging for Learning Power & Authentic Enquiry http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry
28
29
EnquiryBlogger teacher’s dashboard – direct navigation to learners’ blogs from the visual
analytic
We are quantifying our lives
http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets
Dispositional profile from behavioural traces, to complement self-report?
Shaofu Huang: Prototyping Learning Power Modelling in SocialLearn http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium Simon Knight: http://people.kmi.open.ac.uk/knight/2014/02/knowledge-in-search
Social network patterns, teamwork effectiveness
and initiation of relationships
Questioning, arguing and search behaviours
reveal intrinsic curiosity and epistemic commitments
Tagging/sharing/blogging/social patterns
reveal how you see connections between
ideas
Behavioural and somatic traces associated with
perseverance, grit, tenacity; overcoming
panic/stress when stretched
narra7ve/ discourse
analy7cs 32
The Storymaking Project: PhD by Joanna Kwiat hLp://projects.kmi.open.ac.uk/storymaking
1st International Workshop on Discourse-Centric Learning Analytics
analytics that look beneath the surface, and quantify linguistic proxies for ‘deeper learning’
Beyond number / size / frequency of posts; ‘hottest thread’
http://ww
w.glennsasscer.com
/wordpress/w
p-content/uploads/2011/10/iceberg.jpg
http://solaresearch.org/events/lak/lak13/dcla13
Why are stories so important to formation?
34
“Fundamentally different to logical argument, the story positively demands those things that the other either denies or restricts: emotion, agency, character, perspective, and so on. To understand a story on some level is to become engaged in its telling and this will normally be achieved by allowing oneself to empathise or identify with the story’s characters or its narrator.”
Discourse analytics on webinar textchat
Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference on Learning Analytics and Knowledge (Banff, Canada, 2011). ACM, pp. 99-103. Open Access Eprint http://oro.open.ac.uk/28955
Can we spot the quality learning conversations in a 2.5 hr webinar?
-60
-40
-20
0
20
40
60
80
9:2
8 9
:32
9:3
6
9:4
0
9:4
1
9:4
6
9:5
0
9:5
3
9:5
6
10:
00
10:
05
10:
07
10:
07
10:
09
10:
13
10:
17
10:
23
10:
27
10:
31
10:
35
10:
40
10:
45
10:
52
10:
55
11:
04
11:
08
11:
11
11:
17
11:
20
11:
24
11:
26
11:
28
11:
31
11:
32
11:
35
11:
36
11:
38
11:
39
11:
41
11:
44
11:
46
11:
48
11:
52
11:
54
12:
00
12:
03
12:
04
12:
05
Average Exploratory
Discourse analytics on webinar textchat
Sheffield, UK not as sunny as yesterday - still warm
Greetings from Hong Kong
Morning from Wiltshire, sunny here!
See you!
bye for now!
bye, and thank you
Bye all for now
Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar…
Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
-60
-40
-20
0
20
40
60
80
9:2
8 9
:32
9:3
6
9:4
0
9:4
1
9:4
6
9:5
0
9:5
3
9:5
6
10:
00
10:
05
10:
07
10:
07
10:
09
10:
13
10:
17
10:
23
10:
27
10:
31
10:
35
10:
40
10:
45
10:
52
10:
55
11:
04
11:
08
11:
11
11:
17
11:
20
11:
24
11:
26
11:
28
11:
31
11:
32
11:
35
11:
36
11:
38
11:
39
11:
41
11:
44
11:
46
11:
48
11:
52
11:
54
12:
00
12:
03
12:
04
12:
05
Average Exploratory
Discourse analytics on webinar textchat
…but if we zoom in on a peak…
Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
Discourse analytics on webinar textchat
-100
0
100
9:2
8 9
:40
9
:50
1
0:00
1
0:07
1
0:17
1
0:31
1
0:45
1
1:04
1
1:17
1
1:26
1
1:32
1
1:38
1
1:44
1
1:52
1
2:03
Averag
Classified as “exploratory
talk”
(more substantive for learning)
“non-exploratory”
…language is used in a manner more akin to “Exploratory Talk” (Neil Mercer)
Ferguson, R., Wei, Z., He, Y. and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
Rhetorical discourse analytics
41
OPEN QUESTION: “… little is known …” “… role … has been elusive” “Current data is insufficient …”
CONTRASTING IDEAS: “… unorthodox view resolves …” “In contrast with previous hypotheses ...” “... inconsistent with past findings ...”
SURPRISE: “We have recently observed ... surprisingly” “We have identified ... unusual” “The recent discovery ... suggests intriguing roles”
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
Rhetorical discourse analytics
42
Human analyst
Rhetorical discourse analytics
43
Human analyst Computational analyst
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
Rhetorical discourse analytics
44 Duygu Simsek’s PhD: http://people.kmi.open.ac.uk/simsek/research/
Glimpses of analytics capable of detecting higher order thinking.
But humans will always read
differently to machines
Can we correlate this with “academic writing”, and can such
analytics be used as formative feedback on drafts?
Rhetorical discourse analytics
45 Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: http://oro.open.ac.uk/37391
CONTRAST
SUMMARY & CONTRIBUTION
Evidence Hub: semantic storytelling for students, practitioners and researchers
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
A wizard guides the user through the submission of a structured story: • What’s the Issue? • What claim are you making/
addressing? • What kind of evidence
supports/challenges this? • Link it to papers/data • Index it against the NIC’s core
themes
Evidence Hub: Argument Maps
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
The wizard then generates a structured Knowledge Tree showing evidence-based claims (and disagreements)
http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
An alternative graphical network rendering can be generated to show the current Argument Map of supporting and challenging evidence around an Issue
Evidence Hub: Argument Maps
Evidence Hub: professional development http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
Issue
Poten?al Solu?on
Suppor?ng Evidence
(prac??oner story)
epistemic cogni7on analy7cs
50
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829
Argumentation analytics (in Debate Hub & Cohere apps) what epistemic contributions are learners making in the network?
51
Rebecca is playing the role of broker,
connecting different peers’ contributions in
meaningful ways We now have the basis for recommending that
you engage with people NOT like you…
Generates novel visualizations to summarise community arguments on a given issue
http://consider.it
Structured Deliberation/Argument Maps
Shifts in epistemic commitments? (Simon Knight, KMi PhD research)
What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254
Learning sciences research into Epistemic Commitments/Cognition combined with
‘researching search’ methodologies.
Shifts in epistemic commitments? (Simon Knight, KMi PhD research)
Dimensions of Epistemic Belief
Certainty The degree to which knowledge is conceived as stable or changing, ranging from absolute to tentative and evolving knowledge
Simplicity The degree to which knowledge is conceived as compartmentalised or interrelated, ranging from knowledge as made up of discrete and simple facts to knowledge as complex and comprising interrelated concepts
Source The relationship between knower and known, ranging from the belief that knowledge resides outside the self and is transmitted, to the belief that it is constructed by the self
Justification What makes a sufficient knowledge claim, ranging from the belief in observation or authority as sources, to the belief in the use of rules of inquiry and evaluation of expertise
Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226
Shifts in epistemic commitments? (Simon Knight, KMi PhD research)
Behavioural Proxies of Epistemic Belief?
Certainty Frequency and density of positive and negative polarity links (e.g. supports; builds on, is consistent with, versus challenges; refutes; is inconsistent with).
Presence of stability markers – (e.g. current sources, bibliographic references, geographical spread).
Frequency and density of meta-discourse markers in the text of nodes, which moderate assertions in scholarly ways (e.g. in contrast to; remains an unresolved issue)
Simplicity Standard network analytics describing the topography of the structure
Semantic analysis of node and link types describing the variety of types used by a learner, the balance in link polarity (positive/neutral/negative), and the balance of propositional (causes; refutes; solves) versus analogical thinking (is a metaphor of; is analogous to).
Semantic analysis of tags on nodes and links
Encapsulation of node clusters or sub-networks within a broader category expressing a higher order construct
The use of different argumentation schemes, which reflect different sources of authority (argument from expertise; argument by analogy; argument from precedent)
In collaborative maps, the social network view showing who is connected to whom, but what kinds of links
Source Presence within the text of nodes of ‘I think’ or other restatements of fact
Challenges to claims through negative link types and meta-discourse markers in node text
Few additional nodes made other than those created as quotations.
Justification Judgements of relevance, and supporting or explanatory notes (‘this evidences/ explains x’). Ties to method ‘ideas’.
Use of well known Argumentation Schemes, especially evidence and arguments that grounded in the attribution of authority to people (‘experts’) or publications based on their status (‘valuing peer review’)
Rhetorical constructions in node text which show a critical use of sources
analytics for wider educational outcomes
Deakin Crick, R., Barr, S., Green, H. and Pedder, D. (2013). Evaluating the Wider Outcomes of Schools: Complex Systems Modelling. Centre for Systems Learning & Leadership, Graduate School of Education, University of Bristol, UK. http://learningemergence.net/library/reports/
Providing a more complete analytic on the health of the educational community
57
“Test grades are just ONE of a dozen indicators in our vision of a healthy educational community”
• life-long/life-wide learners • citizenship • self-confidence • teamwork • emotional
intelligence • “wellbeing”…
Hierarchical Process Modelling (Univ. Bristol Perimeta tool)
58
Seeing a learning community as a complex adaptive system requires the voices of learners, teachers, leaders, parents and ‘external’
stakeholders
The core mission and values of a network of school academies
60
Hierarchical Process Modelling transforms
qualitative and quantitative inputs into a multi-level
visual analytic
‘Italian Flag’ visual analytic
61
Degree of green / white / red reflects current (un)certainty over availability of evidence:
supporting / unknown / challenging
analytics as mirrors to provoke
(trans?)formative reflection
Your most recent mood comment: “Great, at last I have found all the resources that I have been looking for, thanks to Steve and Ellen.
In your last discussion with your mentor, you decided to work on your resilience by taking on more learning challenges
Your ELLI Spider shows that you have made a start on working on your resilience, and that you are also beginning to work on your creativity, which you identified as another area to work on.
1 2 3
45
Envisioning a social learning analytics dashboard
Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM: New York, 23-33. DOI: http://dx.doi.org/10.1145/2330601.2330616 Eprint: http://oro.open.ac.uk/32910
63
liminal space analy7cs? Use of computational techniques to reflect back to learners apparent shifts in • how they engage in a community of enquiry/practice • how they narrate their learning journeys • how they negotiate meaning • how their worldview seems to have shifted • how they respond cognitively (even somatically) when challenged.
65
towards a networked improvement community
(NIC) for FxD?
hLp://learningemergence.net/2014/05/29/learning-‐analy?cs-‐plus-‐nics-‐for-‐systemic-‐edu-‐improvement
Defining features of a NIC ‘B’ + ‘C’ Level improvement
‘A’ — what prac??oners do in a given ins?tu?on/
context
‘B’ — trying to improve local
prac?ce
‘C’ — the network of ‘B’ people
sharing insight and coordina?ng innova?on
Based on Doug Engelbart’s forma?ve work: hLp://www.dougengelbart.org/about/nics.html
Defining features of an Educational-NIC a sociotechnical infrastructure and modus operandi for coordinated aLack on
educa?onal challenges an inten?onal community facilitated by a coordina?ng hub using agreed representa?ons of the system and improvement strategies
in order to orchestrate ac?on in pursuit of measurable goals by sharing data, insights and arguments in order to build evidence-‐based claims
Bryk et al, Carnegie Founda?on for the Advancement of Teaching: hLp://www.carnegiefounda?on.org/sites/default/files/bryk-‐gomez_building-‐nics-‐educa?on.pdf
A NIC is in essence a “Collaboratory”
“a collaboratory is more than an elaborate collection of ICT; it is
a new networked organizational form that also
includes social processes; collaboration techniques; formal and informal communication; and agreement on norms, principles, values, and rules” (Cogburn, 2003, p. 86).
Cogburn, D. L. (2003). HCI in the so-called developing world: what’s in it for everyone, Interactions, 10(2), 80-87, New York: ACM Press. http://dx.doi.org/10.1145/637848.637866
Building blocks of an Educational-NIC
NIC Data Hub
Collective Sensemaking
Coordinating Maps
(of the system, problem, and
strategies)
Coordinating Maps
(of the system, problem, and
strategies)
Coordinating Maps
(of the system, problem, and
strategies)
Human + Machine Analytics
Educational Context Educational
Context
Building blocks of an Educational-NIC
NIC Data Hub
Collective Sensemaking
Human + Machine Analytics
Educational Context Educational
Context Coordinating
Maps (of the system, problem, and
strategies)
Coordinating Maps
(of the system, problem, and
strategies)
Coordinating Maps
(of the system, problem, and
strategies)
Shared Conceptual Maps: of system / problem / strategies
http://www.carnegiefoundation.org/elibrary/getting-ideas-action-building-networked-improvement-communities-in-education
http://www.carnegiealphalabs.org/persistence/
Shared Conceptual Maps: for knowledge navigation
People / Organizations / Projects / Claims / Evidence Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
Shared Conceptual Maps: for knowledge navigation
People / Organizations / Projects / Claims / Evidence Evidence Hub for Research by Children & Young People: http://rcyp.evidence-hub.net
Building blocks of an Educational-NIC
NIC Data Hub
Human + Machine Analytics
Educational Context Educational
Context Coordinating
Maps (of the system, problem, and
strategies)
Coordinating Maps
(of the system, problem, and
strategies)
Coordinating Maps
(of the system, problem, and
strategies)
Collective Sensemaking
Impact Map
http://chaos.open.ac.uk/hypothesis-list/
Impact Maps were inspired by the Evidence Hub, and are organised around core hypotheses that the NIC is investigating.
Impact Map
http://oermap.org/hypothesis/578/hypothesis-a-performance/
Impact Map
http://oermap.org/hypothesis/578/hypothesis-a-performance/
Impact Maps use Sankey flow diagrams to show how findings from different contexts support or challenge each of the NIC’s core hypotheses
Impact Map
http://oermap.org/hypothesis/578/hypothesis-a-performance/