are great wikis born or made? are students just posting in the same place?
DESCRIPTION
A pair of papers for AERA #2012 derived from research from the DCLC project (edtechresearcher.org).TRANSCRIPT
Are Great Wikis Born or Made?Evaluating Quality Trajectories in
Wiki Learning Environments
Justin Reich
Richard Murnane
John Willett
Just Posting in the Same Place?Evaluating Student Collaboration in
Wiki Learning Environments
@bjfr
edtechresearcher.org
Distributed Collaborative Learning Communities Project:
Web 2.0 in K-12 Settings
• Excellence: How do we make them good?
• Equity: Do only certain kids get the good ones?
• Analytics: What can we learn about learning from real-time usage data from online learning environments 2
3
Initial Quantitative Analysis to
Develop Sampling Strategy
Classroom Observations and Teacher Interviews to
Understand Wiki Practices
Measure Wiki Quality
Develop Wiki Quality
TrajectoriesLiterature Review of CSCL and 21st C. Skill
Scholarship
Path Diagram of Wiki Research
Correlate wiki quality with school level
SES
4
Initial Quantitative Analysis to
Develop Sampling Strategy
Classroom Observations and Teacher Interviews to
Understand Wiki Practices
Measure Wiki Quality
Develop Wiki Quality
TrajectoriesLiterature Review of CSCL and 21st C. Skill
Scholarship
Path Diagram of Wiki Research
Correlate wiki quality with school level
SES
Low Income Schools(n=117)
Mid to High Income Schools
(n=133)
Failed or Trial Wiki 50% 30%
Teacher-Content Wiki 34% 35%
Individual Student-Owned Wiki
15% 35%
Collaborative Student-Owned Wiki
2% 1%
Median Lifetime 6 days 33 days
6
Initial Quantitative Analysis to
Develop Sampling Strategy
Classroom Observations and Teacher Interviews to
Understand Wiki Practices
Measure Wiki Quality
Develop Wiki Quality
TrajectoriesLiterature Review of CSCL and 21st C. Skill
Scholarship
Path Diagram of Wiki Research
Correlate wiki quality with school level
SES
Are great wikis born or made?
“Time is Precious”
• Reimann (2009)– Collaborative learning unfolds over time
• Substantive need
– Online learning environments track data in real time• Methodological opportunity
8
Initial Quantitative Analysis to
Develop Sampling Strategy
Classroom Observations and Teacher Interviews to
Understand Wiki Practices
Measure Wiki Quality
Develop Wiki Quality
TrajectoriesLiterature Review of CSCL and 21st C. Skill
Scholarship
Path Diagram of Wiki Research
Correlate wiki quality with school level
SES
How did we measure collaborative behaviors on wikis, and what did we find?
Theoretical debates in wikis and collaboration (IJCSCL)
Are students meaningfully collaborating?
Dohn (2009): Web 2.0 tools are theoretically well-suited for collaborative knowledge production practices, but the educational logics of individual assessment prevent these practices from being realized.
Or just posting in the same place?
Glassman and Kang (2011): wikis not only enable students to participate in knowledge building, but allow the enactment of Dewey’s vision
Go Big Data!
The Plan• Explain our research design• Explain how we model wiki quality over
time– Answer the question: Are Great Wikis Born or
Made?• Explain in detail how we measure
collaboration– Answer the question: Who’s right: Dohn or
Glassman & Kang?
RESEARCH DESIGN
Which wikis are in my dataset?• All 179,851 publicly-viewable education-related
wikis started on the PBworks platform between June 2005 and August of 2008.
• Does not include “private” wikis (~70,000)
12
179,851PBWorks Wikis
Which wikis are in my sample?• Randomly sampled 1,799 wikis (1%)• Coded to identify 406 U.S. based, K-12 wikis• 255 from specific, identifiable public schools
– Detailed usage statistics provided by PBworks.com– School level SES data from the Common Core of Data (National
Center for Education Statistics, 2007-2008) – 255 wikis are from 41 U.S. states
13
179,851PBWorks Wikis 1,799
1% Random Sample
406U.S. K-12
Wikis
255Public School
How should we measure wiki quality?
Wiki quality as opportunities for 21st Century Skill Development
Expert Thinking
Complex Communi-
cation
New Media Literacy
Wiki Quality Instrument• Information Consumption (2 items):
– Do students use wikis to get information? links?
• Participation (4 items):– Do they contribute? Do they own pages?
• Expert thinking (5 items):– Do students use academic content knowledge in wiki activities?– Do students reflect on the process/product?
• Complex Communication/Collaboration (7 items):– Do students concatenate text on pages?– Do they substantively edit each others work and co-create pages?
• New Media Literacy (6 items):– Do students use formatting?– Do they hyperlink? Embed media?
24 QuestionsWiki Quality Scale
of 0-24
How did we measure wiki quality?• Sample wiki quality at 7, 14, 30, 60, 100, and
400 days• Two raters independently apply wiki quality
instrument– All raters must code “training set” of wikis within 1.5
points of master coders– Weekly meetings while coding to discuss categories,
difficult cases, etc.
• Third rater reconciles disagreements
17
Data Analytic Strategy:Multilevel Model for Change
• We should have time nested within wikis and use the MMFC to model quality trajectories
• What functional form should we use to represent time?
• The decision that we make in setting a functional form will answer the research question about time!
BUT WHICH ONE!!!!
Data Analytic Strategy:Multilevel Model for Change
Selected Empirical Growth Plots
Empirical Velocity Plot of Wiki Quality
0 50 100 150 200 250 300 350 400 4500
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Mean Difference in Score/Days
Data Analytic Strategy:Multilevel Model for Change
Comparing Fit Statistics
Linear Quadratic LogIntercept 2.8371*** 2.5647*** 1.2073***
Time 0.005629*** 0.02078***Time2 -0.00004***Ln(Time) 0.6614***-2LL 5542.2 5468.7 5263.0AIC 5554.2 5482.7 5275.0BIC 5578.3 5510.9 5299.1
Data Analytic Strategy:Multi-level Poisson Regression
Singer and Willett (2008); Rabe-Hesketh and Skrondal (2008)
Prototypical wiki quality trajectory (n=406)
0 50 100 150 200 250 300 350 4000
5
10
15
20
Days
Co
mp
os
ite
Wik
i Q
ua
lity
Sc
ore
Implications• Early wiki quality is a strong predictor of
later wiki quality- Great Wikis are Born– Hypothesis: Norms developed early on in wiki
learning communities are powerful in shaping later use
– FOR TEACHERS: Therefore, wiki design deserves critical attention
– FOR RESEARCHERS: Further qualitative studies of wiki using communities should ensure that observers can study wikis from the start of the process
BACK TO COLLABORATION
Theoretical debates in wikis and collaboration (IJCSCL)
Are students meaningfully collaborating?
Dohn (2009): Web 2.0 tools are theoretically well-suited for collaborative knowledge production practices, but the educational logics of individual assessment prevent these practices from being realized.
Or just posting in the same place?
Glassman and Kang (2011): wikis not only enable students to participate in knowledge building, but allow the enactment of Dewey’s vision
Go Big Data!
29
Initial Quantitative Analysis to
Develop Sampling Strategy
Classroom Observations and Teacher Interviews to
Understand Wiki Practices
Measure Wiki QualityLiterature
Review of CSCL and 21st C. Skill
Scholarship
Path Diagram of Wiki Research
Complex Communication
Concatenation Do multiple students add discrete sections of text to the same page?
Copyediting Does at least one student copyedit text created by another student?
Co-ConstructionDoes at least one student substantively edit text created by another student?
Commenting Does at least one student comment upon another student’s work on the wiki?
Discussion Do students respond to each others’ comments for at least four conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
Concatenation
Concatenation Do multiple students add discrete sections of text to the same page?
Copyediting Does at least one student copyedit text created by another student?
Co-ConstructionDoes at least one student substantively edit text created by another student?
Commenting Does at least one student comment upon another student’s work on the wiki?
Discussion Do students respond to each others’ comments for at least four conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
Complex Communication
Concatenation Do multiple students add discrete sections of text to the same page?
Copyediting Does at least one student copyedit text created by another student?
Co-ConstructionDoes at least one student substantively edit text created by another student?
Commenting Does at least one student comment upon another student’s work on the wiki?
Discussion Do students respond to each others’ comments for at least four conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
Co-Construction
Concatenation Do multiple students add discrete sections of text to the same page?
Copyediting Does at least one student copyedit text created by another student?
Co-ConstructionDoes at least one student substantively edit text created by another student?
Commenting Does at least one student comment upon another student’s work on the wiki?
Discussion Do students respond to each others’ comments for at least four conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
Complex Communication
Concatenation Do multiple students add discrete sections of text to the same page?
Copyediting Does at least one student copyedit text created by another student?
Co-ConstructionDoes at least one student substantively edit text created by another student?
Commenting Does at least one student comment upon another student’s work on the wiki?
Discussion Do students respond to each others’ comments for at least four conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
0 50 100 150 200 250 300 350 4000
5
10
15
20
Days
Co
mp
lex
Co
mm
un
ica
tio
n S
co
re
Prototypical wiki trajectory for Complex Communication WQI subdomain (n=406)
Selected Empirical Growth Plots
Rate Day
Com
posi
te W
iki Q
ualit
y S
core
Distribution of Complex Communication subdomain score in 406 U.S., K-12 wikis on final occasion of
measurement
Complex Communication Score
Percentage Frequency Count
0 88.92 3611 6.40 262 1.48 63 .25 14 1.48 65 .74 36 0 07 .74 3
Distribution of collaborative behaviors in 406 U.S., K-12 wikis on final occasion of measurement.
Percentage Frequency Count
Concatenation 5.91 24
Copyedit 1.72 7
Co-construction 4.43 18
Comment 6.4 26
Discussion 2.22 9
Planning 2.46 10
Scheduling 1.72 7
01
23
45
67
89
1011
1213
1415
1617
1819
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
Information ConsumptionParticipationExpert ThinkingNewMedia LiteracyComplex Communication
Wiki Quality Subdomain Scores(Averages for each WQI Level) (n=255)
n=102
n=49
n=37
n=6n=4
n=4n=5
n=7
n=5
n=9
n=7n=1
1n=3
n=3
n=2
n=1
SourceForge
• 70% of projects have only 1 developer, • 14% of projects have 2 developers• 14% of projects have 3-9 developers• 2% of projects have more than 10 developers
Theoretical debates in wikis and collaboration (IJCSCL)
Are students meaningfully collaborating?
Dohn (2009): Web 2.0 tools are theoretically well-suited for collaborative knowledge production practices, but the educational logics of individual assessment prevent these practices from being realized.
Or just posting in the same place?
Glassman and Kang (2011): wikis not only enable students to participate in knowledge building, but allow the enactment of Dewey’s vision
Implications• Empirical: Few wikis provide
opportunities for student collaborative writing– Very little copyediting or rich discussion– “I DON’T CARE DO UR OWN PAGE!,” (Grant,
2009).
Implications• Hopeful: Research to develop strategies
for scaffolding online student collaboration is really important
• Despairing: What does it mean if Dohn is right? What if the institutional structures of schooling limit collaborative learning?
blogs.edweek.org/edweek/edtechresearcher
Back Deck
What subjects are wikis used for? (n=411)
Education
Classics
ESL
Business
Health/PE
Modern FL
Contained Elementary
Art
Library
Math
Computer Science/ Technology
Science
Social Studies
English / Language Arts
0 20 40 60 80 100 120 140
2
4
5
6
8
10
20
22
26
45
60
61
70
120
49
1,000K
10K
1,000
100
10
1
100K
# of Cases
Days Weeks Months Years
Design Research
Interviews
Surveys
Simulations
Semantic Analysis
Content Analysis
Discursive Analysis
Observational Research
Duration of data collection and capture
Time/Scale Web 2.0 Research State Space Modeling
Usage Statistics
Seconds
Biometric Analysis
Prototypical wiki quality trajectories in subject areas, controlling for SES (n=259)
0 50 100 150 200 250 300 350 400 4500
5
10
15
20
25
Days
Wik
i Q
ua
lity
Sc
ore
Social StudiesEnglish
Computer ScienceScience
Math
0 50 100 150 200 250 3000
10
20
30
40
50
60
70
80
90
100
MrBoyersClass.Pbworks.comPage Saves by Day
Page Saves
Days
ALLPS
0
10
20
30
day
0 100 200 300 400 500 600 700 800 900
Moving average of wiki development measured in page saves