longitudinal analysis of peer feedback in a writing-intensive course: a pilot study
Post on 24-Jan-2017
159 Views
Preview:
TRANSCRIPT
Longitudinal Analysis of Peer Feedback in a Writing-
Intensive Course: A Pilot Study
PI: Christina HendricksCo-PI: Jeremy Biesanz
University of British Columbia-Vancouver
Funded by the UBC Institute for the Scholarship of Teaching and Learning SoTL Seed Fund
Festival of Learning, June 2016Slides licensed CC-BY 4.0
Literature on peer feedbackReceiving peer feedback improves writing (Paulus, 1999; Cho & Schunn, 2007; Cho & MacArthur, 2010; Crossman & Kite, 2012)
Giving peer feedback improves writing(Cho & Cho, 2011; Li, Liu & Steckelberg, 2010)
GAPS: Most studies look at revisions to a single essay, not changes across different essays
Draft 1
Draft 2
Draft 3
Essay 1 Essay 2 Essay 3 Essay 4 Essay …n
PFB
PFB
PFB
PFB PFB PFB
Few studies look at “dose-response curve”
Pilot study research questions1. How do students use peer comments given and
received for improving different essays rather than drafts of the same essay?
2. Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal?
3. Does the quality of peer comments improve over time?
• Interdisciplinary, full year course for first-years
• 18 credits (English, History, Philosophy)• Students write 10-12 essays (1500-2000
words)• Peer feedback tutorials every week (4
students)
http://artsone.arts.ubc.ca
Toni Morrison, Wikimedia Commons, licensed CC BY-SA 2.0
Osamu Tezuka, public domain on Wikimedia Commons
Jane Austen, public domain on Wikimedia Commons
Friedrich Nietzsche, public domain, Wikimedia Commons
Data for pilot study 2013-2014
• 10 essays by 12 participants (n=120)
• Comments by 3 peers on essays (n=1218)
• Comments by instructor (n=3291)
• All coded with same rubric
Coding RubricCategories(plus subcategories, for 11 options)
• Strength of argument• Organization• Insight• Style & Mechanics
Numerical value
1: Significant problem2: Moderate problem3: Positive comment/praise
E.g., STREV 2: could use more textual evidence to support your claims
Change for future
Inter-coder reliability
Fleiss’ Kappa Intra-class correlation
Student comments (n=141)
All categories: 0.61 (moderate)Most used categories: 0.8 (excellent)
0.96 (excellent)
Essays (n=120) 0.71 (adequate)
3 coders:• Daniel Munro & Kosta Prodanovic
(undergrads, former Arts One)• Jessica Stewart (author, editor)
Change for future
LOOKING AT TRENDS IN COMMENTS OVER TIME
INSTRUCTORComments
-.28**
Strength
Style
Organiz.
Insight-.04*
Number of 2 comments over time
STUDENT comments
Strength
Style
Organiz.
Insight
-.16**
Number of 2 comments over time
INSTRUCTORComments .31***
Strength
Style
Organiz.
Insight
.08**
.19**
.11**
Number of 3 comments
STUDENTComments
Strength
Style
Organiz.
Insight
Number of 3 comments over time
HOW DOES ESSAY QUALITY CHANGE OVER TIME?
Essay quality improves linearly
b = .038t(107) = 2.1p = .037
Essays rated on a 7-point scale
MORE COMPLEX ANALYSES
Cross-lagged panel design with auto-regressive structure
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Path A: Instructor Comments
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant relationships• Ratings of 1 in Strength (-.12*) & Org. (-.23**)• Ratings of 2 in Strength (-.06*) & Style (-.08*)• Ratings of 3 in Str, (.11*), Insight (.35*), Style
(.15*) *p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
Path A: Student comments
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant relationships• Ratings of 2 in Insight (-.53*)• Ratings of 3 in Organization (.13*)
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
Path C: instructor commentsEssay Quality
Time 1Essay Quality
Time 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant effects don’t show up if split out by category• Comments ratings of 1 (.29**)• Comments ratings of 2 (.23*)• Comments ratings of 3 (.21, p=.057)*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p
<.00001
Path C: student commentsEssay Quality
Time 1Essay Quality
Time 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant relationships• Comments rated 2 in Strength (.22*) & Style
(.33**)• Comments rated 3 in Style (.31*)
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
Path D: Student & Instructor comments
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant relationship ONLY if combine student & instructor comments, & only for comments rated 1 (all categories combined): (.05, p=.06)
Research question 1How do students use peer comments given and received for improving different essays rather than drafts of same essay?
o Very little significant evidence of relationships in Path D
o No difference between comments given & received
Research question 2Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal?
o No evidence that there is any change over time in path D
o No difference between comments given or received
Research question 3Does the quality of peer comments improve over time?o No evidence of change over time in path
A
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Research Question 3, cont’d
Student/instructor agreement on average numerical ratings on each essay • tends to go down over time (-.04**)
• student ratings increase at only half the rate (.16*) that instructor’s ratings increase (.33*****)
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
Research Question 3, cont’dCorrelations on number of comments, students & instructor• No change in these relationships over
time
*p < .05, **p< .01, ***p< .001, ****p < .0001 *****p <.00001
Comment value 1
Comment value 2
Comment value 3
Strength 0.23*Organization 0.21* 0.17*Insight 0.17*Style
Some conclusionsPilot study: feasible for larger sample? Yes, if:
o instructors code essay quality rather than coders
o “chunk” essays for cross-lagged analyseso have easy collection of comments
References• Cho, K., & MacArthur, C. (2010). Student revision with peer and
expert reviewing, Learning and Instruction. 20, 328-338.• Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving
comments. Instructional Science, 39, 629-643. • Cho, K. & Schunn, C. D. (2007). Scaffolded writing and rewriting
in the discipline: A web-based reciprocal peer review system. Computers & Education, 48, 409–426
• Crossman, J. M., & Kite, S. L. (2012). Facilitating improved writing among students through directed peer review, Active Learning in Higher Education, 13, 219-229.
• Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536.
• Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 8, 265-289.
Thank you! Christina Hendricks
University of British Columbia-Vancouver
Website: http://blogs.ubc.ca/christinahendricksBlog: http://blogs.ubc.ca/chendricksTwitter: @clhendricksbc
Slides available: https://is.gd/PeerFeedbackPilot_FOL16
Slides licensed CC-BY 4.0
Capitals needed
underscore
top related