making sense of online learning: frames, rubrics, tools & coding systems for analyzing...

Making Sense of Online Learning: Frames, Rubrics, Tools & Coding Systems for Analyzing

Asynchronous Online Discourse

Theresa FlynnPepperdine Universitytflynn@pepperdine.edu

Linda PolinPepperdine Universitylpolin@pepperdine.edu

Co-construction of Knowledge & Asynchronous Online Discourse

ASSUMPTIONS:

collaborative learning is happening

online

Asynchronous discourse is a rich source of data for

analysis

Research Questions

• Can we capture evidence of learning in asynchronous online settings?

• If so, how?

• Are there indicators that “deep learning” is

occurring in asynchronous learning settings?

Study Context

• Pepperdine University’s 90% Online Masters in Educational Technology Program—2002/2003

• 5th year of the program

• 3 trimesters July 2002—July 2003

• 23 students—across the USA and abroad

• Only 3 face-to-face meetings in the 13 month program

Rubric Development

• Audience– To whom are messages being addressed in

newsgroups?

• Discourse Function– What roles do different “types” of posts serve

in the learning process?

Audience Rubric

• Self

• Specific Person(s)

• Instructor

• Group

• Other

Discourse Functions Rubric

• Rumination

• Storytelling

• Disagreement/Argumentation

• Social Interaction

• Procedural/Logistical

Discourse Functions Rubric

• Acknowledgement

• Reference/Resource

• Inquire

• Other

Thread Selection

• 2 threads from 1st trimester

• 2 threads from 2nd trimester

• Length of threads—looked for active posting

• Topic:• 2 threads about course readings (chapters)• 2 threads about class process (asynchronous &

synchronous classes)

Results of Coding: Audience

Chapter Threads

Chapter 4: Workplace Settings

Group

38%

Specific Person(s)

62%

Chapter 3: Schooling

Group

50%

Specific Person(s)

50%

Process-Oriented ThreadsSynchronous Chats

Group

40%

Specific Person(s)

60%

Posting Protocol

Specific Person(s)

43%

Instructor

4%

Group

53%

Results of Coding: Discourse Functions

Instructor: A

Trimester 1

# of messages: 29

Thread Length: 20 days

# of Unique Participants: 15

Total # of class members: 23

Chapter 4: Workplace Settings

Rumination

24%

Storytelling

44%

Social Interaction

3%

Acknowledgement

5%

Reference/Resource

3%

Inquiry

21%

Rumination

Storytelling

Argumentation

Social Interaction

Procedural/Logistical

Acknowledgement

Reference/Resource

Inquiry

Other

Instructor: B

Trimester 2

# of messages: 28

Thread Length: 26 days

# of Unique Participants: 17

Total # of class members: 23

Chapter 3: Schooling

Rumination34%

Storytelling32%

Social Bonding4%

Acknowledgement4%

Reference/Resource13%

Inquiry11%

Other2%

Rumination

Storytelling

Argumentation

Social Bonding

Procedural/Logistical

Acknowledgement

Reference/Resource

Inquiry

Other

Instructor: A

Trimester 1

# of messages: 16

Thread Length: 8 days

# of Unique Participants: 11

Total # of class members: 23

Synchronous Chats

Rumination

14%

Storytelling

5%

Procedural/Logistical

24%

Acknowledgement

28%

Reference/Resource

5%

Inquiry

10%

Other

0%

Social Interaction

14%

Rumination

Storytelling

Argumentation

Social Bonding

Procedural/Logistical

Acknowledgement

Reference/Resource

Inquiry

Other

Instructor: B

Trimester: 2

# of messages: 29

Thread Length: 11 days

# of Unique Participants: 22

Total # of class members: 23

Posting Protocol

Procedural/Logistical

33%

Acknowledgement

13%

Reference/Resource

0%

Inquiry

10%

Social Interaction

38%

Rumination

3% Storytelling

3%

Rumination

Storytelling

Argumentation

Social Bonding

Procedural/Logistical

Acknowledgement

Reference/Resource

Inquiry

Other

Validity: Interviews

• 8 students interviewed

• All agreed with the categories presented in both rubrics

• “The rubric covers it—You’ve done a fantastic job framing all of this. I never thought it had structure”

Validity: Interviews• “I think [the self category] is right on track—these are the

posts that usually trigger the best discussions—but also sometimes the hardest to share”

• “Very rarely do I feel that I am writing for the benefit of the professor”

• “When I ruminate it’s because I’ve reflected on something. When I’m storytelling it’s because I’m reflecting on a previous experience and how it’s linked to the current learning experience. When I’m disagreeing it’s because I’m reflecting on how and what you’re saying jives or doesn’t jive with a previous belief I have. In other words, everything that I’m saying in newgroups is a form of reflection”

Reliability

• 2 coders

• Inter-rater agreement

• 89% agreement in Discourse Functions category

• 94.5% agreement in Audience category

Disagreement/Argumentation

• No messages coded as disagreement/argumentation in any thread

• Closer-examination revealed we had missed several instances of disagreement

• Dissent was subtle

• Only apparent when examining the entire context of the newsgroup thread

Disagreement/Argumentation: A Closer Look at the Text

• Tension between “Old-timers” & “Newcomers”

• Sometimes “newcomers” leave or “jump ship”

• “Do students have the option of jumping ship?”

• MLK reference

• “But what do you do as the professional stagnating in a place unwilling to change?”

Conclusion

• Don’t sacrifice validity for reliability

• Large degree of inference involved in coding messages—makes coding more difficult

• The message alone as a unit of analysis is insufficient

• Must consider the context of the discourse

Student Insight

The most important part of the newsgroup process is not the reading or the posting—it’s the part in between where I’ve read it, and then I reflect on it before I post a reply. It’s that little, in that processing place”.

Context

AND

Content