pedagogical discourse: connecting students to past discussions and peer mentors within an online...

Pedagogical Discourse: Connecting Students to Past Discussions and

Peer Mentors within an Online Discussion Board

Jihie Kim

http://ai.isi.edu/discourse

[email protected]

http://ai.isi.edu/discourse

Pedagogical Discourse Jihie Kim

“Talk to as many other people as possible.

CS is learned by talking to others, not by reading,

or so it seems to me now.”

-- Advice from a computer science studenthttp://www-scf.usc.edu/~csci402/


Discussion Board and Corpora

13 semesters running…

Six courses Undergrad/Graduate USC/Non-USC Almost 700 students Over 7000 messages

13 semesters running…

Six courses Undergrad/Graduate USC/Non-USC Almost 700 students Over 7000 messages

Extensible open-source discussion board (phpBB) serves as a platform for bridging ISI research and USC teaching practice


Student Messages in an Undergraduate Operating Systems Course

Text is incoherent and

ungrammatical.

Problem description: Non-

factoid questions are difficult

to identify, dependent on

context, and may include

multiple sentences or

paragraphs.

Answers require explanations.


Thread Length Distribution

Data from an undergraduate CS Course

0

100

200

300

400

500

600

# of threads

1 3 5 7 9 11 13 15 18 20 31 # of posts

Statistics of thread length

Data from a graduate CS Course

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6 8 9 10 12 16

# of threads

# of messages

Threads are often very short, many consisting of only 1-2 messages

Students jump into programming details without understanding larger picture or related concepts

TA and instructors are not always available to fully guide interactions

# of messages

Need of Discussion Assessment and Scaffolding


Discussion Scaffolding

PedaBot: Promote reflection

Find useful information from past discussions • Past discussions could provide suggestions on current problems

although the suggestions may not present exact answers

• Promote further discussion on related technical topics

• Graduate discussions are more concept oriented than undergraduate discussions – could provide interesting references for similar problems

MentorMatch: Promote collaboration among students

Identify student ‘experts’ on topic

Connect help-seekers to mentors


• Pre-process messages from past discussions (both undergr & grads) Model messages with technical terms used Divide related text into coherent sub units (tiles) (e.g.TextTiling) Model topics in the discussions [Feng et al., AAAI-06]

Approach to Generating PedaBot Response

• Identify problems/questions in the current discussion Pick the first post in a thread (80% of first posts are questions)

• In our first study, we include only the first posts• Automatically identifying “Question” messages or discussion

focus --[Ravi & Kim - AIED 2007; Feng et al., HLT-NAACL 06]

• Match students’ problems to similar past discussions Current and past messages represented as term vectors (with

TF/IDF, LSA) Match by similarity (use cosine similarity) Filter candidate responses by topics --[Kim et al., ITS-2008]

• Generate response Return most similar message or set of messages in thread


Evaluating PedaBot Responses : Design

“Current” discussion corpus Fall 2006 – 207 msgs. (first message posted in threads)

Past discussion corpora (taken from 4 semesters prior to Fall 2006) Student messages from Undergraduate discussions – 3788 msgs Instructor messages from Undergraduate discussions - 531 msgs Student messages from Graduate discussions – 957 msgs

PedaBot responses rated by 4 people – average ratings used

Evaluation of system responses – 2 criteria Technical quality of retrieved results Relevance of retrieved results w.r.t asked question


Technical Quality of results (messages) returned by the system

Preliminary Evaluation (a): Technical Quality Rating

2.923.15 3.31

0

1

2

3

4

5

Student Msgs. fromUndergraduate

Course

Instructor Msgs.from Undergraduate

Course

Student Msgs. fromGraduate Course

Technical Quality Rating ( 1 - 5 )

Rating

5 – Very High Quality 4 – Good Quality 3 – Technical 2 – Somewhat technical 1 – Not technical

Technical Rating = 5Page table loading into memory….When we have the page table in disk, we cannot map the physical pages because the page tables are larger than physical space. Using memmap will not work…

M1

M5 How many points will be taken off for assignment #2, if the first test case does not work? …

Technical Rating = 1

…


...

Relevance - How “related” was the message returned by the system w.r.t the question asked by the student ?

Preliminary Evaluation (b) : Relevance/Similarity Rating

2.52 2.67

1.74

0

1

2

3

4

5

Student Msgs. fromUndergraduate

Course

Instructor Msgs.from Undergraduate

Course

Student Msgs. fromGraduate Course

Relevance Rating ( 1 - 5 )

Rating

5 – Very Good Response 4 – Good Response 3 – Related 2 – Somewhat Related 1 – Unrelated

Relevance Rating = 5

Currentmessage

What is RPC ?

M1

M5

RPC stands for “Remote Procedure Call” . It is used in …

How do we implement the test cases for virtual memory in assignment #3 ?

Relevance Rating = 1


PedaBot User Interface

Relevant past discussion or document


Whole discussion can be viewed

Students can rate retrieveddiscussions


Pilot Study of PedaBot

Integration into a live student discussion board (Fall 2007)

Upper-level undergraduate Operating Systems course offered

by the Computer Science department at the University of

Southern California

(Male N=104, Female N=15)

Student surveys collected in Fall 2007 and Fall 2008


PedaBot Usage (Fall 2007)Number of students who… Male Female Combined

Registered on discussion board 104 15 119

Participated in discussions (discussants) 82 9 91

Initiated discussion threads 70 8 78

Viewed entire discussion context of PedaBot retrieved messages

5567%

778%

6268%

Rated PedaBot retrieved messages 15 0 15

Average number of Pedabot ... per student

Male Female Combined

Discussion details viewed(#viewings / #viewers)

6.3 3.3 6.0

Results rated (#ratings / #raters) 2.6 0 2.6


Difference in thread length w, w/o PedaBot

Hypothesis: Use of Pedabot for reflection will increase student participation in discussions

Initial analysis

Fall 2007 with PedaBot without PedaBot

Average number of

messages per thread

Male 3.44 (426/124) 3.15 (533/169)

Female 5.42 (65/12) 2.00 (12/6)

Combined 3.39 (431/127) 3.12 (543/174)


Discussion Scaffolding

PedaBot: Promote reflection

Find useful information from past discussions • Past discussions could provide suggestions on current problems

although the suggestions may not present exact answers

• Promote further discussion on related technical topics

• Graduate discussions are more concept oriented than undergraduate discussions – could provide interesting references for similar problems

MentorMatch: Promote collaboration among students

Identify student ‘experts’ on particular topic

Connect help-seekers to experts (mentors)


MentorMatch

Motivation Get students the help they need Promote collaboration between help-seekers and mentors Peer replies better than instructor replies at furthering discussion Acknowledge mentors for their role in assisting classmates

Pilot Study Integrated into live student discussion board (Fall 2008)

• Run 5 weeks (10 weeks into 15 week semester) Design

• Do students use tool / find it helpful?• Does notifying mentors encourage participation?• Do mentors receive better grades?


Student topic profiles

Build profile of topic categories for each student Classify each student’s message using topic models Models based on higher-level topics covered in course and

textbook index terms that map to them (Feng, Kim, Shaw, Hovy, AAAI-2006)

Distinguish between help seekers & mentors (experts) Messages weighted based on type: question or response - for now, initial post or reply (most initial posts are questions) Include contributions of short messages - e.g. yes/no, acknowledgement

∑−

=

+=1

1

),(*),(*),(i

kkkii TcMSimwTcMSimwTcMScore


Students can elect to send mail to mentors


Can view class and personal mentor info

“Topic experts” link opens a window with mentor names and topics. “Topics requesting your expertise” section displays links to discussions on topics of the student’s expertise.


Did students find tool useful?

Activated ‘Topic mentors’ link. 13/25

Students who commented that they were unaware of the link. (Others said it wasn’t needed or they didn’t think it would help.)

7/9

Saw ‘Topics requesting your help’ section. 13/25

Activated a ‘Topics requesting your expertise’ link. 9/13

Students who sent email to a mentor. 6/25

They never noticed the link 6/11

They didn't think it was necessary or assumed someone else would respond. (A third was a topic mentor herself.)

2/11

Students who reported receiving a request for their assistance. 9/25

Students who were sent email and responded or tried to respond. 5/9

Interest and usefulness (N=20)Feature avg not low n/a some high

How interesting 4.2 1 3 7 9

How useful 4.05 2 3 7 8

Participation (N=20)


Did notifying mentors encourage participation?

0

2

4

6

8

10

12

14

s1 s3 s5 s7 s9 s11 s13 s15 s17 s19 s21 s23

Students

Post Frequency

# Notification topics

# Replies on the notified topics

# Replies on following other topics


Do mentors receive better grades?

TopicExperts

(students)

ExpProjectGrade

NEAProjectGrade

ExpAllMsg

Score

NEAAllMsg Score

ExpReply Score

NEAReply Score

Thread Synchronization

Exp 1 14 29 0.41 0.19 0.29 0.05

Exp 2 28 29 1.01 0.19 0.29 0.05

Exp 3 32 29 0.83 0.19 0.21 0.05

System Calls & Multi-Programming

Exp 1 27 34 1.79 0.13 1.39 0.08

Exp 5 36 34 0.36 0.13 0.25 0.08

Exp 6 32 34 0.44 0.13 0.27 0.08

Exp 7 21 34 1.80 0.13 1.74 0.08

Exp 8 37 34 0.68 0.13 0.61 0.08

• Project grades (max=40), usage stats & profiling scores for 2 projects (topics)• Scores of experts (Exp) compared to averages of non-experts (NEA) Grades Topic Profiling Score


Groups formed from discussion: Network Analysis(Kang, Kim, Shaw 2009)

<Group distribution in fall 2008 semester>

Active Group Participants : 47: Instructor, 1461: TA 1320, 1425, 1348, 1437,

1277, 1459 BRIDGE:

1289, 1294,1435,1371 All Bridge Students

received good grade

24


Summary

Discussion Scaffolding Promote reflection (Kim, et al., ITS 2008; Feng et al., IUI 2006 ) Promote collaboration among students (Kim and Shaw, IAAI 2009; Shaw, Kim

and Supanakoon, AIED 2009)

Discussion Assessment Workflows (distributed computing) for large scale assessment Community detection and information flow (Kang, Kim, Shaw 2009) Identify discussion threads with unanswered questions (Ravi & Kim, AIED

2007) Assess student participation (Kim & Beal AERA 2006; Ravi et al., 2007; Kim et

al., AIED 2009; Kim et al., AIED 2007; Kim et al., AIED 2005) Assess topics discussed over time (Feng, Kim, Shaw, Hovy, AAAI-2006) Identify discussion focus (Feng, Shaw, Kim, Hovy, HLT-NAACL 2006) Assess tutor participation effect (Shaw, AIED 2005) Sentiment in student discussions (Wyner et al., 2009)

More details/papers available at: http://ai.isi.edu/discourse

Supported by NSF CISE/IIS and EHR/CCLI


Related Work

Dialogue modeling in email messages or blog (e.g. AAAI 2008 workshop on Enhanced Messaging)

• Email speech acts

• Requests and commitments Handling noisy data and high variance in text

(Knoblock et al., 2007)

Pedagogical dialogueInstructional discourse modeling (Yuan et al., 2008; Graesser et al., 2005;

McLaren et al., 2007; Boyer et al., 2008; Fossati 2008; ) Course topic and task modeling using information extraction

techniques(Roy et al. 2008; Jovanovic et al., 2006 )

Trace student e-learning activities (Israel and Aiken, 2007; Dringus and Ellis, 2005)

Scaffolding strategies for e-learning tools (Tang and McCalla 2005; Bari and Benzater, 2005; …)

Social Media analysis

pedagogical discourse: connecting students to past discussions and peer mentors within an online...

Documents

past messages

past discussions

undergraduate discussions

msgsstudent messages

msgsinstructor messages

question messages

discussions feng

undergr gradsmodel messages