1

Evidence-based practices in elearning. Collaborative learning in higher education:

empirical evidence.

Prof. dr. Martin Valckehttp://allserv.ugent.be/~mvalcke/CV/CVMVA.htm

Hamburg

February 4, 2007

2

Structure• Collaborative learning without ICT• Setting the scene• But does it lead to learning?• Group characteristics• Task characteristics

– Scripting– Roles– Tagging

• Student characteristics & support: peer tutoring• Conclusions

3

Conclusions

• Collaborative learning: don’t forget « lessons learned »

• Collaborative learning is part of larger learning environment

• Adding structure is the key: roles, scripting, tagging

• Coaching, tutoring, … has an impact• Management issues

4

« Collaborative learning is in the air »

« Everyone wants it. It is the instructional strategy, perhaps the strategy of the decade »

5

What do we know about collaborative learning without ICT?

What does the research say?

6

Collaborative learning without ICT?

• Meta-analysis collaborative learning research– Slavin (1996)

– Johnson & Johnson (1989)

• “The research has an external validity and a generalizability rarely found in the social sciences.”

7

Collaborative learning without ICT?

• Consistent and overwhelming positive impact on performance, motivation, social skills, development of metacognition, etc.

• But, why has it not been implemented to a larger extent?

8

9

Design guidelines

1. Garantee that there are shared learning objectives in a team

2. Build on team responsibility to reach the goals.

3. Build individual responsibility to reach goals.

4. Guarantee equal opportunities in the team activities.

5. Embed a level of competition and/or comparision.

10

Design guidelines

6. Break down larger tasks into subtasks.

7. Take into account individual differences (level, interest, intentions, ...).

8. Blend group activities with face-to-face activities.

9. Develop communication skills.

10.Monitor communication processes.

11

Setting the scene

• University• Large groups of 1st year students (N=286)• Online learning environment• Computer Supported Collaborative

Learning (CSCL): part of this environment• Course ‘Instructional Sciences’• 35 groups of 8 students working in online

groups

12

Integration larger learning environment

13

14

15

16

17

But does this invoke relevant learning?

• Collaboration does not lead automatically to high quality learning.

• There is a need guidance and online support in CSCL settings that is comparable to the need of classroom support in face-to-face settings (Lazonder, Wilhelm, & Ootes, 2003).

18

But does this invoke relevant learning?

• First generation CSCL-research:– Naive use of cooperative learning– Medium orientation– Neglection of context / individual / objectives– Over-estimation of potential technology

19

Does it invoke relevant learning?• First generation:

– Management problems

– No insight into structure of dicsussion

– Low task focus (Henri, 1982)

– Low levels of cognitive processing: new facts, concepts; hardly theory construction, application, evaluation

– Time on task problem

– What with students who are not active?

– …

20

But does this invoke relevant learning?

• Second generation CSCL-research:– Focus on “affordances”– Attention paid to “design guidelines”

21

Applying design guidelines

1. Shared learning objectives

2. Team responsibility

3. Individual responsibility

4. Equal opportunities

5. Level of competition or comparision.

22

Applying design guidelines

6. Subtasks.

7. Individual differences

8. Blend group and face-to-face activities

9. Develop communication skills.

10.Monitor communication processes

23

Design guidelines ~ 3 sets of variables

Taskcharacteristics

Learner characteristics& support

GroupCharacteristics

24

Design guidelines ~ 3 sets of variables

• Group:– Size– level of interaction

• Task characteristics:– Nature of task (open, theme)– Roles (content)– Roles (communication)– Tagging– Timing of role assignment

• Learner: characteristics and support

25

Learning:Nature of dependendent variables

• Level of interaction

• Level of knowledge construction

• Learning performance (test scores)

• Level of critical thinking

• Self & group efficacy

26

Group characteristics

27

Group size

• Differential impact

small (8-10), average (11-13 , large (15-18)

28

Level of interaction

29

Task structure

30

Roles

• Pharmacy education

• 5th year students

• 5 months internship

• Lack of integrated pharmaceutical knowledge

31

32

Roles• Content roles:

– Pharmacyst– Pharmacyst assistant– Theorist– Researcher– Intern

• Communication roles:– Moderator – Question-asker– Summarizer – Source researcher

33

34

Exchange

35

S. TIMMERS, M. VALCKE*, K. DE MIL & W.R.G. BAEYENS (in press). The Impact of Computer Supported Collaborative Learning on InternshipOutcomes of Pharmacy Students. Interactive Learning Environments

ICSIntegrated Curriculum Score

36

S. TIMMERS, M. VALCKE*, K. DE MIL & W.R.G. BAEYENS (in press). The Impact of Computer Supported Collaborative Learning on InternshipOutcomes of Pharmacy Students. Interactive Learning Environments

LKCLevel knowledge Construction

37

Timing roles

• 1ste year course “instructional sciences”• N 250• 20 discussion groups • Transcripts of the entire 12 week discussion

period • 4 discussion themes of 3 weeks each • About 4818 messages or 60450 lines of text

38

Timing: introduction roles

Them e 3

Them e 2

Them e 4

Them e 1

Them e 3

Them e 2

Them e 4

Them e 1

39

Timing: introduction roles

Them e 3

N o R oles

Them e 2

R oles

Them e 4

N o R oles

Them e 1

R oles

Them e 3

R oles

Them e 2

N o R oles

Them e 4

R oles

Them e 1

N o R oles

40

Roles

• Starter: start off the discussion, give new impulses every time the discussions slack off

• Moderator: monitor the discussions, stimulate other students, ask critical questions, inquire for opinions

• Theoretician: bring in theory, ensure all relevant theoretical concepts are used in the discusion

• Source searcher: seek external information on the topics, go beyond the scope of course reader

• Summarizer: post interim summaries, make provisional conclusions, post final summary

41

Moderator

42

Source

43

Gunawardena, Lowe, & Anderson (1997)

• Level 1: sharing/comparing of information• Level 2: the discovery and exploration of dissonance or

inconsistency among ideas, concepts or statements• Level 3: negotiation of meaning / co-construction of

knowledge• Level 4: testing and modification of proposed synthesis or

co-construction• Level 5: agreement statement(s) / applications of newly

constructed meaning

44

Timing: introduction roles

45

Timing: introduction roles

• Role/No-Role condition reaches significantly higher levels of knowledge construction in two themes

• Even when the role support is cut back

46

Differential impact roles

47

M e an LK C for ro le type s

0

0.5

1

1.5

2

T hem e 1 & 2 T hem e 3 & 4

T hem e 1 & 2 1 .315 1 .434 1 .433 1 .34 1 .714 1 .386 1 .218

T hem e 3 & 4 1 .228 1 .43 1 .277 1 .146 1 .524 1 .258 1 .734

St art er M oderat o r T heoret icianSource

s earcherSum m aris er

N o ro le in R S group

N o ro le-sup p ort ed

Differential impact roles

48

• “There is a differential

impact of the different roles”

No role condition

Starter

Moderator

Summarizer

Theoretician

Source Searcher

No role

Ref.cat.

=

=

+

+

+++

+

Differential impact roles

49

Tagging

50

Tagging

51

Tagging

• Aims of tagging:–it obliges students to reflect upon the nature of their

contribution and on how it will add to the ongoing discussion

–the labels improve the outline of the discussion and indicate the predominance or absence of one or more thinking types

• Example: De Bono’s (1991) thinking hats in view of developing critical thinking

52

Tagging• Garrison (1992) identifies five stages of critical

thinking:

• Problem identification• Problem definition• Problem exploration• Problem evaluation/applicability• Problem integration

53

De Bono’s (1991) thinking hats

Critical Thinking Thinking hats

Problem identification White hat

Problem definition Blue hat

Problem exploration Green hat

Problem applicability Black hat

Problem integration Yellow hat

Red hat

54

Tagging

• 3th-year university students

• enrolled for the course ‘Instructional Strategies’ (N=35)

• 6 groups of 6 team members

Experimental condition

Control condition

4 groups

23 students

2 groups

12 students

Tag posts by a

thinking hat

No tags to posts

required

55

Tagging

• Evidence for critical thinking in both conditions

• Significant deeper critical thinking in experimental condition (F(1, 416)=364.544; p<.001)

0.88

0.54

0

0.2

0.4

0.6

0.8

1

Experimental condition Control condition

56

Tagging• Patterns are quite similar for both conditions• Experimental condition

–more focused discussions (F(1, 415)=1550.510; p<.001) –more new info and ideas (F(1, 352)=21.955; p<.001) –more linking facts ideas (F(1, 31)=3.024; p<.092)

-1

-0.5

0

0.5

1Relevance

Importance

Novelty

Bringing outside knowledge

Justification

Critical assessment

Linking ideas

Resolving ambiguity

Practical utility

Focus of the discussion

Experimental condition Control condition

57

Impact of tagging• Multinomial logistic regressions indicate that

• being in the experimental condition increases the probability of engaging in in-depth discussions radically (p<.001)

• experimental students post 2.73 as many messages adding new problem-related information to the discussion (p=.001)

• experimental students were 2.95 times more likely to add new ideas for discussion (p=.009).

• linking ideas and critical assessment occur rarely. When it occurs, it is in the experimental condition.

58

Impact of tagging over time

• Experimental students show a rather constant level of critical thinking

• Control students show a decrease during problem identification (F(1, 416)=1408.838; p<.001) and exploration (F(1, 415)=1101.513; p<.001)

-0.80

-0.60

-0.40

-0.20

0.00

0.20

0.40

0.60

0.80

1.00

1.20

Stage 1 Stage 2 Stage 3 Stage 4 Stage 5

Experimental condition Control condition

59

But ….

• Studies with freshman: no significant impact.

• Tagging interferes with knowledge construction process.

• BUT … tutoring helps

60

But …• More critical thinking in labeling condition

after correction for the different tutor styles–Overall depth of CT–Importance–Discussion of ambiguities–Input of new information–Linking of information–Critical assessment–Defining the problem–Integrating new knowledge

61

Learner characteristics & learner support

62

Support: peer tutors• Given critical results of some CSCL-studies, demand for

structure:– Scripting (roles, tagging, …)– Facilitators (Bonk, Wisher, & Lee, 2004; Garrison,

Anderson, & Archer, 2000; Rickard, 2004; Salmon, 2000)

– Prior research, however, revealed that peer tutors were mainly engaged in social support, while less attention was paid to stimulating ‘knowledge construction’ and ‘personal development’(De Smet, Van Keer, & Valcke, in press)

– Therefore extra support for tutors

63

Method• Effect study: Impact of

labeling on patterns in tutor support.

• E-moderating model(Salmon, 2000)

64

Peer tutoring• Cross-age peer tutoring blended in with

online discussion groups

• One peer clearly takes a supportive role• Fourth-year students help freshmen• Ratio = 1/10• Open-ended group assignments • 2 weeks discussion per theme• 1 trial discussion and 4 discussion themes

65

3 tutor training conditions• Control (N=39)

• All-round instructions, No labeling requirements,

• No pre-service exercises, Focus groups

• Labeling (N=18)

• E-moderating instructions, Labelling requirements

• Pre-service exercises, Focus groups

• Non-labeling (N=17)

• E-moderating instructions, No labeling requirements,

• Pre-service exercises, Focus groups

66

Labeling tutoring activity• Labeling involves self-monitoring• E-moderating taxonomy (Salmon, 2000)

• Access and motivation (Step 1)

• Socialisation (Step 2)

• Information-exchange (Step 3)

• Knowledge construction (Step 4)

• Personal development (Step 5)

67

Impact of labeling on patterns in Emoderating

0

5

10

15

20

25

30

35

40

Access andmotivation

Socialisation Information-exchange

Know ledgeconstruction

Personaldevelopment

Control (N=39)

Non-labelling (N=17)

Labelling (N=18)

68

Impact of labeling in tutoring

• Multinomial logistic regression analysis >>• Variables treated as nominal

• Independent of the training condition, tutors filled all the roles required of e-moderators

• In each training condition, vast majority for ‘information-exchange’ (step 3)

69

Impact of labeling in tutoring

• Compared to the control condition, both the labelling and non-labeling condition positively influenced the adoption of tutoring support that stimulates:

– ‘socialisation’ (step 2)– ‘information-exchange’ (step 3)– ‘personal development’ (step 5)

• Labelling enhanced tutors’ facilitation for ‘personal development’ (step 5)

70

Conclusions

• Collaborative learning: don’t forget « lessons learned »

• Collaborative learning is part of larger learning environment

• Adding structure is the key: roles, scripting, tagging

• Coaching, tutoring, … has an impact• Management issues

71

Publications• De Smet, M., Van Keer, H., & Valcke, M. (in press). Blending asynchronous discussion

groups and peer tutoring in higher education: An exploratory study of online peer tutoring behaviour. Accepted for publication in Computers and Education.

• De Smet, M., Van Keer, H., & Valcke, M. (in press). Cross-age peer tutors in asynchronous discussion groups: A study of the evolution in tutor support. Accepted for publication in Instructional Science.

• De Wever, B., Schellens, T.,Valcke, M & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: a review. Computers & Education, 46(1), 6-28.

• De Wever, B., Van Keer, H., Schellens, T., & Valcke, M. (in press). Applying multilevel modelling on content analysis data: Methodological issues in the study of the impact of role assignment in asynchronous discussion groups. Accepted for publication in Learning and Instruction.

• De Wever, B., Van Winckel, M. & Valcke, M. (in press). Discussing patient management online: The impact of roles on knowledge construction for students interning at the paediatric ward. Accepted for publication in Advances in Health Sciences Education.

• Schellens, T. & Valcke, M. (2005). Collaborative learning in asynchronous discussion groups: What about the impact on cognitive processing? Computers in Human Behavior, 21(6), 957-975.

72

Publications• Schellens, T. & Valcke, M. (2006). Fostering knowledge construction in university

students through asynchronous discussion groups. Computers & Education. 46(4), 349-370.

• Schellens, T., Van Keer, H. & Valcke, M. (2005). The impact of role assignment on knowledge construction in asynchronous discussion groups: a multilevel analysis. Small Group Research, 36, 704-745.

• Schellens, T., Van Keer, H., & Valcke, M. (2007). Learning in asynchronous discussion groups: A multilevel approach to study the influence of student, group and task characteristics. Accepted for publication in Journal of Behavior and Information Technology. 26(1), 55-71.

• Schellens, T., Van Keer, H., De Wever, B., Valcke, M. (in press). Tagging Thinking Types in Asynchronous Discussion Groups: Effects on Critical Thinking. Accepted for publication in International Journal of Interactive Learning Environments.

• Timmers, S., Valcke, M., De Mil, K. & Baeyens, W.R.G. (in press). CSCLE and internships of pharmacy students - The Impact of Computer Supported Collaborative Learning on Internship Outcomes of Pharmacy Students. Accepted for publication in International Journal of Interactive Learning Environments.

• Valcke, M. & De Wever, B. (2006). Information and communication technologies in higher education: Evidence-based practices in medical education. Medical Teacher, 28, 40-48.

73

Evidence-based practices in elearning. Collaborative learning in higher education:

empirical evidence.

Prof. dr. Martin Valckehttp://allserv.ugent.be/~mvalcke/CV/CVMVA.htm

Hamburg

February 4, 2007