Conversations and Quality

Aiming for Educationally Valuable Talk in Online Discussions

Sedef Uzuner

Ruchi MehtaEducational Theory and Practice

University at AlbanyState University of New York

Introduction

“A worthwhile educational experience is embedded within a community of inquiry that is composed of teachers and students – the key participants in the educational process” (Garrison et al, 2000:88).

It is within this community of inquiry that learning occurs when participants use dialogue to question, draw inferences, make connections and validate knowledge about matters of significance (Larreamendy-Joerns & Leinhardt, 2006).

Review of literature

Previous studies that investigated talk in online discussion threads have focused on determining the quality of conversations based on critical thinking constructs. (See for example, Wickersham & Dooley, 2006; Jeong, 2003; Gilbert & Dabbagh, 2005; Hara, Bonk,& Angeli,2000; Garrison, Anderson, & Archer, 2001)

Our focus

This study is intended to analyze and characterize student posts in online discussion threads based on the concept of educationally valuable talk (EVT) – a new concept for determining the educational value of dialogue taking place in online discussions.

Conceptual framework

Educationally valuable talk (EVT) is an evolving construct, but the underlying basis for this notion remains firmly girded in Lev Vygotsky’s socio-cultural theory of learning :

knowledge is socially constructed; knowledge building is created between/among individuals in

their collaborative meaning-making through dialogue.

On dialogue and knowledge building…

Are all conversations concerned with knowledge building?

Sometimes linguistic transactions that take place in educational contexts are nothing more than mere interactions that are not sufficient to make construction or co-construction possible.

On dialogue and knowledge building…

Are some interactions educationally more desirable than others?

Quality is a polemical issue and judging any one kind of talk as a better form of communication than others is not the best practice. However, we argue that there are some talk types that are educationally more valuable, and therefore, more desirable than others.

Mercer’s “Exploratory talk”

In an attempt to investigate the talk of primary school children working together at the computer, Mercer (1994) coined the term “exploratory talk” to refer to a style of interaction in which “hypothesis are proposed, objections are made and justified, and new relevant information is offered” (27).

With his notion of “exploratory talk”, Mercer suggests that individuals learn better when their current view of knowledge is challenged, reformed, and elaborated through their interaction with others.

Educationally valuable talk (EVT)

EVT refers to a particular interactional pattern in online discussion threads characterized as dialogic exchanges whereby participants collaboratively display constructive, and at times, critical engagement with the ideas or key concepts that make up the topic of an online discussion, and build knowledge through reasoning, articulation, and reflection.

In other words…

Learning takes place when individuals engage in exploratory transactions through language.

Educationally less valuable talk (ELVT)

In opposition to EVT, the term educationally less valuable talk (ELVT) refers to talk that lacks substance in regards to critical and meaningful engagement with the formal content or the ideas that are put forward in the posts of others in an online discussion.

EVT indicators in online discussionsIndicator Defined Example Source

Exploratory Recognition of some confusion/curiosity or perplexity as a result of a problem/issue arising out of an experience/course readings; posing a problem and enticing others to take a step deeper into it.

“I wonder…….”

“I am not sure if what the author suggests…….”

“In the article X, the author said …. This brought up a few questions in my mind ….”

Mercer (1994);

Garrison, Anderson & Archer (2001)

Invitational Inviting others to think together, to ponder, to

engage by asking questions, requiring

information, opinion or approval.

“Jane says …….. What do you think?”

“Do you think ……?”

“The authors suggest …., no?

Emergent

Argumentational Expressing reasoning (with analogies, causal, inductive and/or deductive reasoning etc) to trigger discussion

“If teachers ……., then ……..”

“Teaching is like …………..”

“X is important because …….”

Kumpulainen (1996)

Critical Challenging or counter-challenging statements/ideas proposed by others OR playing devil’s advocate

“I agree that …. However, ……. Emergent

Heuristic Expressing discovery (similar to “A ha!” moments or expressions like “I find it!”); directing others’ attention to a newly discovered idea.

“I did not know that there is a name for XXX. I think XXX is …..Has anyone experienced that too?”

Kumpulainen (1996)

EVT indicators continued…Indicator Defined Examples Source

Reflective Examination of past events, practices (why/how they happened) or understandings in relation to formal content

“I’ve noticed that I had a tendency to ….. After reading X’s article, I’ve learned not to ……”

Emergent

Interpretive Interpretation of formal content through opinions that are supported by relevant examples, facts, or evidence.

“In my opinion X is …… Y is a good example of why …….”

Emergent

Analytical Interpretation of content through the analysis, synthesis, and evaluation of others’ understanding

“The original question was … Joe said … Mary said … As for me ….”

Emergent

Informative Providing information from literature and relating it to course content/topic of discussion

“I read an article about X once and the author said …. You can find more information about this in …”

Kumpulainen (1996)

Explanatory Chain of connected messages serving or intended to explain/make clear OR statements serving to elaborate on the ideas suggested in previous posts

“I want to build on your comment that ……..”

Emergent

Implicative Assertions that call for action OR statements whereby participants formulate a proposal/decision about how to achieve a certain end based on the insights they gained from the course readings/discussions

“Teachers should / should not ….”

“X must not be forced ….”

Emergent

Some cautionary notes … These proposed EVT indicators are not in hierarchical order. Determining the educational value of a discourse is subjective.

Talk that does not incorporate these indicators may have potential educational significance in a specific context. Likewise, talk that does not seem to be educationally valuable for some may have significance for others.

ELVT indicatorsIndicator Defined Examples Source

Affective Short posts that ONLY contain a statement of personal feelings (likes & dislikes) or appraisal (praising & thanking someone)

Questions or comments that add social presence to the discussion but do not contribute new information.

“I never liked Math either”

“Thank you for offering your insights into ….”

“I have been to your country once and I visited X, Y, Z when I was there”

Garrison, Anderson & Archer (2000)

Judgmental Short posts that ONLY contain brief statements of agreement or disagreement without elaboration.

“Yes, I agree with you ….”

“I do not think so”

Kumpulainen (1996)

Experiential Posts that only contain personal experiences, narratives, descriptions that are not followed by reflection

“I did the same thing when I was teaching X. I did A, B, C. It was fun”

Kumpulainen (1996)

Reproductional Repeating/reproducing the ideas mentioned/proposed in the previous posts without elaboration

“You are right, X is …… “ (followed by a sentence)

Kumpulainen (1996)

Miscellaneous Opinions that seem to be off topic OR statements regarding technical problems/course logistics

“I am unable to open Jay’s file…” Emergent

Some cautionary notes… To achieve knowledge building, each post has to be a

significant addition to the learning community’s pool of knowledge. Although not unimportant, experiential posts that include mere narratives or descriptions of personal experiences without reflection are not considered to be sufficient to meet this requirement. Therefore, such posts fall under the category of educationally less valuable talk

Posts that include judgmental and reproductional indicators are also of lesser educational value because they are brief posts that lack substance in regards to critical and meaningful engagement with the formal content or the ideas that are put forward in the posts of others in an online discussion.

Some cautionary notes continued…

Although posts that include affective indicators add social presence to the discussions, they do not have any teaching value; therefore, they do not allow participants to build knowledge.

As the name itself suggests, miscellaneous posts are interactions that are not concerned with knowledge building; therefore, they do not have any potential educational value.

Summary of our conceptual framework

There is no talk type that is unimportant; however, there are some that are educationally more valuable and desirable than others.

With the production of EVT, social interactions in online discussion threads will “progress beyond information sharing to knowledge construction” (Anderson et al, 2001: 9) and transform into purposeful and constructive dialogues which move toward the improved understandings of all parties involved.

Research Questions

1) What are the nature and characteristics of talk generated by learners in online discussions?

2) What are the indicators of educationally valuable talk in online discussion threads and how often do students engage in such conversations?

3) What factors seem to facilitate or hinder the production of educationally valuable talk in online discussions?

Methodology: Context of the study

A master’s level course offered through The State University of New York’s online instructional program - SLN (SUNY Learning Network) - provided the context of this study.

The course was co-taught by a professor and a teaching assistant (the first presenter)

It consisted of 7 modules that dealt with issues related to language, literacy and technology.

Participation in this course was asynchronous and the instructors placed a great deal of value in the discussion component of the course.

Methodology: Participants

19 graduate students participated in the course: 4 male & 15 female 4 were international students from Korea & China 13 had experiences taking online courses before 2 were PhD students

Methodology: Sources of Data

The study draws on data collected from a variety of sources:

1. Students’ online posts (N = 706) from 3 separate modules (Only the messages posted in the whole-class discussions area were analyzed, because in other learning activities (such as group/pair discussions), students were given the option to conduct their discussions on IM, the phone, or other means)

2. Direct observations Certain features of the course including the instructors’ pedagogical approach to facilitating discussions, learning activities, and other course happenings were used to obtain information on the situational context in which students’ talk occurred.

3. Informal interviews with the course instructors4. Students’ retrospective analysis

Data analysis Content analysis was chosen as the main methodology. The

content analysis of data focused on manifestation of EVT and ELVT in students’ posts

The base unit of analysis was a message (post). When a message contained more than one indicator, they were

each counted as separate indicators. And, one instance of a code category in a message was agreed to be sufficient to assign that code

Methodology: Procedures One of the investigators is the co-instructor of the course

under study. To reduce potential bias, she conducted the study with another investigator, a graduate assistant who had no connection with this course or participants in it.

No information is revealed to the course participants regarding the study and its purposes.

The posts of the following people are excluded from the data analysis:

a) The instructors

b) The 2 PhD students who knew about the study and its purposes

The coding scheme

1. Discussions were printed after taking out individual student names and any other personal information.

2. Using EVT and ELVT as a framework, the first presenter made a list of coding categories which was used as the preliminary coding criteria.

3. Both investigators independently coded the messages from the first module based on this preliminary criteria and then they met to reach a consensus on the coding scheme.

The coding protocol

For each module, the coding protocol was conducted in two stages:

Stage 1: The investigators coded the messages independently by using the coding scheme/instrument that they have agreed on using.

Stage 2: The investigators met as a second time to discuss their findings to establish the extent of consensus on the use of the coding instrument (Cohen’s kappa is also used)

Inter-rater Reliability (Cohen’s kappa)

Cohen's kappa (as described by Anderson et al, 2001) is a chance-corrected measure of inter-rater reliability that assumes two raters, n cases, and m mutually exclusive and exhaustive nominal categories. The formula for calculating kappa is:

k = (Fo - Fc) / (N - Fc) where:

N = the total number of judgments made by each coderFo = the number of judgments on which the coders agreeFc = the number of judgments for which there is no agreement.

Inter-rater Reliability

Inter-rater reliability scores were as follows:

After Module 1: 0.73

After Module 3: 0.87

By convention, a Kappa> .70 is acceptable inter-rater reliability, but this depends highly on the researcher’s purpose.

Another thumb of rule is that K= 0.40 to 0.59 is moderate inter-rater reliability; 0.60 to 0.79 is substantial, and 0.80 is outstanding.

(Landis & Kock, 1977)

1) What are the nature and characteristics of talk generated by learners in online discussions?

Results

Total TalkTotal Talk

0

50

100

150

200

250

300

1 2 3

Module

Am

ou

nt

of

Tal

k (t

ota

l po

sts)

Total Talk

Amount of talk: Indicator Share

Amount of talk by Indicator

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3

Modules

Pro

po

rtio

n o

f T

alk

rep

mis

unin

exp

jda

ja

asp

aa

af

imp

expl

inf

anl

intp

ref

he

crt

arg

invt

epl

Results

2) What are the indicators of educationally valuable talk in online discussion threads and how often do

students engage in such conversations?

Proportion of Talk: EVT and ELVT

Proportion of Talk: EVT and ELVT

0

10

20

30

40

50

60

70

80

90

1 2 3

Modules

Prop

ortio

n of

talk

EVT

ELVT

Some numbers..Percentage of talk in Modules: A comparison

Module1 Module 2 Module 3 Total

EVT 80.5 81.4 83 81.4

ELVT 19.5 18.5 17 18.5

What factors…..???What factors seemed to facilitate the production of educationally valuable talk in online discussions?

1. Class norms generated by students

On the first week of class, the students were asked to generate ideas to define the class rules for this course. These norms the students came up with presented themselves in all modules as a reminder of class expectations

A synthesis of class norms: Discussions should be well-organized, concise, relevant, and meaningful. Posts should contain interesting ideas and questions that will stimulate the

discussion and move it forward. In addition to saying “I agree”, participants should accurately state why

and give some facts or personal accounts to back it up. The atmosphere of the discussion sessions should be supportive and

trustworthy with posts conforming to proper etiquette. Posts that do not agree with the general consensus should be welcomed.

Posts should be made in a timely manner. Participants should not wait until the very end of an assignment to post their responses.

Postings that stem from participants’ diverse backgrounds are welcomed, but they should not rely heavily on personal opinions and experiences.

Ideas and experiences should be tied to the course readings whenever possible.

When a participant attaches an outside reading to the discussion, he/she should give a brief overview of it.

Factors continued…

2. Grice’s Maxims:

Starting from Module # 1, students were asked to rate their posts according to Grice's Cooperative Principle of Conversation (Grice, 1975.) (1 = my posting reflects this maxim 100%; 4 = I could do much better when it comes to this maxim)

Quantity: make your contribution as informative as is required, but not more, or less, than is required.

Quality: do not say that which you believe to be false or for which you lack evidence.

Relation: be relevant. Manner: avoid ambiguity and obscurity, be clear, brief and orderly.

Students’ perceptions…What are students’ perceptions of the quality of talk that occurred in whole-class discussions?

Excerpts from students’ retrospective analysis:“The discussions are enlightening and I often walk away from my computer thinking about the ideas being introduced. […]My posts need to be much better developed and clearer.”

“I am usually very careful about writing my posts, with regards to both the content and the manner, so I would have to say that I am quite satisfied with the quality of my posts. For the most part, I am quite happy about others’ posts as well”

Students’ perceptions continued…

“In general, the posts in this course have been better than what I've seen in some of the other courses. There is less of "fluff 'n stuff" or essentially vacuous replies posted for requirement's sake only”

Student perceptions

How did the students feel about using Grice’s maxims and establishing class norms on their own?

“The norms and Grice's maxims compelled me to expand on other postings, reflect on my own experience, connect to course content, explore possibilities, or challenge ideas. I think that class norms and criteria for postings are essential components to quality educational discussions. I think that our class has done a great job creating quality postings. As I was looking back on our past conversations I found many examples of EVT. Personally, I learned a great deal from my classmates due to the consistent quality of postings.”

“Overall, we kept this online discussion very interactive, almost no discussion went without several responses and most responses were relevant and well thought-out because there were guidelines that kept us from answering just yes or no. Grice’s maxims are good guidelines for the online course discussion”

What was the instructors’ perspective?

The instructors found the participation in the course to be well above average and were pleased with the frequent and quality contributions students made to the discussions.

Teaching Implications

1) Have students establish class norms on online discussions

2) Have them use Grice’s maxims to evaluate their own posts as well as those of their peers

Teaching Implications: Suggested activity:

In Module 6, the students were introduced to these two constructs: educationally valuable talk (EVT) & educationally less valuable talk (ELVT). Once they read and discussed the conceptual underpinnings for each talk type, they were asked to select and analyze a conversation/discussion thread of their choice that exemplify interaction which leads to learning and broadening understanding (EVT).

Teaching Implications/ Suggested Activity

To conduct this assignment, the students were given the following options for their selection:

1. Select sample exchanges from any past module of this course

2. Select sample exchanges from another online course they were taking.

3. Select sample exchanges from Internet sites where some form of tutoring or instruction is the objective (the subject matter is irrelevant so you might find conversations about fixing a vacuum cleaner or analyzing a Shakespearian sonnet etc)

4. Select sample exchanges from another source they may have (check with the instructors first)

Student reactions to this assignment:

“I found this assignment [conversation analysis] to be rewarding and worthwhile. It may be a good starter to an on-line course. What do you think?”

Next steps: Suggestions for future research

This research can be further expanded into a treatment-control group study by comparing discussions in modules with treatment (Class-norms and Grice’s maxims) with the ones without it.

References Anderson, T., Rourke, L., Garrison, R., & Archer, W. (2001). Assessing

Teaching Presence in a Computer Conferencing Context. Journal of Asynchronous Learning Networks, 5 (2), 2-17.

Ebel, Robert L. (1951). Estimation of the reliability of ratings. Psychometrika 16: 407-424.

Garrison, R., Anderson, T., & Archer, W. (2000). Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education. The Internet and Higher Education, 2 (2-3), 87-105.

Gilbert, P. K. & Dabbagh, N. (2005). How to structure online discussions for meaningful discourse: a case study. British Journal of Educational Technology, 36 (1), 5-18.

Hara, N., Bonk, C., Angel, C. (2000). Content analysis of online discussion in an applied educational psychology course. Instructional Science, 28, 115-152.

Jeong, C. A. (2003). The Sequential Analysis of Group Interaction and Critical Thinking in Online. American Journal of Distance Education, 17(1), 25-43.

References continued… Kumpulainen, K. (1996). The nature of peer interaction in the social

context created by the use of word processors. Learning and Instruction, 6(3), 243-261.

Landis, J. R., Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33:159-174. This article sets cut-offs for Cohen's Kappa.

Larreamendy-Joerns, J. & Leinhardt, G. (2006). Going the Distance With Online Education. Review of Educational Research,76 (4), 567-605.

Mercer, N. (1994). The quality of talk in children’s joint activity at the computer. Journal of Computer Assisted Learning, 10, 24-32.

Vygotsky, L.S. (1962). Thought and language. Cambridge, MA: MIT Press. (Original work published 1934)

Wickersham, L.E., & Dooley, K.E. (2006). A content analysis of critical thinking skills as an indicator of quality of online discussion in virtual learning communities. Quarterly Review of Distance Education, 7(2),185-193.

Questions? Comments?

Sedef: usedef@yahoo.comRuchi: mruchi@gmail.com

Educational Theory and PracticeUniversity at Albany

State University of New York