ieee transactions on professional ...bieber/pub/shb08_ieee_pc.pdfieee transactions on professional...

IEEE TRANSACTIONS ON PROFESSIONAL COMMUNICATION, VOL. 51, NO. 1, MARCH 2008 63

Learning Strategies in Online Collaborative Examinations—JIA SHEN, STARR ROXANNE HILTZ, AND MICHAEL BIEBER

Abstract—New forms of computer-mediated, online learning can benefit from new forms of assessment that fit themedium and the pedagogical style of the online environment. This paper investigates students’ learning styles andlearning strategies in taking online collaborative exams. Applying constructivist and collaborative learning theories,the collaborative examination features students’ active participation in various phases of the exam process throughsmall group activities online. Students’ learning strategies, including deep learning and collaborative learning, areinvestigated using a 1 � 3 field quasi-experiment to compare the team-based collaborative online exam with thetraditional in-class exam and with the participatory exam, where students participate in the online exam processesindividually. Data analysis using results from 485 students indicates that collaborative examinations significantlyreduced surface learning in exam study, enhanced interactions and the sense of an online learning community, andincreased perceived learning. The results also suggest learning predispositions were significantly correlated withexam study strategies, and provide indications of their effects on learning strategies.

Index Terms—Collaborative examinations, collaborative learning, deep learning, online learning, peer assessment,surface learning.

With over three million university students takingonline courses in the US alone in 2005, and withonline education becoming part of the long-termstrategy for many schools [1], enhancements to theonline learning process take on practical as wellas theoretical importance. Most current publishedstudies focus on learning outcomes only, such assatisfaction and learning. Prominent researcherssuch as Alavi and Leidner, however, have called formore depth and breadth in the empirical studies inonline learning [2], and Hiltz and Goldman haveproposed comprehensive input-process-outputmodels to guide research on the interaction of themany variables that influence the effectivenessof asynchronous learning networks (ALN) [3].Benbunan-Fich and Hiltz, for example, examinedmediators of the effectiveness of online courses,and found that motivation and active participation,among other variables, significantly mediatedlearning results [4]. Thus, there is an increasingneed to examine other variables such as students’learning predispositions and learning strategies inorder to provide a better understanding of onlinelearning.

Manuscript received February 15, 2006;revised August 30, 2006.J. Shen is with the Department of Computer and InformationSystems, College of Business Administration, Rider University,Lawrenceville, NJ 08648-3099 USA (email: [email protected]).S. R. Hiltz and M. Bieber are with the Information SystemsDepartment, New Jersey Institute of Technology, UniversityHeights, Newark, NJ 07102-1982 USA (e-mail: [email protected];[email protected]).Color versions of one or more of the figures in this paper areavailable online at http://ieeexplore.ieee.org.

IEEE 10.1109/TPC.2007.2000053

Our research aims at a deep investigation ofstudents’ learning dispositions, strategies, andoutcomes in the context of online exams. Studentsnot only should be evaluated by, but shouldalso learn through assessment. The traditionalinstructor-controlled exam reflects the objectivistlearning model [5], which regards learning as theuncritical transfer of objective knowledge frominstructor to students. New assessment approachesbased on constructivism theory [6] propose thatlearners actually construct knowledge, guided bytheir own ideas and interests. Instead of presentinglearners with information perhaps unrelatedto their own ways of thinking, constructivistapproaches emphasize the role prior knowledgeplays in building new knowledge [7]. For example,learner-centered assessment [8] and classroomassessment [9] shift attention from instructorsand teaching to students and learning throughassessment.

COLLABORATIVE LEARNING, an offspring of theconstructivist model, is a learner-centered andteam-oriented approach that assumes learningemerges as learners interact with each other [10].Studies have shown the superiority of collaborativelearning in both face-to-face settings and onlineeducation using group support systems (GSS) [3],[11].

Despite the collaborative learning paradigmthat dominates ALN, only a few studies havebeen conducted to incorporate students’ activeparticipation and collaboration into the assessmentprocess online. With the use of GSS, students’participation and collaboration have beenintegrated into specific phases of collaborative

0361-1434/$25.00 © 2008 IEEE

Authorized licensed use limited to: New Jersey Institute of Technology. Downloaded on December 7, 2008 at 15:17 from IEEE Xplore. Restrictions apply.

64 IEEE TRANSACTIONS ON PROFESSIONAL COMMUNICATION, VOL. 51, NO. 1, MARCH 2008

assessment, such as collaborative development ofthe grading scheme [12], question composition [13],collaborative question answering [14]–[16], andpeer- and self-grading [17]–[19].

Incorporating constructivism and collaborativelearning theories, the online exams in thisstudy feature students’ active participation invarious phases of the exam process throughsmall group activities. Students designed examquestions, answered questions designed by theirpeers, and graded answers to the questions theyauthored. In the online participatory exam,students participate in the online exam processindividually. When overall positive results of theonline participatory exam were revealed in previousstudies [20], [21], the online collaborative examwas designed and examined in this study in thehope of further improving students’ learning. Theonline collaborative exam features students workingin small groups in the question design and gradingphases, but individually in the answering phase.Although the most common form of collaborativetesting refers to students providing a common setof exam answers [16], we believe question designand grading phases are important opportunities forstudents to learn, which can be further enhancedwith group work. The answering phase remainsindividual to allow the exam to assess an individualstudent’s learning.

A field experiment was conducted to evaluate theonline exams compared with the traditional in-classexam. This paper examines students’ learningstyles and their adoption of learning strategies intaking online exams, in particular deep learningand collaborative learning, as well as studentlearning outcomes.

THEORETICAL CONTEXT

In contemporary educational theory, one influentialgroup of researchers has identified learners’approaches to be either surface level or deep level[22]–[25]. A deep learning approach is consistentwith a search for knowledge and understanding,whereas a surface learner is concerned only withpassing exams by memorizing facts. Learningapproaches have a motivation and a strategyelement, which are related [22], [24]. Studentsattempt to understand a topic (deeply) if it is of realinterest to them or if they can see its relevance totheir current or future professional roles; whereas asurface approach is associated with limited interestin a task or an extrinsic motivation. While studentsnormally have a predisposition to either deep

or surface approaches in general, this preferredapproach can be modified by the teaching contextor learning environment for individual courses orparticular learning tasks. Measures of approachesto learning can, therefore, be related to aspectsof the teaching and learning environment. Inassessment, prior studies have found that amongother variables, students dynamically form examstudy strategies according to different assessmentmethods, while their predisposition towards asurface or deep learning approach also impactstheir exam studying strategies [26], [27].

Collaborative learning has been identified asessential for creating an effective online learningenvironment [3]. Collaborative learning involvessocial (interpersonal) processes by which asmall group of students work together (i.e.,collaborate and work as a team) to complete anacademic problem-solving task designed to promotelearning. Many theories that are distinctivelysocial have been advanced, such as situatedlearning [28], distributed cognition [29], andlearning communities [30]. Applying these sociallearning theories to the online environment, onlineresearchers have studied the capacity of theonline environment to support social activities andinteraction [31]–[33]. They found that the extent ofstudents’ perceptions of their learning experiencesonline as actively involving, socially rich, andcollaborative can improve their learning outcomes.

RESEARCH MODEL AND HYPOTHESES

Fig. 1 shows the research model that will bediscussed in this paper. This is part of a largerresearch framework [34] developed based on theOnline Interaction Learning Model [35].

Fig. 1. Research model.


SHEN et al.: LEARNING STRATEGIES IN ONLINE COLLABORATIVE EXAMINATIONS 65

Exam Modes The independent variable consistsof three exam modes: the traditional exam, theparticipatory exam, and the collaborative exam.Fig. 2 compares and contrasts the three examprocesses as they were conducted in this study.

In both the participatory exam and the collaborativeexam, the process of involving students in questiondesign and grading is conducted online. Whilestudents in the participatory exam perform thesesteps individually, students in the collaborativeexam first participate individually (i.e., each designsquestions and grades), and then the group (of threeto five students) as a whole discusses, improves,and reaches consensus on the questions andgrades. Typically, essay type questions are designedand answered by students. As discussed above,students answer exam questions individually in allthree exam modes to maintain the exam’s ability toprovide accountability of individuals’ learning.

Learning Predispositions Many studies inweb-based learning have indicated that studentcharacteristics [36], among other contextual factors[37], are likely to influence learning. As part ofstudent characteristics, students’ predispositionson learning are expected to moderate the strategiesthey adopt. In particular, the following two aspectsof students’ learning predispositions are examined

Fig. 2. Comparison of the three exam modes.

in this study: (1) deep learning, and (2) collaborativelearning.

As discussed previously, DEEP LEARNING is astudent’s approach to learning which is consistentwith a search for knowledge and understanding;whereas a surface approach is concerned only withpassing exams by memorizing facts. Studies haveshown that students normally have a predispositionto either deep or surface learning. In this study,students’ deep learning predisposition wasmeasured before the exam through questions abouttheir approaches to learning the course materials.

Collaborative learning predisposition is developedin this study to describe students’ orientationsin using collaborative learning strategies andlearning from other students. A COLLABORATIVE

LEARNING APPROACH is consistent with learningfrom interacting with other students and enjoyingworking with other students. As with deep learning,students’ collaborative learning predispositionswere measured before the exam.

Exam Study Strategies The exam study processincludes the learning activities that occur inpreparing for and participating in an exam. Inthe traditional exam, the exam study processincludes preparation before the exam, learningwhile answering questions during an exam, andreviewing the instructor’s feedback after the exam.In the participatory and collaborative exams, theexam study process includes not only the activitiesassociated with the traditional exam, but alsoadditional activities during the question designphase, the grading phase, and review of others’postings in the ALN environment.

While deep learning is associated with students’general learning approach to the course, deepexam study is associated with the specific learningstrategies students adopt in exam study. DEEP

EXAM STUDY is defined in this study as theextent of the student’s search for knowledge andunderstanding during the exam studying process.Low adoption of deep exam study (i.e., surfaceexam study) is associated with passing examsby memorizing facts, while high adoption of deepexam study is associated with seeking knowledgeand understanding. Deep exam study is reflectedin a number of strategies the student may usein exam studying, such as adopting professionalperspectives to understand course materials, andspending extra time to obtain more informationon interesting topics [26], [27]. In this study, deepexam study was measured after the exam through



questions about the specific approaches studentsadopted.

Similarly, social engagement is associated withthe specific learning strategies students adoptin exam study. SOCIAL ENGAGEMENT is defined asthe extent of a student’s active involvement inlearning from other students through the examstudy process, and the sense of forming a learningcommunity. High adoption of social engagement inexam studying is using strategies such as gettingto know other students better through the examprocess, acknowledging learning from others, andforming a learning community.

Student Learning Student learning outcomesare examined from two perspectives in this study:student perceived learning, and actual learning.Perceived learning was measured using 12 itemsin the post-exam surveys. Among the 12 items,10 were designed based on Bloom’s taxonomy [38]to measure perceived learning from lower levels,such as understanding the materials, to higherlevels, such as comprehension, understanding,application, analysis, synthesizing, and evaluation.The other two items were adapted from thecollaborative exam questionnaire used in previousstudies conducted by the authors [21], [39].

Actual learning was measured using exam gradesin final exams whenever possible, where theexperiment was conducted during the midterm anda conventional exam was given as the final exam.In those courses where there are no final examsbut final projects instead, the final project gradeswere used as a measure of actual learning.

Research Questions and Hypotheses Sincethe collaborative exam is the main innovation ofthis research and thus the focus of the paper, thefollowing hypotheses compare the collaborativewith the participatory exam, and the collaborativewith the traditional exam. For comparison of theparticipatory and traditional exam, see otherpublications [20].

Based on constructivism and collaborative learningtheories, actively engaging students in the entireexam process as online collaborative groups shouldenhance learning and the sense of a learningcommunity. Thus it was hypothesized that studentsin the collaborative exam would adopt higherlevels of deep exam study (H1) and higher levelsof social engagement (H2) than students in theother two exam modes. In addition, students in thecollaborative exam would achieve better learning

outcomes, including perceived learning (H7) andactual learning as measured using final exam orproject grades (H8). As students’ predispositionstowards a surface or a deep learning approachalso affect their exam studying strategy, it ishypothesized that students’ predispositions on deeplearning or collaborative learning will affect theiradoption of deep learning (H3) or social engagement(H4) in exam studying. The interaction effects ofexam modes and learning predispositions (H5, H6),and exam modes and exam study strategies (H9,H10) are also investigated. While previous studieson online examinations indicate overall positiveresults, student interviews suggest that theremay be a wider range of exam strategies adoptedin taking the online exam compared with thetraditional exam [20], [21], [40], [41]. The researchquestions and hypotheses are listed below, withcorresponding hypothesis numbers shown in theresearch model in Fig. 1.

RQ1: Do students adopt different exam studystrategies in the collaborative exam mode thanthe other two exam modes?

H1: Students taking the collaborativeexamination will have higher adoption ofdeep learning in the exam studying processthan students taking the traditional exam(H1.1) or the participatory exam (H1.2).H2: Students taking the collaborativeexamination will have higher socialengagement in the exam studying processthan students taking the traditional exam(H2.1) or the participatory exam (H2.2).

RQ2: Do students’ exam study strategiescorrelate to their learning predispositions?

H3: Students’ adoption of deep learning inthe exam studying process will be positivelyrelated to their predispositions to deeplearning.H4: Students’ level of social engagementin the exam studying process will bepositively related to their predispositions tocollaborative learning.

RQ3: Do the relationships between students’learning predispositions and their exam studystrategies differ in the collaborative exam andthe traditional exam?

H5: The difference in the adoption of deepexam study between students who are theleast deep-oriented and those who are themost deep-oriented will be larger in thecollaborative exam than in the traditionalexam.H6: The difference in the adoption ofsocial engagement in exam studying



between students who are the leastcollaborative-oriented and the mostcollaborative-oriented will be larger in thecollaborative exam than in the traditionalexam.

RQ4: Do students achieve better learning inthe collaborative exam than the other two exammodes?

H7: Students taking the collaborativeexamination will have higher perceptionsof learning than students taking thetraditional exam (H7.1) or the participatoryexam (H7.2).H8: Students taking the collaborativeexamination will achieve higher gradesin the final exam/projects than studentstaking the traditional exam (H8.1) or theparticipatory exam (H8.2).

RQ5: Do the relationships between students’exam study strategies and perceived learningdiffer in the collaborative exam and thetraditional exam?

H9: The difference in perceived learningbetween students who adopt the leastdeep learning and the most deep learningin exam study will be higher in thecollaborative exam than in the traditionalexam.H10: The difference in perceived learningbetween students who are most sociallyengaged and those who are least engagedin exam study will be higher in thecollaborative exam than in the traditionalexam.

RESEARCH DESIGN

Subjects To investigate the research questions,a 1 3 quasi-experimental nonequivalent groupsdesign with pre- and post-measures was employed.This is appropriate given the situation of the studywhere the participants were aggregated into groups(i.e., course sections) for educational purposes, andwhere random assignment was not possible. Thestudy was conducted in the spring, summer, andfall semesters of 2004 at a US university. A totalof 22 course sections at the undergraduate andgraduate levels participated in the experiment. Abalancing technique was used to assign sectionswith similar characteristics to different exam modesin order to counterbalance pre-existing differencesamong students.

Table I shows the number of subjects in eachcondition and the response ratio. Extra credit

towards the exam grade was provided as incentiveto complete the surveys.

Procedures Detailed exam instructions includingquestion design and answer grading criteria wereprovided to students before the exam. The entireonline exam took about two and a half weeks,including about three days for each of the mainphases: question design, answering, and grading,and a few days for the instructor’s activities, suchas question review and assignment. The onlineexams were conducted using WebBoard and WebCT,and the process was anonymous using pen names.Students in the traditional exam condition took theexam in class, or through proctors in remote sites indistance-learning courses. The instructor providedthe questions and graded students’ answers. Withthe instructors’ permissions, students were allowedto use certain course materials in the traditionalexams in this study, such as course notes, books,or other references, in order to make the examconditions comparable in this regard. Gradingcriteria and guidelines equivalent to those of theonline exams were provided in advance of the examto students. Questions of similar types (e.g., essay)were asked in matching course sections in differentexam modes whenever possible (e.g., because of amatching section in the same semester, questionsdesigned by students in the online section werere-used for the traditional exam in the samecourse.)

Students in all three conditions filled out thepre-exam questionnaire regarding demographicsand learning predispositions. Shortly followingthe exam, students were asked to complete thepost-exam questionnaires, which have many of thesame questions for the three conditions to allowcomparison. The questionnaires adapted itemsthat have been validated [26], [42], and items from

TABLE I1 � 3 FIELD EXPERIMENT AND NUMBER OF SUBJECTS



previous studies on the online exam [21], [39] withrevisions and additions. The questions used areshown in the results section of this paper.

Collaboration Support Systems Used inthe Study WebBoard and WebCT were theasynchronous conferencing systems used tofacilitate the participatory and the collaborativeexams in this study. Both systems have“conferences,” which are threaded discussionareas where students can post, read, and replyto comments of others. One main conference wascreated for the main exam activities; administrationinformation and student feedback were posted ingeneral administration and feedback conferences.

FINDINGS

The raw data were analyzed using SPSS. Thissection first presents the factor analysis andnormalization test results, followed by item-levelresponse analysis, and finally hypothesis testingoutcomes.

Factor Analysis, Reliability of Scales, andAddition of Hypotheses Through principlecomponent factor analysis with PROMAX rotation,the following three factors were extracted from thepre-exam questionnaire:(1) Deep learning: 5 items, Chronbach's alpha

.(2) Surface learning: 5 items, Chronbach's alpha

.(3) Collaborative learning: 7 items,

Chronbach's alpha .

Items measuring the deep learning predispositionconstruct split into two factors. The positiveitems converged into one factor (deep learning),and the negative items converged into anotherfactor (surface learning). The items measuringcollaborative learning converged into one factor asexpected.

Using a similar procedure, the following factorswere extracted from the post-exam survey results.(1) Deep exam study: 6 items,

Chronbach's alpha .(2) Surface exam study: 2 items,

Chronbach's alpha .(3) Social engagement: 4 items,

Chronbach's alpha .(4) Perceived learning: 12 items,

Chronbach's alpha .

The items from the post-survey measuring thedeep exam study construct split into two factors.

The positive items converged into one factor(termed deep exam study), and the negative itemsconverged into another factor (termed surface examstudy). The items measuring social engagementand perceived learning converged into factors asexpected. These items are shown in Tables II andIII.

The reliability analysis of the factors suggeststhe factors are highly reliable, with Chronbach’salpha all above the 0.7 level. The mean scoreswere calculated for all the factors, and the datawere tested for normal distribution. Among theseven factors, four were normally distributed(deep learning, collaborative learning, deep examstudy, and social engagement). Through datatransformation, surface learning and perceivedlearning were successfully normalized. In addition,final exam/project grades were also tested fornormal distribution, and they were normallydistributed. In the next section, parametric dataanalysis was performed on factors that werenormally distributed, and nonparametric dataanalysis was performed on surface exam study,which was not normally distributed.

Given deep learning splits into two factors (deeplearning and surface learning) through the factoranalysis above, H1 and H3 are each revised withan additional hypothesis addressing the newfactor. Although deep exam study also splits intotwo factors, surface exam study factor is notnormalized. Given the limitations of nonparametrictests, H5 and H9 are not revised, and only deepexam study is used in testing those hypotheses.The additional hypotheses are:

H1a: Students taking the collaborativeexamination will have lower adoption of surfacelearning in the exam studying process thanstudents taking the traditional exam (H1a.1) orthe participatory exam (H1a.2).H3a: Students’ adoption of surface learning inthe exam studying process will be positivelyrelated to their predispositions in surfacelearning.

Before presenting the main results, it is necessaryto point out that analysis using the pre-examsurvey data shows no significant difference amongstudents in the three exam modes before theexam. The one-way ANOVA comparing the learningdispositions revealed that there is no significantpre-existing differences of students among thethree exam modes at the 0.05 level, includingcollaborative learning . ,deep learning , , and surface



learning , . Students’demographic data were analyzed using Chi-Squaretests, and no significant differences existed amongthe three conditions, including gender ,

, work experience , ,and previous online exam experiences ,

. This suggests the matching techniqueused in assigning courses to different exam modeswas quite successful in preventing pre-existingdifferences among the three conditions.

Univariate Analysis This section presents keydescriptive results from the post-exam survey onthe individual question item level.

Deep Exam Study: Table II provides the univariateanalysis results of the six items of the deep examstudy factor, including means and standarddeviations.

In five out of six items, students in the participatoryor the collaborative exam reported adopting adeeper approach to exam study compared with

TABLE IIITEMS MEASURING DEEP EXAM STUDY

� : Traditional Exam; � : Participatory Exam;�: Collaborative Exam.

those in the traditional exam. For example,students in the participatory exam reported thehighest adoption of deep exam study in termsof finding the topics interesting, being willing tospend extra time to study, adopting professionalroles in understanding materials (e.g., puttingoneself in the position of a system analyst anddesigner), spending extra time to find out additionalinformation, and relating course materials to othersubject areas. Students in the collaborative examreported the highest level of achieving satisfactionby researching topics to form their own conclusionsin exam study. The only item that had the highestscore in the traditional exam is testing oneselfon important topics until one understands themcompletely. The results suggest that students in theparticipatory and the collaborative exam adopteda deeper approach to exam study, includingtaking professional and academic perspectives inunderstanding materials, while students in thetraditional exam tested themselves on importanttopics before the exams.

Surface Exam Study: Table III shows the resultsof the two items of the surface exam study factor.Note the negative items were converted back sothat the item statement matches the mean (i.e., thehigher the number, the more adoption of surfaceexam study).

For both items, students in the traditional examadopted noticeably higher levels of surface studyin their processing of information compared withthose in the online exams, including trying toremember answers to likely questions as the bestway to pass the exam ( , , ),and learning by rote ( , , ).Results suggest that students in the traditional

TABLE IIIITEMS MEASURING SURFACE EXAM STUDY




exam had the highest adoption of surface examstudy strategy in their processing of information.

Social Engagement: Table IV provides theresults of the four items measuring the socialengagement factor. For all four items, studentsin the collaborative online exam noticeably hadthe highest level of social engagement, includinggetting to know other students through the examprocess ( , , ), perceivingthe exam as a group process ( , ,

), forming a learning community throughexam study ( , , ), andenhancing understanding of course materials byinteracting with other students ( , ,

). Results suggest that students in thecollaborative exam had the highest adoption ofsocial engagement in exam study. Interestingly, it isalso noticeable that the level of social engagement inthe participatory exam was lower than or as low asthe traditional exam on most items. This suggeststhat without the small group activities, the level ofsocial engagement in the online environment is aslow as or even lower than the traditional settings.

Perceived Learning: Table V provides the results forthe 12 items of the perceived learning construct.

In 10 out of the 12 items, students in thecollaborative exam reported the highest level of

TABLE IVITEMS MEASURING SOCIAL ENGAGEMENT


learning among the three exams. For the other twoitems, perceptions of students in the participatory

TABLE VITEMS MEASURING PERCEIVED LEARNING

� : Traditional Exam; � : Participatory Exam;�: Collaborative Exam. � reverse scored item.



exam were the highest. For example, students inthe collaborative exam reported the highest level ofimprovement in their knowledge of course concepts,methods and theories, and in their understanding ofcourse materials. They also reported enhancementin skills such as using knowledge in new situations,solving problems, recognizing patterns, makinggeneralizations and predictions, comparing anddiscriminating ideas, and making judgments andassessments of the quality of ideas and arguments.In addition, students in the collaborative examreported the highest level of learning in readingexam questions posted online and in answeringexam questions. Students in the participatory examreported the highest level of improvement in seeingdifferent course components and organizing themin a meaningful way, and relating knowledge fromdifferent academic areas. In 11 out of the 12 items,students in the traditional exam reported the lowestlevel of perceived learning among the three exams.

Hypothesis Testing To test the hypotheses, themean scores were calculated for all the factors. Tocompare the differences among the three exammodes, one-way ANOVA tests were conducted ondeep exam study and social engagement, whichwere normally distributed. The Kruskal Wallistest, which is the nonparametric equivalent of theone-way ANOVA, was conducted on surface examstudy. The ANOVA results are shown in Table VIand the Kruskal Wallis test results are shown inTable VII, where the higher the mean rank, thehigher the score.

Significant differences were found among the threeexam modes in social engagement andsurface exam study , but not in deep examstudy. Therefore, H1.1 and H1.2 are not supported.

TABLE VIEXAM STUDY STRATEGIES (ANOVA TEST)

� : Traditional Exam; � : Participatory Exam;�: Collaborative Exam

Post-hoc analysis reveals that in terms of socialengagement, students in the collaborative examwere significantly more engaged than studentsin the traditional and the participatoryexams . The effect size eta is weak(0.11), but both H2.1 and H2.2 are supported

. There is no post-hoc analysis for aKruskal Wallis test. Two separate Mann-Whitneytests were conducted to compare the differencesof surface exam study between the collaborativeexam and the traditional exam, and between thecollaborative exam and the participatory exam.Results show that students in the collaborativeexam adopted significantly lower levels of surfaceexam study than students in the traditional exam

. Therefore, H1a.1 is supported ,and H1a.2 is not supported.

To examine the relationship between students’deep/surface learning predispositions and theirdeep/surface exam study strategies, correlationanalysis using Pearson’s was conducted ondeep exam study strategy. The Spearman’s rhotest was conducted on surface exam study. Asshown in Table VIII, learning predispositions weresignificantly correlated with exam study strategies.Therefore, H3 is supported ( , ), andH3a is supported (Spearman's rho , ).

TABLE VIIEXAM STUDY STRATEGIES (KRUSKAL WALLIS TEST)


TABLE VIIIDEEP/SURFACE STUDY AND LEARNING PREDISPOSITIONS

(CORRELATION)

Correlation is significant at the 0.01 level (2-tailed).Spearman’s rho is used instead of Pearson’s � in this

column.



To examine the relationship between students’pre-exam collaborative learning and their socialengagement in exam studying, a correlationanalysis using Pearson’s was conducted. Resultsshow that the two variables were significantlycorrelated ( , ). Therefore, H4 issupported.

To test H5, deep learning predisposition asmeasured by the pre-exam survey was firstcategorized into low, medium, and high. The twoextreme categories were used and the mediumcategory was excluded from further analysis. A 2

2 factorial ANOVA test was then conducted tocompare the two exam modes and the two deeplearning predispositions. Tables IX and X show themeans and the significance test results in factorialANOVA. As shown in Fig. 3, the least deep-orientedstudents had a slightly larger increase in thelevel of deep exam study than those who weremost deep-oriented (i.e., the slope of the diamondline is more steep than the square line). Theinteraction effect is not significant. Therefore, H5is not supported. Though there is no relationshipbetween exam mode and deep exam study, there isa moderately strong positive relationship between

TABLE IXDEEP EXAM STUDY (EXAM MODE�PRE-EXAM DEEP LEARNING)

� : Traditional Exam; �: Collaborative Exam.

TABLE XFACTORIAL ANOVA OF DATA FROM TABLE IX

pre-examination learning disposition and deepexam study, with an effect size of 0.277.

To test H6, collaborative learning predispositionas measured by the pre-exam survey was firstcategorized into low, medium, and high. The twoextreme categories were used, and a 2 2 factorialANOVA test was conducted. Tables XI and XIIshow the means and the significance test resultsof factorial ANOVA tests. There is a moderatelystrong relationship between collaborative learningpredisposition and social engagement, with aneffect size of 0.26. As shown in Fig. 4, the leastcollaboratively oriented students had a greaterincrease in the level of social engagement comparedto those who were most collaboratively oriented (i.e.,the slope of the diamond line is more steep thanthe square line). In addition, the difference in socialengagement between the two types of studentswas smaller in the collaborative exam than thetraditional exam. However, the interaction effect isnot significant. Therefore, H6 is not supported.

Fig. 3. Effects of learning predisposition and exammode on deep exam study.

Table XISOCIAL ENGAGEMENT (EXAM MODE� PRE-EXAM

COLLABORATIVE LEARNING)




To test hypotheses on student learning outcomes,one-way ANOVA tests were conducted on perceivedlearning and final exam/project grades. Asmentioned before, final exam or project gradeswere collected and used as a measurement ofstudents’ actual learning in those courses wherethe experiment was conducted as the midtermexam and the final was a traditional exam. Finalexaminations or projects are comprehensive,covering the entire semester and not just theportion after the midterm; thus they include whatwas learned overall in the course. The ANOVAresults are shown in Table XIII.

Significant differences were found among thethree exam modes in perceived learning ,although the effect size is negligible (0.02). Post-hocanalysis reveals that students in the collaborativeexam reported significantly higher perceptionsof learning than those in the traditional exam

TABLE XIIFACTORIAL ANOVA FOR DATA IN TABLE XI

Fig. 4. Effects of learning predisposition and exammode on social engagement.

. Therefore, H7.1 is supported, and H7.2is not supported.

As shown in Table XIII, students in the collaborativeexam achieved higher final exam/project gradesthan students in the other two exam modes. Usingthe final exam or project scores as the reflectionof actual learning suggests that students in thecollaborative online exam may have indeed learnedthe most from the midterm exam compared withthe other two exams. However, the difference is notsignificant. Therefore, H8 is not supported.

To test H9, deep learning as measured in thepost-exam survey was first categorized into low,medium, and high. The two extreme categorieswere used, and a 2 2 factorial ANOVA test wasconducted. Tables XIV and XV show the meansand the factorial ANOVA significance test results.As shown in Fig. 5, the interaction effect is notsignificant, even though the distance between thetwo lines is slightly smaller in the collaborativeexam than the traditional exam. Therefore, H9 isnot supported. It is notable that the effect size forthe relationship between deep exam study strategyand perceived learning is substantial (0.326).

To test H10, social engagement as measured in thepost-exam survey was first categorized into low,medium, and high. The two extreme categorieswere used and a 2 2 factorial ANOVA test wasconducted. Tables XVI and XVII show the meansand the factorial ANOVA significance test results.There is a moderate effect size for the relationshipbetween social engagement and perceived learning,but no other relationship is significant. Fig. 6shows that the least socially engaged have a drop in

TABLE XIIILEARNING OUTCOMES (ANOVA TEST)

� : Traditional Exam; � : Participatory Exam;�: Collaborative Exam. � transformed scale.� final exam/project grade.



perceived learning in the collaborative exam, whilethe most socially engaged stay almost the samein the two exams, but the interaction effect is notsignificant. Therefore, H10 is not supported.

The results are summarized in Fig. 7, which isthe research model shown in Fig. 1 with theresults added. The numbers beside lines are

TABLE XIVPERCEIVED LEARNING (EXAM MODE � DEEP EXAM STUDY)

TABLE XVFACTORIAL ANOVA FOR DATA IN TABLE XIV

Fig. 5. Effects of deep exam study and exam mode onperceived learning.

correlation coefficient Pearson’s (indicated as )or Spearman’s rho. Significant differences foundamong the exams are highlighted with bold boxoutlines. Note that the two new constructs areadded in the diagram, and the means for surfaceexam study are used in the diagram because thedifferences are significant.

TABLE XVIPERCEIVED LEARNING (EXAM MODE � SOCIAL ENGAGEMENT)


TABLE XVIIFACTORIAL ANOVA FOR DATA IN TABLE XVI

� transformed scale.

Fig. 6. Effects of social engagement and exam modeon perceived learning.



DISCUSSION

This paper examines students’ learningpredispositions, adoption of exam study strategies,and learning outcomes when taking online exams.First, the results show that students’ adoptionof deep or surface exam study strategies weresignificantly associated with their deep or surfacelearning predispositions. More importantly, theparticipatory and collaborative online examsresulted in students adopting significantly lowerlevels of the surface exam study strategy. Studentsin the online exams reported significantly lessuse of memorization and rote learning withoutunderstanding the materials in exam study,compared with those in the traditional exam. Asobserved through the study, the online examsallowed students to prepare for the examsdifferently than for the traditional exam bymotivating them to conduct in-depth research,cover a broader breadth of knowledge, andsynthesize materials.

Second, the level of social engagement in examstudy was not only significantly associated withstudents’ predispositions in collaborative learning,but, more importantly, was significantly higher inthe collaborative exam than both the participatoryexam and the traditional exam. Students inthe collaborative exam reported perceiving theexam process as a group experience, forming asense of a learning community, and enhancingtheir understanding of course materials throughinteractions with other students. Strikingly,students’ level of social engagement in theparticipatory online exam was significantly lowerthan that in the collaborative exam, and was at thesame level as the traditional exam. This suggests

Fig. 7. Research model with results.

that without incorporating the small group activitiesinto the online process, the level of engagement inlearning from others and the sense of a learningcommunity are as low as in the traditional settings.

Significant differences were also found instudent learning outcomes. Students in thecollaborative exam reported significantly higherlevels of perceived learning than those in thetraditional exam. Students reported improvementin their understanding of course materials, andenhancement in skills such as using knowledgein new situations, solving problems, recognizingpatterns, comparing ideas, and making judgments.As a measurement of actual learning, finalexam/project grades were the highest amongstudents who took the collaborative exam ratherthan the other two exam modes. Although theresults were not significant, this suggests thatstudents may indeed learn the most through thecollaborative exam. Further studies are neededto investigate this, especially those based onlongitudinal data.

While neither the interaction effects of the learningpredispositions and exam modes on exam studystrategies nor those of exam study strategiesand exam modes on learning outcomes werestatistically significant, they do provide someinteresting indications that there may be aneffect that is masked by sources of varianceintroduced by the field study design, in whichmany variables could not be controlled. Theresults using deep exam study indicate that thecollaborative exam seemed to be the most effectivein motivating the least deep-oriented students toadopt a higher level of deep exam study strategy.The collaborative exam also motivated the leastcollaborative-oriented students to adopt a higherlevel of social engagement than those who weremost collaborative-oriented. In addition, there was atendency towards a negative synergistic interactioneffect, where the students who were least sociallyengaged in exam studying perceived less learningin the collaborative exam. The results suggest thatthe collaborative exam may be most effective inpromoting deep learning and collaborative learningamong those students who need to be motivatedmost, and this is important because students needto be actively engaged in group activities in order toachieve the most learning. This possibility shouldbe pursued in a future study through refining examand experimental procedures to focus on variablesrelated to student motivation.



CONCLUSIONS

This study contributes the first major investigationrelating students’ deep study and collaborativelearning dispositions with their learning strategiesfor online collaborative exams. It also identifiesand provides instruments for measuring learningstrategies towards exam studying and learningoutcomes.

Short of adopting the collaborative exam, whatcould instructors take away from this studyto improve student study habits and overalllearning? Even if students are predisposed tosurface learning, they can be encouraged throughstructural changes involving collaboration andsocial engagement to adopt a deeper learningapproach. This corroborates research by Tang [26],[27]. Incorporating peer review, as the collaborativeexam does, leads to better preparedness. Thiscorroborates research showing that when studentsknow peers will read their assignments, it motivateslearning [43], [44], which of course is our overallgoal.

From a practical point of view, this researchproposes an innovative form of online assessmentthat promotes teamwork skills and shifts theinstructors’ roles. Many university instructorsfeel the need to produce graduates with the skillsdesired by future employers (e.g., teamwork skills).Our study offers one way to incorporate teamworkinto one of the most important learning activities:assessment in the online environment. Thisstudy shows that students’ active engagement inteamwork is a significant mediator of students’learning outcomes. From the instructor’sperspective, his or her role shifts from the designerand the grader to the facilitator of the exam. Inaddition, students’ participation in the questiondesign and grading phases make it possible toadopt question formats usually requiring moretime in grading, such as essay questions orprogram coding questions. This reduces theconstraints on the question types that can beused in collaborative testing without significantlyimpacting an instructor’s workload. As the resultsshow, students’ perceived learning is significantlyhigher when they are involved in the whole examprocess than in the traditional exams.

More generally, our results support the premisethat when computer technology mediates andchanges the form of interaction among participantsin a process by moving it from physical spaceto cyberspace, then it can be beneficial to think

about how all aspects of the process may bebest adapted to fit the cyberspace environment.The various forms of “groupware” platforms thatsupport threaded discussions and collaborationabout the content of a course can also be adaptedto support collaboration in the assessment process.The assessment process can become an integralpart of the learning in the course, rather than aseparate procedure completely divorced from it.

ACKNOWLEDGMENT

This work was supported in part by the AlfredP. Sloan Foundation; the United Parcel ServiceFoundation; the New Jersey Commission onScience and Technology through a grant to theNew Jersey Center for Pervasive InformationTechnology; the National Science Foundationunder Grant IIS-0135531, Grant DUE-0226075,Grant DUE-0434581, and Grant DUE-0434998;and the Institute for Museum and Library Servicesunder Grant LG-02-04-0002-04. The authorsgratefully acknowledge Dr. I. Im, Dr. K. Passerini,Dr. J. Scher, and Dr. D. Leidner for their valuableinput and support for this research. Thanksto all the faculty members and students whoparticipated in the experiments of this study. Theopinions suggested are those of the authors and notnecessarily those of the sponsors of this research.

REFERENCES

[1] E. Allen and J. Seaman, Making the Grade: OnlineEducation in the United States, 2006. Needham,MA: Sloan Consortium, 2006.

[2] M. Alavi and D. E. Leidner, “Researchcommentary: Technology-mediated learning—Acall for greater depth and breadth of research,”Inform. Syst. Res., vol. 12, no. 1, pp. 1–10, 2001.

[3] S. R. Hiltz and R. Goldman, Learning TogetherOnline: Research on Asynchronous LearningNetworks. Mahwah, NJ: Lawrence Erlbaum,2004.

[4] R. Benbunan-Fich and S. R. Hiltz, “Mediators ofthe effectiveness of online courses,” IEEE Trans.Prof. Commun., vol. 46, no. 4, pp. 298–312, Dec.,2003.

[5] D. E. Leidner and S. L. Jarvenpaa, “The use ofinformation technology to enhance managementschool education: A theoretical view,” MIS Quart.,vol. 19, no. 3, pp. 265–291, 1995.

[6] L. S. Vygotsky, Mind in Society: The Developmentof Higher Psychological Processes. Cambridge,MA: Harvard Univ. Press, 1978.

[7] J. Hawkins and R. Pea, “Tools for bridgingeveryday and scientific thinking,” J. Res. Sci.Teaching, vol. 24, no. 4, pp. 291–307, 1987.



[8] M. E. Huba and J. E. Freed, Learner-CenteredAssessment on College Campuses: Shiftingthe Focus From Teaching to Learning, 1sted. Needham Heights, MA: Pearson Allyn &Bacon, 1999.

[9] T. A. Angelo and K. P. Cross, AssessmentTechniques: A Handbook for College Teachers, 2nded. San Francisco, CA: Jossey-Bass, 1993.

[10] L. Vygotsky, Thought and Language. Cambridge,MA: MIT Press, 1962.

[11] M. Alavi and D. Dufner, “Technology-mediatedcollaborative learning: A research perspective,”in Learning Together Online: Research onAsynchronous Learning Networks, S. R. Hiltzand R. Goldman, Eds. Mahwah, NJ: LawrenceErlbaum, 2004.

[12] R. C. W. Kwok and J. Ma, “Use of a group supportsystem for collaborative assessment,” Computers &Education, vol. 32, no. 2, pp. 109–125, 1999.

[13] E. V. Wilson, “Examnet asynchronous learningnetwork: Augmenting face-to-face course withstudent-developed exam questions,” Comput. Edu.,vol. 42, no. 1, pp. 87–107, 2004.

[14] J. Shindler, “Examining the soundness ofcollaborative essay exams in teacher educationcourses,” Nat. Forum Teacher Edu. J., vol. 12, no.3, p. 25, 2002.

[15] J. V. Shindler, “Greater than the sum of the parts?Examining the soundness of collaborative examsin teacher education courses,” Innovative HigherEdu., vol. 28, no. 4, p. 273, 2004.

[16] M. G. Simkin, “An experimental study of theeffectiveness of collaborative testing in anentry-level computer programming class,” J.Inform. Syst. Edu., vol. 16, no. 3, pp. 273–280,2005.

[17] N. Falchikov and J. Goldfinch, “Student peerassessment in higher education: A meta-analysiscomparing peer and teacher marks,” Rev.Educational Res., vol. 70, pp. 287–322, 2000.

[18] D. M. A. Sluijsmans, S. Brand-Gruwel, J. VanMerriënboer, and T. R. Bastiaens, “The training ofpeer assessment skills to promote the developmentof reflection skills in teacher education,” Stud.Educational Evaluation, vol. 29, no. 1, pp. 23–42,2003.

[19] K. J. Topping, “Peer assessment between studentsin college and university,” Rev. Edu. Res., vol. 68,no. 3, pp. 249–276, 1998.

[20] J. Shen, M. Bieber, and S. R. Hiltz, “Participatoryexaminations in asynchronous learning networks:Longitudinal evaluation results,” J. AsynchronousLearning Networks, vol. 9, no. 3, pp. 93–113, 2005.

[21] J. Shen, S. R. Hiltz, K. E. Cheng, Y. Cho,and M. Bieber, “Collaborative examinations forasynchronous learning networks: Evaluationresults,” in Proc. 34th Hawaii Int. Conf. SystemsSciences, 2001, pp. 1–10.

[22] N. Entwistle, “Promoting deep learning throughteaching and assessment,” in Proc. AAHE’s2000 Assessment Conf.: Rising Expectations forAssessment, 2000, pp. 9–20.

[23] C. Gordon and R. Debus, “Developing deeplearning approaches and personal teachingefficacy within a preservice teacher educationcontext,” Brit. J. Educational Psychology, vol. 72,pp. 483–511, 2002.

[24] J. B. Biggs, Student Approaches to Learningand Studying. Camberwell, Victoria: AustralianCouncil for Educational Research, 1987.

[25] F. Marton and R. Saljo, “On qualitative differencesin learning, outcome and process,” Brit. J.Educational Psychology, vol. 46, no. 1, pp. 4–11,1976.

[26] C. Tang, “Effects of different assessmentprocedures on tertiary students’ approaches tostudying,” Ph.D. thesis, University of Hong Kong,Hong Kong, 1991.

[27] C. Tang, “Effects of modes of assessment onstudents’ preparation strategies,” in ImprovingStudent Learning—Theory and Practice, G. Gibbs,Ed. Oxford: Oxford Center for Staff Development,1994, pp. 10–15.

[28] L. Suchman, Plans and Situated Actions.Cambridge, UK: Cambridge Univ. Press, 1987.

[29] E. Hutchins, Cognition in the Wild. Cambridge,MA: MIT Press, 1995.

[30] E. Wenger, Communities of Practice: Learning,Meaning, and Identity. New York: CambridgeUniv. Press, 1998.

[31] C. Gunawardena and F. Zittle, “Social presenceas a predictor of satisfaction within a computermediated conferencing environment,” Amer. J.Distance Learning, vol. 11, no. 3, pp. 8–26, 1997.

[32] J. Richardson and K. Swan, “An examinationof social presence in online learning: Students’perceived learning and satisfaction,” J.Asynchronous Learning Networks, vol. 7, no. 1, pp.68–88, 2003.

[33] A. G. Picciano, “Beyond student perceptions:Issues of interaction, presence, and performancein an online course,” J. Asynchronous LearningNetworks, vol. 6, no. 1, pp. 21–40, 2002.

[34] J. Shen, “Collaborative Examinations inAsynchronous Learning Networks: FieldExperiments on Collaborative Learning ThroughOnline Assessments,” Ph.D. thesis, Departmentof Information Systems, New Jersey Institute ofTechnology, Newark, 2005.

[35] R. Benbunan-Fich, S. R. Hiltz, and L. Harasim,“The online interaction learning model: Anintegrated theoretical framework for learningnetworks,” in Learning Together Online: Researchon Asynchronous Learning Networks, S. R. Hiltzand R. Goldman, Eds. Mahwah, NJ: LawrenceErlbaum, 2004, pp. 19–38.

[36] S. R. Hiltz and P. J. Shea, “The student in theonline classroom,” in Learning Together Online:Research on Asynchronous Learning Networks, S.R. Hiltz and R. Goldman, Eds. Mahwah, NJ:Lawrence Erlbaum, 2004, pp. 145–168.

[37] J. B. B. Arbaugh and R. Benbunan-Fich,“Contextual factors that influence ALNeffectiveness,” in Learning Together Online:Research on Asynchronous Learning Networks, S.R. Hiltz and R. Goldman, Eds. Mahwah, NJ:Lawrence Erlbaum, 2004, pp. 123–144.

[38] B. Bloom, A Taxonomy of Educational Objectives:The Classification of Educational Goals. New York:D. McKay Co., 1956.

[39] J. Shen, K. E. Cheng, Y. Cho, S. R. Hiltz, andM. Bieber, “Evaluation of an on-line collaborativeexamination process,” in Proc. Americas Conf.Information Systems, CA, 2000, pp. 1791–1797.



[40] J. Shen, K. E. Cheng, M. Bieber, and S. R. Hiltz,“Traditional in-class examination vs. collaborativeonline examination in asynchronous learningnetworks: Field evaluation results,” in Proc.Americas Conf. Information Systems, 2004, pp.1–10.

[41] D. Wu, M. Bieber, S. R. Hiltz, and H. Han,“Constructivist learning with participatoryexaminations,” in Proc. HICSS 37th Hawaii Int.Conf. Systems Sciences, 2004, pp. 12–20.

[42] J. Biggs, D. Kember, and D. Y. P. Leung, “Therevised two-factor study process questionnaire:R-Spq-2f,” Brit. J. Educational Psychology, vol. 71,no. 1, pp. 133–149, 2001.

[43] K. Cho and C. D. Schunn, “Scaffolded writing andrewriting in the discipline: A web-based reciprocalpeer review system,” Comput. Edu., vol. 48, no. 3,pp. 409–426, 2005.

[44] D. McConnell, “Examining collaborativeassessment processes in networked lifelonglearning,” J. Comput. Assisted Learning, vol. 15,no. 3, pp. 232–243, 1999.

Jia Shen received the Ph.D. degree in information systems fromNew Jersey Institute of Technology (NJIT) in 2005. She is anAssistant Professor in the College of Business Administration,Rider University, Lawrenceville, NJ. She has published in IEEETRANSACTIONS, the Journal of Asynchronous Learning Networks,and presented at many conferences in information systems.

Her research interests include human–computer interactions,E-Commerce, online learning, and virtual teamwork.

Starr Roxanne Hiltz received the Ph.D. degree from ColumbiaUniversity in 1969. She is Distinguished Professor Emerita,College of Computing Sciences, New Jersey Institute ofTechnology, Newark. A sociologist by training, she has spentmost of the last 30 years engaged in research on applicationsand social impacts of computer technology, publishing widely injournals including JMIS, Communications of the ACM, and IEEETRANSACTIONS. Her research interests currently include groupsupport systems, asynchronous learning networks, pervasivecomputing, and emergency response information systems. Hermost recent book, co-edited with Ricki Goldman, is LearningTogether Online: Research on Asynchronous Learning Networks(Erlbaum, 2005).

Michael Bieber received the Ph.D. degree in decision sciencesfrom the University of Pennsylvania in 1990. He currently is aProfessor in the Information Systems Department of the Collegeof Computing Sciences, New Jersey Institute of Technology,Newark. He is conducting research in several related areas:participatory learning, hypermedia functionality, automaticallygenerating links and services for web applications and fordigital libraries, relationship analysis (as part of the softwareengineering process), and virtual communities. Prof. Bieber isa member of the Association of Information Systems and ACMSIGWEB.


ieee transactions on professional ...bieber/pub/shb08_ieee_pc.pdfieee transactions on professional...

Documents