comparative effects of test-enhanced learning and self-explanation on long-term retention

Comparative effects of test-enhanced learning andself-explanation on long-term retentionDouglas P Larsen,1 Andrew C Butler2 & Henry L Roediger III3

CONTEXT Educators often encouragestudents to engage in active learning by generat-ing explanations for thematerial being learned,a method called self-explanation. Studies havealso demonstrated that repeated testingimproves retention. However, no studies havedirectly compared the two learningmethods.

METHODS Forty-seven Year 1 medical stu-dents completed the study. All students partic-ipated in a teaching session that covered fourclinical topics and was followed by four weeklylearning sessions. In the learning sessions, stu-dents were randomised to perform one offour learning activities for each topic: testingwith self-generated explanations (TE); testingwithout explanations (T); studying a reviewsheet with self-generated explanations (SE),and studying a review sheet without explana-tions (S). Students repeated the same activityfor each topic in all four sessions. Six monthslater, they took a free-recall clinical applica-tion test on all four topics.

RESULTS Repeated testing led to better long-term retention and application than repeatedly

studying the material (p < 0.0001, g2 = 0.33).Repeated generation of self-explanations alsoimproved long-term retention and application,but the effect was smaller (p < 0.0001,g2 = 0.08). When data were collapsed acrosstopics, both testing conditions produced betterfinal test performance than studying withself-explanation (TE = 40% > SE = 29%[p = 0.001, d = 0.70]; T = 36% > SE = 29%[p = 0.02, d = 0.48]). Studying with self-explanation led to better retention andapplication than studying without self-explanation (SE = 29% > S = 20%; p = 0.001,d = 0.68). Our analyses showed significantinteraction by topic (p = 0.001, g2 = 0.06),indicating some variation in the effectiveness ofthe interventions among topics.

CONCLUSIONS Testing and generating self-explanations are both learning activities thatcan be used to produce superior long-termretention and application of knowledge, buttesting is generally more effective than self-explanation alone.

Medical Education 2013: 47: 674–682doi: 10.1111/medu.12141

Discuss ideas arising from the article at

www.meduedu.com ‘discuss’

1Department of Neurology, Washington University in St. LouisSchool of Medicine, St. Louis, Missouri, USA2Department of Pscyhology & Neuroscience, Duke Univeristy,Durham, North Carolina, USA3Department of Psychology, Washington University in St. Louis,St. Louis, Missouri, USA

Correspondence: Dr Douglas Larsen, Department of Neurology,Washington University in St. Louis, 660 South Euclid Avenue,Campus Box 8111, St. Louis, Missouri 63110, USA.Tel: 00 1 314 454 6120; E-mail: [email protected]

674 ª 2013 John Wiley & Sons Ltd. MEDICAL EDUCATION 2013; 47: 674–682

test-enhanced learning

INTRODUCTION

One of the primary goals of medical education isto help students acquire a large body of knowl-edge in the hope that they will retain it for longperiods until they need to apply it in a futureclinical setting. Despite the immense challengespresented by this goal, relatively little research hasinvestigated the direct effects on long-term reten-tion of particular educational interventions. Onepotentially beneficial intervention that has recentlygarnered attention is a method called ‘test-enhanced learning’. Test-enhanced learning is aprocess that involves learners repeatedly retrievinginformation through tests (or other opportunitiesto practise retrieval), which results in superiorlong-term retention of that information.1,2 In test-enhanced learning, tests are conceptualised aslearning tools, rather than as assessment tools.Although testing may have indirect effects on stu-dent motivation and study habits, a large body ofresearch has shown that the direct act of retriev-ing information from memory enhances learningand retention better than simply increasing thequantity or quality of study.2 Research in cognitivepsychology laboratories has shown that thismethod for learning is superior to repeated studyof the same information.2–4 Recent studies inactual educational settings have also demonstratedthat test-enhanced learning can increase the reten-tion of knowledge for extended intervals of6–9 months.5–7

Despite the established benefits of test-enhancedlearning, its implementation in educational prac-tice raises two important questions. Does test-enhanced learning simply represent rote memori-sation of facts with little ability to transfer or applythat knowledge to new settings? How well doestest-enhanced learning compare with other meth-ods of active learning? A major strand of researchin educational psychology has focused on construc-tivism, which emphasises the construction by learn-ers of their own personal systems of under-standing.8 In constructivism, the recall of facts isonly important to the degree that it representsthe framework of why information is importantand how items of knowledge work together.9 Inthis light, the testing of factual information isoften seen as a misdirection of educational priori-ties.8 In order to help learners accomplish thegoal of generating their own systems of under-standing, several elaborative techniques have beendeveloped.

One technique that has shown promise involves hav-ing students generate explanations about why a par-ticular piece of information is important and how itrelates to their existing knowledge. In this method,often referred to as ‘self-explanation’, learners arethought to benefit from generating their own per-sonal understanding of the to-be-learned informa-tion because this helps them to integrate the newinformation with existing knowledge.10 Studies haveshown that this technique can be beneficial over awide variety of materials and learners. However, thedurability of the benefits obtained by engaging inself-explanation is unclear because most studieshave measured the effects after very brief intervalsof time (e.g. anywhere from minutes to a couple ofweeks).11–14 Clearly, more research is needed toascertain whether these benefits last over education-ally meaningful intervals of time.

In the present study, we directly compared the effi-cacy of, respectively, testing and generating self-explanations in terms of promoting the long-termretention and application of clinical material.Importantly, the two techniques are not mutuallyexclusive and thus we also investigated the efficacyof combining testing and self-explanation. Bymanipulating the presence or absence of each tech-nique as an independent variable, we were able toisolate the individual and combined contributionsof these learning methods to long-term retention.In our design, the two independent variables werefully crossed to result in four different learningactivities: testing with self-generated explanations(TE); testing without explanations (T); studying areview sheet with self-generated explanations (SE),and studying a review sheet without explanations(S). Students were given an initial teaching sessionthat introduced four neurology topics, after whichthey engaged in four testing/study sessions atweekly intervals in which they performed one of thefour learning activities for each topic. Long-termretention and transfer of knowledge were assessedin a free-recall clinical application test given6 months after the initial teaching session.

METHODS

Participants

Forty-nine Year 1 medical students were recruited toparticipate in the study. Year 1 medical studentswere selected because they had no prior exposureto the topics taught in the study, which ensured amore uniform level of baseline knowledge across

ª 2013 John Wiley & Sons Ltd. MEDICAL EDUCATION 2013; 47: 674–682 675

Testing and self-explanation

the cohort. All students completed the initial testingor study sessions; however, two students did notcomplete the final test and therefore their data wereexcluded from the statistical analyses. Participationwas voluntary; all students provided consent. Allactivities related to the study were performed out-side standard class time. The study was conductedindependently of any course taught in the medicalschool. Students were reimbursed for the time theyspent participating in the study at a rate of approxi-mately US$15/hour. Students spent a total of about9 hours in study-related activities. The study wasapproved by the institution’s Human ResearchProtection Office.

Materials

Materials were developed to cover four clinical neu-rology topics: seizures; optic neuritis; myastheniagravis, and migraine. The materials covered theinformation a student would be expected to drawupon in interviewing, diagnosing and treatingpatients with these conditions. Four types of learn-ing materials were developed for each topic: (i) ashort-answer written test (T); (ii) a written test thatalso required students to write out their explana-tions of their answers (TE); (iii) a review sheet thatrestated all of the information covered (S, forstudy), and (iv) a review sheet that contained spacefor students to write out their explanations for thestatements on the review sheet (SE). The contentcovered in each learning activity was identical. Thecontent required students to learn 26–30 items foreach topic. An essay application test was developedto assess long-term retention and transfer. This testconsisted of a short clinical scenario: students weregiven a patient’s demographic information andchief complaint, and were then asked to use theinformation they had learned about the topic tooutline their approach to managing the patient’scondition. They were told to include all of the infor-mation that had been covered in the initial teachingsession and further practised through the variouslearning activities. The test sheet presented the briefclinical scenario at the top of the page and theremainder was left blank for students to write outtheir responses.

Questionnaires were developed to ascertain fromstudents how the various activities influenced theirlearning. These were administered both after theinitial learning sessions were completed and afterthe final test at the end of the study. The question-naires used open-ended questions in order to allowstudents to fully describe their thoughts and obser-

vations. For example, students were asked: ‘Describehow the review sheet helped (or did not help) youto learn’ and ‘Describe how the explanations on thetests or review sheets helped (or did not help) youto learn.’

Procedures

Students participated in two teaching sessions heldon consecutive days. Two topics were covered oneach day; 1 hour was dedicated to each topic. Thetopics were taught in an interactive didactic format.At the end of each teaching session, each studentwas randomised in a counterbalanced fashion toengage in one of the four learning activities (i.e. T,TE, S or SE) for each of the topics covered in thatsession. In this way, each student performed all fourlearning activities, but each activity was paired witha different topic. For example, Student 1 may havebeen randomised to testing alone on seizures, test-ing with self-explanations on optic neuritis, studyingthe review sheet alone on myasthenia gravis, andstudying with self-explanations on migraine. How-ever, Student 2 may have been randomised to test-ing alone on optic neuritis, testing with self-explanations on myasthenia gravis, studying thereview sheet alone on migraine, and studying thereview sheet with self-explanations on seizures. Thecounterbalancing ensured that the pairing of topicand learning activity occurred equally often acrossthe student sample.

Once a week for the next 3 weeks students returnedfor additional test/study sessions in which they per-formed the same learning activities they had donepreviously for each of the four topics. The students’pairings of topics and learning activities did notchange between test/study sessions (i.e. Student 1would perform the same learning activities for eachtopic as in the initial test/study session and Student2 would perform the same learning activities for hisor her assigned topics as in the initial learning/study session). Including the test/study sessionimmediately after the teaching session, studentscompleted four test/study sessions in total. In thisway, each student gained equal practice for eachlearning activity over four repetitions. No time limitswere placed on any of the activities, but generallystudents completed all four activities in about anhour per test/study session.

During each test/study session, once students hadcompleted the tests and studied the review sheets,they were asked to score their tests and comparetheir explanations (for those activities requiring


D P Larsen et al

explanations) with an answer key. Explanations werescored only according to their completion because thepurpose of generating self-explanations was to havestudents use their own unique explanations, whichthey believed would help them to understand andretain the information. The example explanations onthe answer key were provided to help them if they wereunable to come up with an explanation on their own.Students were also asked to rate their overall effort inproviding the explanations. After the fourth test/studysession, students were asked to fill out a questionnairedescribing their perceptions of their learning.

Students were explicitly instructed not to quiz them-selves with the review sheets in order to avoid con-founding the study activities with self-testing. Theywere allowed to read and re-read the review sheets asmany times as they felt necessary to learn the material.Students were also explicitly instructed not to studythe material outside the test/study sessions. They werealso told not to review the material with one another.

About 6 months after the initial teaching sessions,students returned to take an application test that cov-ered all four topics. In the application test, studentswere given a clinical scenario and asked to use all ofthe information they had learned on the appropriatetopic to describe how they would deal with the sce-nario. After taking the final test, they filled out aquestionnaire. Responses to the interim and finalquestionnaires were compiled and analysed.

Scoring and statistical analyses

Two raters scored each final test in a blinded fash-ion. Because students were asked to include allinformation from the initial teaching session intheir free-recall tests, raters used the answer sheetsfrom the initial tests as checklists with which toscore the presence or absence of the required infor-mation in the students’ final tests. Kappa (j) statis-tics were calculated to measure inter-rater reliabilityin the scoring of the final tests. Inter-rater reliabilityfor the application test was excellent (j = 0.93) andthus the initial tests were scored by a single rater. Aspart of our analysis of the initial tests, we alsodirectly measured students’ compliance with writingexplanations in the SE and TE conditions by scor-ing the presence or absence of explanations foritems in each condition. For the TE condition, wescored only items that students answered correctlybecause it is difficult to interpret the reason for theabsence of an explanation for an item that wasincorrect; students could have failed to provide anexplanation for an incorrect item because they

failed to retrieve that item (i.e. the response wasomitted), because they did not know the materialwell enough to generate an explanation or becausethey simply chose not to write an explanation. Stu-dents were also asked to rate their effort in generat-ing explanations on the review sheets and tests ineach of the four learning sessions using a 10-pointscale on which a score of 10 represented maximumeffort (‘Full effort, I did my best to make eachexplanation useful for me to remember’) and ascore of 1 represented minimal effort (‘No effort, Ionly wrote explanations to fill the space’).

All analyses were performed using SPSS Version 18(SPSS, Inc., Chicago, IL, USA) and Microsoft Excelfor Mac. The results for the final application testwere analysed using a repeated-measures analysis ofvariance (ANOVA) that included testing (i.e. test versusre-study), self-explanation (i.e. explanation versus noexplanation), and neurological topic as independentvariables. The proportion of correct responses onthe final tests, a measure of long-term retention andapplication, served as the dependent variable.Planned post hoc t-test comparisons were madebetween the various learning activities. Eta-squared(g2) and Cohen’s d are the effect sizes reported forthe ANOVA and t-test analyses, respectively.

RESULTS

Initial learning

Performance on the initial tests showed that the stu-dents had good retention of the material immedi-ately after the teaching sessions. As expected,because no manipulation had yet been imple-mented, students showed roughly equivalent perfor-mance on Test 1 for the TE (0.74) and T (0.72)activities. After an initial decline in performancefrom Test 1 to Test 2, performance steadilyimproved in both testing conditions from Test 2until Test 4 (Table 1). However, the rate ofimprovement was greater when self-explanation wasincluded with testing. On Test 4, which occurred3 weeks after the initial teaching sessions, the TEactivity (0.86) resulted in better performance thanthe T activity (0.78). A 4 (test number) 9 2 (testingcondition) ANOVA on the initial learning phase per-formance confirmed these observations by revealinga significant main effect of test number(F3,138 = 69.91, mean squared error [MSE] = 0.01,p < 0.0001, g2 = 0.26), as well as a significant inter-action (F3,138 = 3.91, MSE = 0.01, p = 0.005,g2 = 0.01), indicating that the rate of change dif-



fered significantly between conditions. The maineffect of testing condition was marginally significant(F1,46 = 3.91, MSE = 0.06, p = 0.054, g2 = 0.03).Overall, these findings indicate that performance atthe start of the two testing conditions was similar,but the testing with self-explanation (TE) activityproduced a slightly greater increase in performanceacross the four initial tests relative to the testingwithout self-explanation (T) activity.

In addition to test performance, a few other mea-sures of student performance during initial learningwere collected. After each study activity, studentswere asked how many times they had read throughthe review sheets. On average, they reported havingread through the review sheet 1.7 times per sessionfor the study with explanations (SE) activity and 2.0times per session for the study without explanations(S) activity (p = 0.02, d = 0.29). In our analysis ofstudent compliance with the request to write self-explanations, students were found to have produced

an explanation for 96% of the items in the studywith self-explanation condition; however, theygenerated an explanation for only 71% of the itemsthey successfully retrieved in the test with self-explanation condition (p < 0.0001, d = 0.96).Finally, students were asked to rate on a 10-pointscale their effort in generating explanations on thereview sheets and tests; no significant differenceswere identified in ratings of effort. When students’reports of effort were collapsed across the fourtesting/study sessions, mean scores of 6.7 and 6.4 inthe study with self-explanation condition and testwith self-explanation condition, respectively(p = 0.11) were identified.

Final test of retention and application

The final test was administered approximately6 months after the initial teaching sessions. Figure 1depicts the proportion of correct responses on thefinal free-recall application test. Overall, the TEactivity produced the best performance followed,respectively, by the T, SE and S activities. A 2 (test-ing) 9 2 (self-explanation) 9 4 (topic) repeated-measures ANOVA showed a significant main effect oftesting (F1,43 = 122.59, MSE = 0.01, p < 0.0001,g2 = 0.33) and a significant main effect of self-explanation (F1,43 = 20.90, MSE = 0.01, p < 0.0001,g2 = 0.08). Although both testing and generatingself-explanations increased long-term retention andapplication, it is important to note that the magni-tude of the effect size for testing is much larger. Inaddition to the two main effects, there was a signifi-cant interaction between the learning conditions(F1,43 = 4.12, MSE = 0.01, p = 0.049, g2 = 0.01),indicating that generating self-explanations helpedmore when combined with studying a review sheet(i.e. SE versus S) than when used with testing (i.e. TEversus T). Follow-up pairwise comparisons showedthat both testing conditions produced significantlybetter retention and application than studying withself-explanation (TE = 0.40 > SE = 0.29 [p = 0.001,d = 0.70]; T = 0.36 > SE = 0.29 [p = 0.02,d = 0.48]). However, there was no significant differ-ence between the TE and T activities (p = 0.077,d = 0.28). The SE activity produced significantly bet-ter performance than the S activity(SE = 0.29 > S = 0.20; p = 0.001, d = 0.68).

The primary findings were qualified by interactionsbetween the learning activities and the neurologicaltopics. The repeated-measures ANOVA revealed a sig-nificant interaction between testing and topic(F3,43 = 4.03, MSE = 0.01, p = 0.013, g2 = 0.03), aswell as between self-explanation and topic

Table 1 Mean proportions of correct answers on the initiallearning tests in the testing with self-explanation (TE) andtesting without self-explanation (T) conditions

Learning

activity Test 1 Test 2 Test 3 Test 4

TE 0.74 0.63 0.77 0.86

T 0.72 0.61 0.68 0.78

Figure 1 Mean proportions of correct answers on the finalapplication test as a function of testing (test versus study)and self-explanation (self-explanation versus no self-explanation). Error bars represent 95% confidenceintervals


D P Larsen et al

(F3,43 = 4.31, MSE = 0.01, p = 0.01, g2 = 0.05). Inaddition, there was significant three-way interactionamong testing, self-explanation and topic(F1.43 = 6.45, MSE = 0.01, p = 0.001, g2 = 0.06).Overall, these interactions indicate some variationin the relative effectiveness of various learning activi-ties among the topics. Table 2 shows the final testperformance by topic. The variation among topicsappears to be driven in part by differences in theeffectiveness of generating self-explanations. Recallin the SE activity condition was equivalent to recallin the two testing activity conditions for some topics,but not for others. Similarly, the TE and T activitiesled to equivalent performance on some topics, butnot others. Despite this variation by topic, one keyresult remained consistent across topics: both test-ing activities yielded superior performance relativeto studying without self-explanation.

Questionnaire responses

After completing the four testing/study sessions, stu-dents were asked to describe how taking the testsand studying the review sheets had or had nothelped in their learning. They were also asked toexplain how writing out the explanations hadaffected their learning. Students overwhelmingly feltthat repeated testing was beneficial: 90% of studentsreported testing to be helpful in their learning.Conversely, 42% of students reported that studyingthe review sheet was helpful to varying degrees. Atotal of 67% of students reported that writing expla-nations was helpful in some form and 50% com-mented that the explanations helped them tounderstand why material was important, to make con-nections between the study material and other facts,or to group information into overarching themes.However, 31% of students regarded the explanations

as redundant or unnecessary. Many of these studentscommented that the ability to answer the question ona test indicated that they had sufficient knowledge ofthe material and giving an explanation did not addsignificantly to their understanding.

At the end of the study, the students completed a sec-ond questionnaire asking whether they had reviewedmaterial outside of the study and whether they werewilling to take repeated tests as part of futurecourses. None of the students indicated that theyhad studied the material outside the testing or studysessions, but a few of them mentioned that some ofthe topics had arisen in their regular medical schoolcourses. Despite the fact that students wereinstructed not to quiz themselves with the reviewsheets, 17% reported that they had engaged in someform of self-quizzing with the review sheet. Overall,60% of students stated that they would be willing totake repeated tests as part of future courses. Interest-ingly, 11% of the students spontaneously commentedthat they would only want to take repeated tests if thetests were to be used for learning purposes only andwere not to contribute to their grades.

DISCUSSION

Our experiment demonstrates that both repeatedtesting and the repeated generation of self-explana-tions produced superior long-term retention andtransfer of knowledge on a free-recall clinical appli-cation test compared with repeated studying withoutself-explanation. Our findings have both theoreticaland practical implications.

One goal of the present study was to compare therelative efficacies of retrieval practice (i.e. testing)

Table 2 Mean proportions of correct answers on the final application test as a function of initial learning activity and topic in thetesting with self-explanation (TE), testing without self-explanation (T), studying with self-explanation (SE) and studying without self-explanation (S) conditions

Learning

activity

Topic

Seizures Migraine Myasthenia

Optic

neuritis Grand mean

TE 0.40 0.39 0.49 0.32 0.40

T 0.44 0.32 0.36 0.33 0.36

SE 0.39 0.32 0.21 0.25 0.29

S 0.24 0.26 0.19 0.12 0.20



and self-explanation as methods for facilitatinglearning. Although both interventions significantlyimproved long-term retention and transfer, theireffects in isolation were not equal in magnitude.Testing had a much larger effect on final test per-formance (g2 = 0.33) relative to self-explanation(g2 = 0.08). In planned pairwise comparisons, bothtesting with self-explanation and testing without self-explanation produced moderate to large effectscompared with repeated study with self-explanation(d = 0.70 and d = 0.48, respectively). Thus, ourstudy suggests that repeated testing produces morerobust long-term retention and transfer of knowl-edge relative to repeated self-explanation. This find-ing is concordant with the results of other researchin which retrieval practice has been compared withother active learning techniques. For example, Kar-picke and Blunt found that repeated testing over a1-week interval resulted in superior retention com-pared with concept mapping, an active study tech-nique that is considered by many in education to bean excellent method of integrative learning.15

It should be noted that 17% of students reportedthat they had engaged in some form of self-quizzingwhile studying the review sheets despite explicitinstructions not to do so. Because of the uncon-trolled nature of this activity and the relatively smallpercentage of students who had engaged in it, it isdifficult to measure its effect on final test perfor-mance. However, given the benefits of testingobserved in the present study and others, it wouldbe expected that the presence of some self-testingwould increase final test performance in the studycondition, thereby reducing the differencesobserved among the four learning activities. Thus,the presence of self-quizzing in the study conditionmakes it all the more remarkable that repeated test-ing produced such a robust effect on retention.

The comparison between testing and self-explanationyields some insights into why they may have differenteffects on retention. Completing a test may producebetter retention because it is an inherently more diffi-cult task than generating explanations for informa-tion. Research has shown that the difficulty inherentin encoding and retrieving knowledge leads to moredurable learning.16,17 The work involved in the actualact of retrieval has a direct effect on the learner’sability to recall that information later. Agrawal et al.have suggested that testing serves as a self-monitoringintervention in which learners track and adjust theirstrategies of knowing and understanding informationbased on their ability to retrieve that knowledge.18

Self-explanation may also be thought of as a self-mon-

itoring intervention as students identify their own sys-tems of knowing information. However, our resultswould indicate that testing might lead to more robustretention because it is more effective than self-expla-nation at causing students to develop successfulsystems of organising and retrieving informationbecause it forces them to more fully assess the effi-cacy of their current systems when they are con-fronted with a question. The findings of Pyc andRawson would support this conclusion.19 Theseauthors demonstrated that testing does lead to achange in knowledge organisation compared withsimply re-studying when testing is coupled with feed-back.19 This effect is caused by students’ reassessingof their structures of knowledge and discarding ofineffective or inaccurate systems for more successfulstructures. Zaromb and Roediger also showed thatthe retrieval effort in free-recall testing improves alearner’s mental organisation of information moreeffectively than does study of the same information.20

Interestingly, combining testing with self-explana-tion significantly improved performance during theinitial learning phase relative to testing withoutexplanation. Although this significant advantage didnot endure to the final assessment 5 months later,there was still a small numerical difference in favourof testing with self-explanation. One reason why thisadditional benefit may not have been as durable asmight have been hoped is that the rate of generat-ing explanations in the TE activity is lower (71% ofcorrect items versus 96% in the SE activity). Alterna-tively, the lower rate of generating explanations mayhave been driven by the students’ perception thatthe explanations were unnecessary on the tests andwere redundant. Many students commented thattheir answering of questions on the test indicatedthat they already knew the information from theexplanation and had no need to make it explicit.The benefit of adding self-explanation to testingmay depend on the type of information beinglearned. Understanding how to identify topics thatmost benefit from the combination of self-explana-tion and testing may be a fruitful direction forfuture research.

Although our findings suggest that self-explanationis not as robust as testing, it is important to notethat generating explanations had a significant effecton long-term retention and transfer of knowledge.The finding that studying with self-explanation pro-duced superior performance on the final test rela-tive to studying without self-explanation (d = 0.68)is important because it demonstrates the durabilityof the benefits of engaging in self-explanation dur-


D P Larsen et al

ing learning on a test given 6 months after initialteaching. Previous studies on self-explanation haveonly assessed performance after relatively shortretention intervals ranging from a few minutes to afew weeks.11–14 The present study extends this find-ing to show that generating self-explanationsincreases long-term retention over an educationallymeaningful retention interval of 6 months from ini-tial learning.

Another goal of the present study was to investigatewhether test-enhanced learning would support thetransfer of knowledge to new settings (i.e. relative tosimply promoting the rote memorisation of facts).To this end, we used a final assessment thatrequired students to apply their knowledge to a clin-ical scenario in which they had to outline anapproach to determining a hypothetical patient’scondition but were given only the patient’s demo-graphic data and information on his or her chiefcomplaint. Our findings show that repeated retrievalpractice during learning significantly improved stu-dents’ ability to apply their knowledge in this newsetting. Although the overall level of performancewas lower than that hoped for in medical education,it is important to consider the nature of the finalapplication test. Six months after learning the mate-rial, students were asked to retrieve and apply asmuch of their knowledge as possible (from 26 to 30items) with very little prompting or structure; this issimilar to the requirements of an essay test. Giventhat the students had minimal exposure to thematerial in the intervening months, and in view ofthe difficulty of a final essay test, we think it isimpressive that repeated testing produced 36–40%correct performance (almost double the amountderived by studying without self-explanations).

Our finding that repeated testing promotes betterlong-term retention and transfer fits well with thefindings of recent studies which have also reportedthat retrieval practice improves learners’ applicationof knowledge to new situations.21–25 Whereas manystudies show that testing improves the retention ofinformation that is retrieved from memory, thesenew studies suggest that repeated retrieval practicemay also improve understanding of that informa-tion. By contrast, generating self-explanations mayimprove understanding, but may not be as powerfula tool for promoting long-term retention.

Our study does have some limitations. Our studydesign does not fully distinguish the effect of writ-ing self-explanations from that of reading an answersheet because sample explanations were provided

on the answer sheet. The sample explanations weregiven in an effort to avoid having to penalisestudents who might have forgotten so much infor-mation that they could not generate an explanationor who were unsure how to write an explanation. Itshould be noted that even with this additional help,testing had a more robust effect than making theexplanations available. Future studies of self-expla-nations will need to provide additional comparisonsto further tease out the effects of providing sampleexplanations.

In terms of other limitations, we also observed aninteraction indicating some variation in the effec-tiveness of the learning techniques among topics.In interpreting this result, it is important to con-sider that our materials consisted of four heteroge-neous topics. Studies in laboratory settings typicallyuse carefully constructed materials, such as wordpairs and short texts, which are matched on manydifferent dimensions. Our materials are more repre-sentative of the uncontrolled variety of informationfound in a typical medical school course. Althoughwe were careful to have each topic cover roughlythe same amount of information, the nature of theinformation in each topic is inherently different.Some types of information may be learned throughself-explanation just as well as through testing. Simi-larly, self-explanation may be more effective whenlearners have significant prior knowledge to helpgenerate an explanation or when students havereceived explicit training in generating explana-tions.10,14,26 Our study showed that repeated testingwith and without self-explanation led to perfor-mance equivalent to that derived by studying withself-explanation on certain topics, but to superiorperformance for others. However, self-explanationnever produced significantly better retention andtransfer of knowledge than testing. Thus, our find-ings suggest that for educators who must choosebetween the two learning activities, repeated testingappears to be the more robust and reliable tech-nique.

CONCLUSIONS

Our study has important implications for educators.We demonstrated that repeated testing and self-explanation both lead to superior long-term reten-tion and transfer compared with repeated studyingof the same information. Although for some topicsthese two learning interventions produce equivalentperformance, our findings show that the act ofrepeated retrieval generally produces superior reten-



tion than the act of self-explanation. Educatorsshould seek to incorporate both of these powerfullearning techniques into their curricula in order toenhance long-term retention and the application oflearned material to clinical settings.

Contributors: DPL developed the study concept,contributed to its design, conducted the study, collectedand analysed the data and drafted the paper. ACB con-tributed to the study design and the analysis of data. HLRcontributed to the study design. All authors contributedto the critical revision of the paper and approved the finalmanuscript for publication.Acknowledgements: we would like to thank Jessye Brick andLaura Najjar for their assistance in scoring tests and com-piling data.Funding: this study was funded by an Education ResearchGrant from the American Academy of Neurology and bythe McDonnell Center for Systems Neuroscience at theWashington University in St. Louis School of Medicine.Conflicts of interest: none.Ethical approval: this study was approved by the Washing-ton University in St Louis School of Medicine’s HumanResearch Protection Office.

REFERENCES

1 Larsen DP, Butler AC, Roediger HL III. Test-enhanced learning in medical education. Med Educ2008;42:959–66.

2 Roediger HL III, Karpicke JD. The power of testingmemory: basic research and implications foreducational practice. Perspect Psychol Sci 2006;1:181–210.

3 Karpicke JD, Roediger HL III. The critical importanceof retrieval for learning. Science 2008;319 (5865):966–8.

4 Roediger HL III, Karpicke JD. Test-enhancedlearning: taking memory improves long-termretention. Psychol Sci 2006;17:249–55.

5 Larsen DP, Butler AC, Roediger HL III. Repeatedtesting improves long-term retention relative torepeated study: a randomised, controlled trial. MedEduc 2009;43:1174–81.

6 McDaniel MA, Agarwal PK, Huelser BJ, McDermott KB,Roediger HL III. Test-enhanced learning in a middleschool science classroom: the effects of quiz frequencyand placement. J Educ Psychol 2011;103:399–414.

7 Carpenter SK, Pashler H, Cepeda NJ. Using tests toenhance 8th grade students’ retention of US historyfacts. Appl Cogn Psychol 2009;23:760–71.

8 Duffy TM, Cunningham DJ. Constructivism:implications for the design and delivery ofinstruction. In: Jonassen DJ, ed. Handbook of Researchfor Educational Communication and Technology. NewYork, NY: McMillan 1996;170–98.

9 Biggs J. Enhancing teaching through constructivealignment. High Educ 1996;32:347–64.

10 Dunlosky J, Rawson KA, Marsh EJ, Nathan MJ,Willingham DT. Improving students’ learning witheffective learning techniques: promising directionsfrom cognitive and educational psychology. Psychol SciPublic Interest 2013;14:4–58.

11 Berry DC. Metacognitive experience and transfer oflogical reasoning. Q J Exp Psychol 1983;35A:39–49.

12 Chi MTH, de Leeuw N, Chiu M-H, LaVancher C.Eliciting self-explanations improves understanding.Cogn Sci 1994;18:439–77.

13 Rittle-Johnson B. Promoting transfer: effects of self-explanation and direct instruction. Child Dev 2006;77:1–15.

14 Wong RMF, Lawson MJ, Keeves J. The effects of self-explanation training on students’ problem solving inhigh-school mathematics. Learn Instr 2002;12:233–62.

15 Karpicke JD, Blunt JR. Retrieval practice producesmore learning than elaborative studying with conceptmapping. Science 2011;331:772–5.

16 Pyc , MA , Rawson KA. Testing the retrieval efforthypothesis: does greater difficulty correctly recallinginformation lead to higher levels of memory? J MemLang 2009;60:437–47.

17 Bjork RA. Memory and metamemory considerationsin the training of human beings. In: Metcalfe J,Shimamura A, eds. Metacognition: Knowing aboutKnowing. Cambridge, MA: MIT Press 1994;185–205.

18 Agrawal S, Norman GR, Eva KW. Influences onmedical students’ self-regulated learning after testcompletion. Med Educ 2012;46:326–35.

19 Pyc MA, Rawson KA. Why is test-restudy practicebeneficial for memory? An evaluation of the mediatorshift hypothesis. J Exp Psychol Learn Mem Cogn2012;38:737–46.

20 Zaromb FM, Roediger HL III. The testing effect infree recall is associated with enhanced organisationprocesses. Mem Cognit 2010;38:995–1008.

21 Butler AC. Repeated testing produces superiortransfer of learning relative to repeated studying.J Exp Psychol Learn Mem Cogn 2010;36:1118–33.

22 Johnson CI, Mayer RE. A testing effect withmultimedialearning. J Educ Psychol 2009;101:621–9.

23 Rohrer D, Taylor K, Sholar B. Tests enhance thetransfer of learning. J Exp Psychol Learn Mem Cogn2010;36:233–9.

24 Kang SHK, McDaniel MA, Pashler H. Effects oftesting on learning of functions. Psychon Bull Rev2011;18:998–1005.

25 Larsen DP, Butler AC, Roediger HL III. Theimportance of seeing the patient: test-enhancedlearning with standardised patients and written testsimproves clinical application of knowledge. AdvHealth Sci Educ 2012. doi: 10.1007/s10459-012-9379-7(Epub ahead of print).

26 Woloshyn VE, Pressley M, Schneider W. Elaborative-interrogation and prior knowledge effects on learningof facts. J Educ Psychol 1992;84:115–24.

Received 11 June 2012; editorial comments to author 4 September2012, accepted for publication 18 December 2012


D P Larsen et al

comparative effects of test-enhanced learning and self-explanation on long-term retention

Documents