RESEARCH INTO PRACTICE

The Effect of Retrieval on Post-taskEnjoyment of Studying

Daniel A. Clark & Marilla Svinicki

# Springer Science+Business Media New York 2014

Abstract Although active retrieval is an extremely effective study method, students continueto use less effective methods (Karpicke, Journal of Experimental Psychology General, 138(4),469–486, 2009; Hartwig and Dunlosky, Psychonomic Bulletin & Review, 19(1), 126–134,2012). There are likely many underlying reasons for using less effective methods, but onepossibility could be that active retrieval is simply not as enjoyable as other methods. In thepresent study, following a learning task, undergraduates rated active retrieval as less enjoyablethan re-reading. Performance on the initial retrieval task, preexisting rating of enjoyment of thecontent, and performance incentives moderated the students’ ratings of the study techniques.

Keywords Testing effect .Motivation . Incentives . Learning .Memory.Metacognition .

Intrinsic motivation . Study strategies

Dunlosky et al. (2013) recently reviewed evidence of several different study techniques,finding that retrieval was the most empirically supported study method. Similarly, in a studycommissioned by the National Institute for Education Research, active retrieval was one of twoinstructional interventions that was rated as having sufficient empirical evidence to warrantwidespread application in educational settings (Pashler et al. 2007). Two meta-analyses ofstudies in college settings found that instructors may be able to improve student learning by asmuch as half of a standard deviation by using frequent quizzes in the classroom whenpreparing students for exams (Bangert-Drowns et al. 1991; Phelps 2012).

Although research has consistently demonstrated the effectiveness of active retrieval forlearning, it is not the most commonly used study strategy among college students (Karpickeet al. 2009; Kornell et al. 2011). Karpicke et al. asked college students if they would like to“self-test (retrieval),” re-read a chapter, or use a different technique. When given this choice,only 18 % of the students chose to use retrieval. The majority of the students (57 %) chose tosimply re-read the text before taking tests.

There are likely multiple reasons why students do not regularly use active retrieval. Onepossibility is that students view retrieval only as a method for assessing their learning ratherthan an actual learning strategy. Students with this perspective might choose not to use

Educ Psychol RevDOI 10.1007/s10648-014-9272-4

D. A. Clark (*) :M. SvinickiEducational Psychology, The University of Texas at Austin, 1 University Station D5800,Austin, TX 78712, USAe-mail: dac99a@gmail.com

retrieval while they are in the process of learning, but only after they feel they have learned thematerial. Two recent studies have found that approximately 60 % of students used retrieval forthis purpose, instead of using retrieval as a method for learning (Hartwig and Dunlosky 2012;Kornell et al. 2011). Unfortunately, using this method results in suboptimal long-term learningbecause students discontinue using retrieval too quickly (Karpicke 2009). Another possibilityis that students do not use retrieval simply because they are not aware of the benefits. McCabe(2010) found that college students predicted that they would learn more from re-reading a textinstead of using retrieval. The reality of their learning performance was that retrieval was moreeffective than re-reading, despite the student’s lack of awareness (McCabe). In a recentlypublished paper, Einstein et al. (2012) demonstrated the efficacy of retrieval to students byrequiring the students to engage in a lab experiment where they used both re-reading andretrieval. After analyzing the results of the lab experiment themselves, the students reportedusing retrieval more in their studying (Einstein et al.).

Self-Determination Theory and Utilizing Retrieval

From the empirical perspective of researchers, it seems entirely irrational that students do notregularly use retrieval while studying. However, research from the perspective of Self-Determination Theory (SDT; Deci and Ryan 1980) could partially explain a possible under-lying mechanism concerning the experience of using retrieval. SDT is a complex multifacetedoverarching theory that includes five different mini-theories related to motivation, individualdifferences, and contextual effects. Cognitive Evaluation Theory (CET) is a mini-theory withinthe overall SDT framework that attempts to account for situations conducive to different typesof motivation. More specifically, it separates intrinsic motivation, which “involves doingsomething because it is inherently interesting or enjoyable (Deci and Ryan 2000)” fromextrinsic motivation, which involves engaging in an activity because it leads to a separableoutcome. Although it is difficult to determine whether or not a particular person willfind a particular activity intrinsically enjoyable, SDT theorists Deci and Ryan assert thatsome activities are more intrinsically interesting than others for a larger percentage ofindividuals.

Along this train of thought, one important question to be considered in the current study iswhether or not students generally experience retrieval as an inherently enjoyable activity whencompared to using re-studying as a study method. If retrieval is inherently enjoyable in additionto being effective, it seems that research should find retrieval to be the most commonly utilizedstudy method among students. However, this does not appear to be the current situation.

Conditions That May Moderate the Effect of Retrieval on Motivation

Feedback Although Deci and Ryan (2000) asserted that some activities may be moreinteresting than others, they suggested that intrinsic motivation is more likely the resultof an interaction between internal and situational characteristics. CET suggests that condi-tions of positive feedback may enhance intrinsic motivation, while negative feedback maydecrease intrinsic motivation (Deci and Cascio 1972). In some conditions, feedback is notprovided at all. Vallerand and Reid (1984) conducted a study in which college studentswere asked to balance themselves on a stabilometer across three different conditions. Theseconditions involved receiving positive, negative, or no feedback. Not surprisingly, theparticipants in the positive feedback condition reported the highest level of intrinsic

Educ Psychol Rev

motivation following the task. Following the positive feedback group, the group who didnot receive any verbal feedback reported the second highest ratings of intrinsic motivation.Finally, receiving negative verbal feedback resulted in the lowest levels of intrinsicmotivation (Vallerand and Reid 1984).

Based on the Vallerand and Reid (1984) finding that no feedback was better thannegative feedback, it was predicted that re-studying information, in which no feedbackconcerning performance is involved, might result in higher levels of post-task enjoymentthan engaging in retrieval. Although participants were as likely to incur positive feedbackas negative, it is unlikely that they will be able to obtain a perfect score and evade negativefeedback altogether.

In addition, using retrieval involves the individual being opened up to the possibility ofnegative feedback, which SDT research suggests may deflate intrinsic motivation for a task.Therefore, it was also predicted that the performance on an initial retrieval attempt (which is arequired element of using retrieval as a study method) would moderate the extent to whichretrieval reduces intrinsic motivation. More specifically, it was predicted that performance onthe initial retrieval task would be positively related to enjoyment of the task.

Extrinsic Incentives Another moderating condition could be the incentive for the studymethod. SDT research tends to emphasize how extrinsic incentives may reduce enjoymentin some contexts (Deci et al. 1999). By contrast, Wolf and Smith (1995) found that students’performance and self-reported enjoyment and interest were higher when they were taking agraded exam instead of an inconsequential exam. In this study, it was predicted that incentiveswould improve the participants’ post-task rating of using retrieval as a study method.

Pre-task Intrinsic Motivation The final moderating variable of the relationship between studymethods and enjoyment to be investigated may include the participants’ pre-task enjoyment ofthe content to be learned. In a meta-analysis of studies, Cameron (2001) found that the effect ofincentives for performance was moderated by pre-existing enjoyment of the task. Morespecifically, incentives are more effective at increasing motivation if pre-existing enjoymentis low. When pre-existing enjoyment is higher, incentives may actually reduce or have no effecton enjoyment of a task (Cameron 2001). In the current study, it was predicted that incentivesand pre-task enjoyment would both have moderating effects on student performance.

The purpose of the current study was to investigate one aspect of why students may notpractice retrieval. Specifically, is retrieval perceived as enjoyable compared to re-reading?Within this effect, the moderating variables of incentives and pre-task enjoyment are includedin order to determine if some of these variables could mitigate the effect of retrieval onmotivation in the expected directions.

Method

Participants

The participants were 157 undergraduates assigned to participate in research studies by theeducational psychology subject pool at a large southwestern university. Students in this poolwere drawn from a set of undergraduate educational psychology courses that represent a widearray of subject matters. They tend to mirror the general characteristics of the undergraduatepopulation of the institution.

Educ Psychol Rev

Materials

A set of two short passages that were used in a previous study (Butler and Roediger 2008) wasused as content to be learned for this experiment. Written permission to use these materials wasobtained from the first author of the previous study. Passage one had 552 words and was aboutthe artist Salvador Dali. Passage two had 602 words and was about the war that occurredbetween the Israelis and several Arab nations in 1973. The participants were randomlyassigned to read one passage first and the other second.

Intrinsic Motivation Inventory Modified versions of the Intrinsic Motivation Inventory (Ryan1982; IMI) were used to investigate the participants’ pre- and post-task motivation concerningthe content and procedures in this study. This inventory has been used in several differentways, but it was originally designed as a task evaluation questionnaire (Ryan 1982). Theparticipants chose responses to items of this inventory in a 7-point Likert format. During theinitial assessment of pre-task motivation, a 13-item version was utilized. This inventorycontained seven items that assessed the participants’ pre-task enjoyment of the content(Cronbach’s alpha for this sample=0.92). For example, an item was “I think I will enjoyreading about an artist very much.” When the participant answered this item for the otherpassage, it was worded: “I think I will enjoy reading about an Arab-Israeli conflict very much.”The remaining six items assessed the participants’ pre-task feelings of competence concerningthe content (alpha for this sample=0.89). For example, one item was “I think I can do prettywell at this activity compared to other students.”

After completing the studying task, the participants’ post-task motivation was assessed withanother modified version of the IMI. Since this shorter version of the inventory has been foundto be psychometrically valid and reliable (McAuley et al. 1987), it was utilized to save timeand reduce the monotony of participation. This was also important since the participantscompleted this measure four separate times in this study. The shorter version had six of theaforementioned items from the pre-task version (modified for the task) that assessed theparticipants’ interest and enjoyment (alpha for this sample=0.87). The modification of thisshorter version involved orienting the questions toward the particular task. For example,instead of stating, “while I was working on this task,” item one stated either “while answeringthese questions” or “while reading these statements.” The reason for this modification was toreduce confusion, since the participants had completed several different tasks during the study.This shorter modified IMI was given twice during the study, after each of the study events. Asimilar scale has been used in several previous studies (Vallerand 1983; Vallerand and Reid1984; Plant and Ryan 1985). There was moderate agreement between the different forms of theIMI (k=0.47). Overall, the participants completed a version of the IMI four times. The first twowere oriented specifically toward the content to be read, while the third and fourth were moregeneric, referring to the task that was just completed.

Design

A design that incorporated both experimental manipulations and self-report survey instrumentswas chosen. Pre-task self-report questionnaires concerning intrinsic motivation were taken byall of the participants before experiencing any experimental manipulations. An extrinsicincentive was offered to some students as the primary between-groups variable, whereas studymethod was varied within groups. All of the participants answered self-report survey instru-ments after utilizing each study method.

Educ Psychol Rev

Between-Group Manipulation The between-group independent variable manipulated was theincentive offered for participation (see Fig. 1). One third of the participants were told that theycould leave early from the second session if they did well on the initial retrieval assessment.The second third of the participants were told similarly that they could leave early if theyperformed above a certain score on the delayed test during the second session.

The logic behind offering this type of incentive had several different components. Since thestudents were participating in a university subject pool requirement, they were already suffi-ciently motivated to be involved in the study. Not surprisingly to many who perform academicresearch, funds were limited for this project. This limitation was especially potent given howprevious studies have shown the unreliable nature of monetary or tangible rewards (Henderlongand Lepper 2002). To overcome this obstacle, the authors drew on previous research experiencein which undergraduate participants often dislike study participation and seek to complete it asquickly as possible. In light of previous research concerning the effect of regulatory focus onlearning (e.g., Grimm et al. 2008), the words spoken and displayed on the video were chosencarefully to say “leaving early” to emphasize the students’ seeking to obtain an incentive (toleave early) instead of avoiding the negative state of having to stay longer. The students knewbefore participation that the study included two full hours of participation; therefore, staying theentire time should be considered normal, although leaving early would certainly be desirable.

Within-Subject Manipulation In addition to the between-group experimental manipulation,there was a within-subject variable that was manipulated. The within-subject variable was thedifferent study methods by which the participants reviewed the material: retrieval in the formof answering short-answer questions (the retrieval condition) or re-studying.

When the participants were practicing retrieval, they answered 10 short-answer questionsconcerning the content of the passage. These questions ranged from factual content, such as“What was the name of Salvador Dali’s wife?” to more conceptual questions such as “What

Fig. 1 Study design

Educ Psychol Rev

was the main difference between Salvador Dali’s public and private lives?” After answeringeach question, the participants were given feedback by being shown the correct answer to thequestion by Qualtrics® computer software. Since the focus of this particular study was notsolely on the effects of retrieval, the amount of time spent on the questions was left up to theparticipant. However, the time was measured by the Qualtrics software. The participants in thisstudy spent an average of 16.5 s answering each short-answer question, but they only viewedthe feedback with the correct answer for an average of 1.7 s. In the follow-up session the nextweek, all of the participants answered the same 10 questions in a multiple-choice format.

When the participants were re-studying, they were shown the items and the answers from theretrieval questions in the form of a statement. This same procedure was used in a previous study(Butler and Roediger 2008). For example, one of the questions asks, “In what region of Spain wasSalvador Dali born?” and the answer is “Catalonia.” For this question, the participants who are re-studying this content read a sentence that said: “Salvador Dali was born in the town of Figueres inthe Catalonia region of Spain.” The timing was also not controlled for the re-studying procedure,but the participants spent an average of 8.6 s on each statement.

Procedures

The participants initially completed two modified versions of the Intrinsic MotivationInventory (Ryan 1982) to assess their pre-task feelings about the content of the passagesutilized in this study. Subsequently, all of the participants individually watched a shortinstructional video. The participants who were randomly selected to receive an incentive weretold that their level of performance would allow them to complete their study participation andleave earlier. The incentive video used was created using the website XTRAnormal and can befound at the following url: http://www.youtube.com/watch?v=81Epgy94mIs. The controlgroup viewed a similar animated video that simply encouraged them to remember to attendthe second session of the study. This second video can be found at this url: http://www.youtube.com/watch?v=nS1RGUhRCmo.

During the first learning phase, the participants read one of the passages, followed by aninterpolating letter-number sequencing task. During this task, the participants watched briefvideos in which letters and numbers appeared briefly. After each video, the participants wereasked to recall the letters and numbers from the video. This task was designed to take about5 min. The participants then studied the content from the passage through reading statements(the re-study condition) or answering short-answer questions with feedback (the retrievalcondition). At the end of the first phase, the participants answered the modified version ofthe IMI about the study method they had just completed. The procedure for the second phasewas the same, except for the content, interpolating task, and study method was switched(participants who re-studied in phase 1 used retrieval in phase 2, etc.). The second interpolatingtask was a portion of Raven’s Progressive Matrices Test. The participants then returned a weeklater to answer 10 multiple-choice questions concerning each of the two passages.

Results

Learning Performance Results A mixed between and within-subjects analysis of covariance(ANCOVA) was computed in which the independent between-subjects variables were theincentives offered for performance (incentive vs. no incentive) and self-reported pre-taskenjoyment (low vs. high, based on a median-split). The within-subject independent variable

Educ Psychol Rev

was the study method utilized. The covariate was the participants’ number of correct answerson the initial retrieval test, which was included to control for the differential effect it may havehad across the different study method conditions. The dependent variable involved was thenumber of questions answered correctly on the final test. The analysis showed that retrieval ledto better learning overall. Both the incentive and the covariate of recall performance also hadan effect on overall learning (see Table 1).

There was a marginal interaction between the incentive and pre-task enjoyment level inwhich the group that had higher pre-task enjoyment outperformed the group with lower pre-task enjoyment in the absence of an incentive (t(49)=2.15, p<0.05). Pre-task enjoyment didnot have the same effect among the participants who received the incentive. Post hoc t testsalso revealed that the group who received an incentive and had higher levels of pre-taskenjoyment outperformed the group who did not receive an incentive and had lower levels ofpre-task enjoyment (t(49)=2.15, p<0.05).

Post-task Enjoyment: Overall Results Another 2×2×2 mixed between and within repeated-measures ANCOVA was computed. The between-subject independent variables were theincentives offered for performance (incentive vs. no incentive) and self-reported pre-taskenjoyment (low vs. high, median-split). The within-subject independent variable was thedifferent study methods utilized. The covariate was the participants’ performance on the initialretrieval test.

Post-task Enjoyment: Within-Subjects Variables The within-subject dependent variable in-volved was the rating of the study method, in which students either re-read statements or usedretrieval by taking a multiple-choice quiz. For enjoyment, the repeated-measures ANCOVAfound a main effect of study method, suggesting that the participants rated the re-studying ofstatements as more enjoyable than answering short-answer questions (see Tables 2 and 3).

Although the incentives for performance improved learning overall, they did not have anymain effects on appraisals of enjoyment. There was not a two-way interaction between thebetween-group variable of pre-task predicted enjoyment and the within-group variable of study

Table 1 Mixed repeated-measures analysis of covariance for long-term memory performance

Source F df p value η2

Between subjects

Recall performance* 49.01* 1 <0.01* 0.24*

Incentive* 5.27* 1 0.02* 0.03*

Pre-task enjoyment 1.32 1 0.25 < 0.01

Incentive x pre-task enjoyment 2.96 1 0.09 0.02

Within-group error 152

Within-subjects

Study method* 11.90* 1 <0.01* 0.09*

Study method x recall performance 0.12 1 0.74 < 0.01

Study method x incentive 0.19 1 0.67 < 0.01

Study method x pre-task enjoyment 1.18 1 0.23 < 0.01

Study method x pre-task enjoyment x incentive 0.04 1 0.84 < 0.01

Within-group error 152

*p=0.05 is significant at alpha level

Educ Psychol Rev

method. Similarly, there was not a two-way interaction between the incentive and the within-group variable of study method on post-task enjoyment. However, a three-way interaction wasobserved across the groups of incentives and differing levels of pre-task enjoyment on therepeated measures of re-study and retrieval post-task enjoyment. As shown on Fig. 2, thethree-way interaction appears to be the result of how the discrepancy between the enjoyment ofthe different study methods was not equivalent across the conditions in the study. Controllingfor recall performance, separate repeated-measures ANCOVAs found that there was a differ-ence between the enjoyment ratings of the respective study methods (favoring re-studying) forthe individuals who were in either the high pre-task enjoyment/no incentive group,F(1, 27)=14.51, MSE=1.91, p=0.001, η2=0.35, or the low enjoyment/incentive group,F(1, 52)=13.04, MSE=1.32, p=0.001, η2=0.20.

Post-task Enjoyment: Between-Subjects Variables A post hoc univariate ANOVA showed amarginal interaction between pre-task enjoyment and incentives on the participants’ enjoymentof the retrieval procedure, F(1, 153)=3.13, MSE=87.94, p=0.07. Within this interaction, post

Table 2 Mixed repeated-measures analysis of covariance for post-task enjoyment

Source F df p value η2

Between subjects

Recall performance* 22.32* 1 <0.01* 0.13*

Incentive 0.07 1 0.78 <0.01

Pre-task enjoyment* 47.90* 1 <0.01* 0.28*

Incentive x pre-task enjoyment 0.16 1 0.68 <0.01

Within-group error 152

Within subjects

Study method* 27.32* 1 <0.01* 0.15*

Study method x recall performance* 9.98* 1 0.02* 0.06*

Study method x incentive 0.86 1 0.35 <0.01

Study method x pre-task enjoyment 3.34 1 0.07 0.02

Study method x pre-task enjoyment x incentive* 9.91* 1 <0.01* 0.06*

Within-group error 152

*p=0.05 is significant at alpha level

Table 3 Descriptive statistics for groups

Incentive group x enjoyment level Post-task enjoyment Performance

Re-study Retrieval Re-study Retrieval

n m sd m sd m sd m sd

No incentive/low pre-task enjoyment 22 2.5 1.4 2.2 1.2 6.7 2.5 8.1 1.8

No incentive/high pre-task enjoyment 29 4.6 1.4 3.0 1.2 7.8 1.7 8.8 1.4

Incentive/low pre-task enjoyment 54 2.9 1.3 2.0 1.0 7.7 1.7 8.9 1.4

Incentive/high pre-task enjoyment 52 4.9 1.5 3.5 1.6 8.1 1.8 9.1 1.3

Educ Psychol Rev

hoc t tests revealed that receiving an incentive marginally improved the post-task enjoymentrating of participants who reported higher pre-task enjoyment, t(79)=1.75, p=0.08. Althoughthese effects are marginal, it was predicted that recall performance also had an effect on thepost-task enjoyment of retrieval. These results are discussed below in an additional analysis.The incentive did not have the same effect among the participants who were below themedian on the pre-task enjoyment variable, t(74)=0.7, p=0.48. Another post hoc univariateANOVA also showed an interaction between pre-task enjoyment and incentive conditionson the rating of the re-studying procedure, F(1, 153)=5.22, MSE=161.29, p<0.05. Theincentive reduced the participants’ post-task enjoyment rating of the re-study procedureamong those who reported higher levels of pre-task enjoyment, t(79)=2.10, p<0.05,whereas the incentive did not have an effect among the participants with lower pre-taskenjoyment ratings, t(74)=1.1, p=0.25.

Although the results reported in Table 2 suggest that performance on the initial memorytest had a strong relationship with post-task enjoyment of retrieval, the type of analysisutilized did not probe the possible interaction effect between the different experimentalconditions and the initial recall performance. To investigate the possibility of interactioneffects, a hierarchical linear regression was computed. In the first step of the regression, thevariables of pre-task enjoyment, incentive, and the interaction between the pre-task and theincentive were entered. In the second step, the performance on the retrieval task was entered,as well as the interactions between retrieval task performance and the other variables in themodel (see Table 4). The dependent variable was the post-task enjoyment rating followingthe retrieval task.

This analysis revealed that including the retrieval performance improved the predictionof the variance in post-task enjoyment (ΔR2=0.12, F (4, 149)=7.47, MSE=1.37, p<0.001).Controlling for all of the variables in the model, post-task enjoyment rating was bestpredicted by performance on the retrieval task, β=0.38, followed by the pre-task enjoymentlevel of those who received the incentive, β=0.23, and pre-task enjoyment rating amongthose who did not receive an incentive (β=0.20). Since the interactions between thesepredictors were not significant, it appears that the respective effects of the pre-task motiva-tional variables and retrieval performance were somewhat parallel in how they affectedpost-task enjoyment.

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

2.87

2.48

3.85

4.55

1.992.18

3.59

2.98

Re-Study

Retrieval

Incentive IncentiveNo No Incentive

Low Pre-task Enjoyment High Pre-task Enjoyment

Fig. 2 Post-task enjoyment across conditions

Educ Psychol Rev

Discussion

Similar to many previous studies, long-term memory was enhanced when students usedretrieval instead of re-reading as the learning strategy (see Roediger and Butler 2011). Incontrast to the enhanced memory associated with using retrieval, these results showed thatstudents enjoyed re-reading as a study method more than they did retrieval. Although it seemscounterintuitive, this result is consistent with research in self-determination theory of motiva-tion (Deci and Ryan 2000). Given this evidence, it is not surprising that previous studies havefound that students tend to re-read more than they use retrieval while studying.

Although retrieval affected enjoyment negatively overall, the current study was alsoconcerned with variables that might moderate this relationship. In coherence with SDT, itwas predicted that there would be a relationship between feedback encountered and the extentto which participants enjoyed the task. When the participants were simply re-studying thestatements from the passages, no possibility of feedback was involved. As opposed to re-studying, using retrieval required answering questions and the possibility of receiving positiveor negative feedback. When using retrieval, the participants who performed better on theimmediate retrieval test also tended to enjoy using retrieval more than the other participants,supporting the previous results from SDT research. Similar to the results of Vallerand and Reid(1984), the study method without feedback (re-reading) was preferred by the participants eventhough using retrieval led to better learning.

More specifically, it seems possible that motivational variables might also have an effect on post-study ratings. Pre-task self-reported enjoyment wasmore closely related to post-task ratings after there-reading studymethod as opposed to the retrieval studymethod. It seems logical that these pre-taskratings might begin to matter less during retrieval because participants’ feelings of enjoyment werelikely also affected by the feedback they were being given during the retrieval task.

The incentives that were offered had effects across some of the conditions in the study.When theincentive was administered to participants with higher ratings of pre-task enjoyment, they endorsedmarginally higher post-task enjoyment ratings of the retrieval study method. Conversely, the sameparticipants with higher pre-task enjoyment ratings had lower post-task enjoyment ratings for the re-

Table 4 The effect of recall performance on post-task retrieval enjoyment rating

Predictors β R2 ΔR2 F change

Step 1: pre-task motivation variables 0.29 0.29 20.45*

Pre-task enjoyment* 0.26*

Incentive 0.07

Incentive x pre-task enjoyment* 0.30*

Step 2: the effect of recall performance 0.40 0.12 7.46*

Pre-task enjoyment 0.20

Incentive 0.04

Incentive x pre-task enjoyment* 0.23*

Retrieval performance* 0.38*

Retrieval performance x pre-task enjoyment −0.08Retrieval performance x incentive −0.04Retrieval perf. x pre-task enjoyment x incentive 0.17

Dependent variable: post-task enjoyment rating

*α=0.05 is a significant predictor

Educ Psychol Rev

readingmethodwhentheyweregivenanincentive(seeFig.2).Thisresultmayseemcontradictory,butit is in accordance with the previously mentioned findings concerning motivation. Cameron (2001)suggested that when a task is not itself enjoyable (such as retrieval was in this study), incentives aremore effective at improvingmotivation. Conversely, when an activity is more enjoyable (such as re-reading for the higher pre-task enjoyment participants), incentives may reducemotivation.

Limitations

There were several limitations encountered in this study. One of them involved the amount oftime the participants spent on the respective study methods. Not surprisingly, the participantsspent more time on the retrieval task than they did on the re-studying task. Although previousstudies have chosen to procedurally control this particular issue, it was deliberately uncontrolledin this study due to the possibility that the motivational variables might have had an impact onthe time spent using these study methods. It is entirely possible that this time difference mayhave contributed to the advantage that using retrieval had in overall learning. Since previousresearch has shown very consistently the positive effect of retrieval on learning (see Roedigerand Butler 2011 for a review), this was not the main focus of this particular study.

Another limitation encountered was regarding the relationship between items retrievedcorrectly on the initial test and post-task enjoyment rating. Although it appears from the datathat there is a linear relationship between these two values, there is no way to know if therelationship is causal. It is possible that characteristics of the participants involved led to thisrelationship instead of one of these variables directly causing the other. Based on thislimitation, future research may be able to experimentally manipulate this aspect of usingretrieval and assess the effect it has on motivation.

A final limitation in the study was the compulsory nature of using the respective studymethods in the current experiment. The participants were not given a choice concerning whichstudy method they would prefer to utilize. This issue may be important, because it is entirelypossible that enjoyment of using retrieval may change in conditions of free choice. For example,the popularity of board games like “Trivial Pursuit” and “Scene it?” as well as televisionprograms such as “Jeopardy” and “Who Wants to Be a Millionaire?” suggests that retrievinginformation frommemorymay be enjoyable in some conditions. Similarly, it is possible that thelinear relationship between feedback and enjoyment could also change.

Implications for Future Research

Based on these results, it appears that one possible reason why students do not use retrieval isthat it is not as enjoyable as other methods, even when controlling for moderating variables.The current paper was not designed to implicate reasons why students did not enjoy retrievalwhile studying. It is certainly possible that this lack of enjoyment could be related to distastefor current standardized and classroom-based testing procedures often utilized in schools(Segool et al. 2013). Therefore, future research should investigate if negative experiences withtesting are related to students preferring not to use retrieval when studying. In addition, therecould be interventions that involve retrieval without the cost of reducing the enjoymentexperienced by the student. Another possibility would be to find innovative methods to engagestudents in using retrieval despite their disdain for the study method. For example, thecomputer basically led the participants through the study methods. Future studies might assessthe effect of giving participants a choice of whether or not they wanted to use retrieval.

Unfortunately, the current study results are not congruent with all of the available studiesconcerning students’ enjoyment of retrieval. In a meta-analysis, Bangert-Drowns et al. (1991)

Educ Psychol Rev

suggested that most students reported that frequent quizzing in the classroom was avaluable learning activity. There could be many different caveats that could explain thisdiscrepancy. It seems possible that a student’s perspective may moderate their enjoymentof using retrieval. A student may view it more positively retrospectively after retrievalhas been used repeatedly. It could also be that retrieval may not be seen as negative aftertaking the final exam, especially since it is an effective learning method. Future studiesshould investigate this relationship in closer detail.

In this study, the study method that was the most effective (retrieval) was not the same asstudy method that was preferred the most by the participants (re-reading). Therefore, it seemsworth pursuing that motivation may be an important component of this learning technique.Unfortunately, researchers often do not reach across different areas of inquiry. Future studiesinvestigating the interactions between these different constructs may increase theoretical andpractical understanding of the effect of retrieval.

Appendixes

Appendix A: Questions

Passage 1: Short-Answer Questions

1. Salvador Dali created some of the most widely recognized images to come out of whatartistic movement?

2. In what region of Spain was Salvador Dalí born?3. For what action did Salvador Dalí praise Francisco Franco?4. What was the name of Salvador Dalí's wife?5. What image did Salvador Dali portray in The Persistence of Memory?6. What was a major influence on Salvador Dalí's eccentric personality?7. What was the main difference between Salvador Dali's public and private political views?8. Why might Salvador Dalí have attempted to commit suicide?9. Dali praised the work of Federico Garcia Lorca even though his works were…10. What was the name of Dali’s brother who died of Meningitis?

Passage 1: Multiple-Choice Questions

1. Salvador Dali created some of the most widely recognized images to come out of whatartistic movement?• The Surrealist movement• The Futurist movement• The Fauvist movement• The Art Nouveau movement

2. In what region of Spain was Salvador Dalí born?• He was born in the region of Catalonia• He was born in the region of Andalucia• He was born in the region of Aragon• He was born in the region of Cantabria

Educ Psychol Rev

3. For what action did Salvador Dalí praise Francisco Franco?• Signing death warrants for political prisoners• Imposing martial law after capturing Madrid• Proclaiming Spain a monarchy, but not designating a monarch.• Banning the usage of any language other than Spanish

4. What was the name of Salvador Dalí’s wife?• Her name was Gala• Her name was Aracelia• Her name was Elisa• Her name was Inez

5. What image did Salvador Dali portray in The Persistence of Memory?• The image of the soft, melting pocket watch• The image of a dark cluster of ants• The image of rocks morphing into each other• The image of a broken phone over a plate of eggs

6. What was a major influence on Salvador Dalí’s eccentric personality?• His unusual childhood• His peers at the Art Academy of Madrid• His relationship with fellow artist Federico García Lorca• His obsession with Freudian theories

7. What was the main difference between Salvador Dali’s public and private politicalviews?• Publically he supported fascism, but privately he was more open-minded.• Publically he denounced Franco, but privately he encouraged him.• Publically he posed as a monarchist, but privately he liked communism.• Publically he appeared uninvolved in politics, but he was active privately.

8. Why might Salvador Dalí have attempted to commit suicide?• He was devastated by his wife’s death• He was upset about the public’s disapproval of his work• His only son died in an automobile accident• He was very poor and destitute

9. Dali praised the work of Federico Garcia Lorca even though his works were…• Banned• Rigid and deterministic• Facist• Jewish

10. What was the name of Dali’s brother who died of Meningitis?• Salvador• Figueres• Felipe• Francisco

Educ Psychol Rev

Passage 2: Short-Answer Questions

1. Before the 1973 Arab-Israeli War, the broader Arab-Israeli conflict had been going onsince what year?

2. What did the Egyptian and Syrian forces use to breach Israeli defenses during the start ofthe 1973 Arab-Israeli War?

3. Launched on October 15th during the 1973 Arab-Israeli War, Operation “Abiray-Lev”translates to what in English?

4. What was the name of the diplomat who brokered a disengagement agreement betweenIsrael and Syria at the end of the 1973 Arab-Israeli War?

5. The 1973 Arab-Israeli War was incited in part because of what two disputedregions?

6. As a result of Yom Kippur and Ramadan, what were both sides of the conflict doingduring the 1973 Arab-Israeli War?

7. What change in strategy by the Egyptians led to a turning point in the 1973 Arab-IsraeliWar?

8. Why were the early events of the 1973 Arab-Israeli War critical to the later normalizationof relations between Egypt and Israel?

9. What international organization passed a resolution that called for a cease-fire betweenthe two nations?

10. What was unique about the Camp David Accords?

Passage 2: Multiple-Choice Questions

1. Before the 1973 Arab-Israeli War, the broader Arab-Israeli conflict had been going onsince what year?• 1948• 1928• 1901• 1963

2. What did the Egyptian and Syrian forces use to breach Israeli defenses during the start ofthe 1973 Arab-Israeli War?• Water Cannons• Mechanized infantry• Tanks• Cruise missiles

3. Launched on October 15th during the 1973 Arab-Israeli War, Operation “Abiray-Lev”translates to what in English?• Operation “Stouthearted Men”• Operation “Protect the Heart”• Operation “Brave Lion”• Operation “Lightening Strike”

4. What was the name of the diplomat who brokered a disengagement agreement betweenIsrael and Syria at the end of the 1973 Arab-Israeli War?• Henry Kissinger• John Foster Dulles• Kurt Waldheim• Margaret Thatcher

Educ Psychol Rev

5. The 1973 Arab-Israeli War was incited in part because of what two disputedregions?• The Golan Heights and the Sinai region• The West Bank and the Gaza Strip• Haifa and Elat• Beth Shean and Hadera

6. As a result of Yom Kippur and Ramadan, what were both sides of the conflict doingduring the 1973 Arab-Israeli War?• Fasting during the day• Praying three times a day• Refraining from eating fish• Resting during the day

7. What change in strategy by the Egyptians led to a turning point in the 1973 Arab-Israeli War?• Instead of continuing to sit back defensively, the Egyptian army went on an all-out attack• Instead of continuing to occupy Israeli land, the Egyptian army pulled out abruptly• Instead of continuing to fight on foot, the Egyptian forces used artillery and planes• Instead of attacking Israel from the South, the Egyptian army attacked by water from the

8. Why were the early events of the 1973 Arab-Israeli War critical to the later normalizationof relations between Egypt and Israel?• The early victories helped Egyptian moral and pride, allowing them to recognize

Israel• The early victories showed that Israel had a superior military and diplomacy was the

only option.• The co-occurrence of Muslim and Jewish holidays helped both sides realize they had

much in common.• The early victories left Israel’s defenses in ruins, making diplomacy their only option

9. What international organization passed a resolution that called for a cease-fire betweenthe two nations?• The United Nations• NATO• The Arab Alliance• Abiray-Lev

10. What does the passage say was historically unique about the Camp David Accords?• An Arab country recognizing the state of Israel• The location of the negotiation• Vindication for Egypt• Syria exchanged prisoners

Appendix B: Modified Version of the IMI Examples

Pre-task Modified IMI for Passage 1

Directions: You will be reading one passage about an artists’ life. For each of the followingstatements, please indicate how true it is for you:

Not at all true Somewhat True Very True1 2 3 4 5 6 7

1. I think I will enjoy reading a passage about an artist very much.

Educ Psychol Rev

2. Reading and learning about artists can be fun to do.3. I think reading about an artist would be a boring activity (R).4. Reading about an artist will probably not hold my attention at all.5. Reading about an artist’s life might be interesting.6. I think reading about an artists’ life will be quite enjoyable.7. Before reading the passage about an artist, I am thinking I could enjoy it.

Pre-task IMI Passage 2

Directions: You will also be reading a passage about the six days war, which occurred in theMiddle East between the Israelis and several Arab nations. Considering that content area,please indicate how true it is for you:

1. I think I will enjoy reading about this historic event.2. Reading and learning about historical conflicts such as this one can be fun to do.3. I think reading and learning about the Six Days war will be boring.4. Reading about the Six Days War will probably not hold my attention at all.5. Reading about the Six Days War might be interesting.6. I think learning about these historically significant events will be quite enjoyable.7. Before reading this passages about the Six Days War, I am thinking I could enjoy it.

Modified IMI Following Retrieval

Directions: For each of the following statements, please indicate how true it is for you,considering the questions you just answered:

1. While answering these questions, I was thinking about how much I enjoyed it.2. These questions did not hold my attention at all. (R)3. Reading and answering these questions was fun.4. I would describe these questions as very interesting.5. I enjoyed answering these questions very much.6. These questions were fun to answer.

Modified IMI Following Re-Studying

Directions: For each of the following statements, please indicate how true it is for you,considering the statements you just read:

1. While I was reading this material, I was thinking about how much I enjoyed it.2. This reading material did not hold my attention at all. (R)3. Studying this material was fun.4. I would describe this material as very interesting.5. I enjoyed reading this material very much.6. This material was fun to read.

Educ Psychol Rev

References

Bangert-Drowns, R. L., Kulik, J. A., & Kulik, C. L. C. (1991). Effects of frequent classroom testing. The Journalof Educational Research, 85, 89–99.

Butler, A., & Roediger, H. (2008). Feedback enhances the positive effects and reduces the negative effects ofmultiple-choice testing. Memory & Cognition, 36(3), 604.

Cameron, J. (2001). Negative effects of reward on intrinsic motivation‚ a limited phenomenon: comment onDeci, Koestner, and Ryan (2001). Review of Educational Research, 71(1), 29–42.

Deci, E. L., & Cascio, W. F. (1972). Changes in intrinsic motivation as a function of negative feedback andthreats. Boston, MA: Paper presented a meeting of the Eastern Psychological Association. Retrieved fromhttp://eric.ed.gov/?id=ED063558. Accessed 5 June 2014.

Deci, E. L., & Ryan, R. M. (1980). The empirical exploration of intrinsic motivational processes. Advances inExperimental Social Psychology, 13, 39–80.

Deci, E. L., & Ryan, R. M. (2000). The” what” and” why” of goal pursuits: human needs and the self-determination of behavior. Psychological Inquiry, 11(4), 227–268.

Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects ofextrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6), 627.

Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’learning with effective learning techniques: promising directions from cognitive and educational psychology.Psychological Science in the Public Interest, 14(1), 4–58. doi:10.1177/152910061245326.

Einstein, G. O., Mullet, H. G., & Harrison, T. L. (2012). The testing effect: illustrating a fundamental concept andchanging study strategies. Teaching of Psychology, 39(3), 190–193.

Grimm, L. R., Markman, A. B., Todd Maddox, W., & Baldwin, G. C. (2008). Differential effects of regulatory fiton category learning. Journal of Experimental Social Psychology, 44(3), 920–927.

Hartwig, M., & Dunlosky, J. (2012). Study strategies of college students: are self-testing and scheduling relatedto achievement? Psychonomic Bulletin & Review, 19(1), 126–134.

Henderlong, J., & Lepper, M. (2002). The effects of praise on children’s intrinsic motivation: a review andsynthesis. Psychological Bulletin, 128(5), 774–795.

Karpicke, J. D. (2009). Metacognitive control and strategy selection: deciding to practice retrieval duringlearning. Journal of Experimental Psychology General, 138(4), 469–486.

Karpicke, J., Butler, A., & Roediger, H. (2009). Metacognitive strategies in student learning: do students practiseretrieval when they study on their own?

Kornell, N., Bjork, R. A., & Garcia, M. A. (2011). Why tests appear to prevent forgetting: a distribution-basedbifurcation model. Journal of Memory and Language, 65(2), 85–97.

McAuley, E., Duncan, T., & Tammen, V. V. (1987). Psychometric properties of the Intrinsic MotivationInventory in a competitive sport setting: a confirmatory factor analysis. Research Quarterly for Exerciseand Sport, 60, 48–58.

McCabe, J. (2010). Metacognitive awareness of learning strategies in undergraduates. Memory & Cognition,39(3), 462–476.

Pashler, H., Bain, P., Bottge, B., Graesser, A., Koedinger, K., McDaniel, M.et al. (2007). Organizing instructionand study to improve student learning. IES Practice Guide. NCER 2007-2004. National Center forEducation Research, 63.

Phelps, R. P. (2012). The effect of testing on student achievement, 1910–2010. International Journal of Testing,12(1), 21–43.

Plant, R. W., & Ryan, M. (1985). Intrinsic motivation and the effects of self-consciousness, self-awareness, andego-involvement: an investigation of internally controlling styles. Journal of Personality, 53(3), 435–449.

Roediger, H. L., & Butler, A. (2011). The critical role of retrieval practice in long-term retention. Trends inCognitive Sciences, 15(1), 20–27.

Ryan, R. M. (1982). Control and information in the intrapersonal sphere: an extension of cognitive evaluationtheory. Journal of Personality and Social Psychology, 43(3), 450–461.

Segool, N. K., Carlson, J. S., & von der Embse, N. (2013). Heightened test anxiety among young children:elementary school students’ anxious responses to high‐stakes testing. Psychology in the Schools, 50(5).

Vallerand, R. (1983). The effect of different amounts of verbal feedback on intrinsic motivation of male hockeyplayers. Journal of Sports Psychology, 5, 100–107.

Vallerand, R., & Reid, G. (1984). On the causal effects of perceived competence on intrinsic motivation: a test ofcognitive evaluation theory. Journal of Sports Psychology, 6(1), 94–102.

Wolf, L. F., & Smith, J. K. (1995). The consequence of consequence: motivation, anxiety, and test performance.Applied Measurement in Education, 8(3), 227–242.

Educ Psychol Rev