REGULAR REVISION STUPID! - THE RETENTION OF KNOWLEDGE

AFTER THE END OF MODULE DELIVERY

Paulina K. KlichDr. Piotr M. Jaworski

School of Accounting Economics and Statistics The Business School

p.jaworski@napier.ac.uk

Fifth International Developmentsin Economics Education (DEE) Conference

Wales Millennium Centre, Cardiff, UK9-10 September 2009

2

Introduction

Introduction• The study, which was financed by the Higher Education

Academy Economics Network within the framework of 2008/09 Mini Project Programme.

• It aimed to measure effectiveness of IT-assisted continuous assessment regime used for Economics 1 module delivered to four cohorts of first year undergraduates’ students of NUBS in two academic years 2006/07 and 2007/08.

• The study involved retesting a sample of 60 participants from all cohorts with the same tests and measuring the retention of knowledge delivered in the course of the module: – multiple regression analysis confronted this retention level with

the amount of preparation devoted to particular tests in the first week of material presentation and later.

– also relation between the preparation and the initial tests’ results was researched. 3

4

The Module: Economics 1

• First year undergraduate one-trimester module presenting basic economic concepts of micro- and macroeconomics consisting of ten units of material

• Addressed to Business School students as an introduction to more advanced Economics 2 module and to all other interested, who would like to gain elementary economic knowledge.

• Average semester enrolment: 200 students. • Every unit involved 2h lecture and 1 hour tutorial

5

Assessment Regime

• The assessment was divided into two parts: an essay (40% of final mark) and two IT-assisted tests constituting summative assessment (60% of final mark).

• Another part of the assessment regime were weekly 10-question quizzes, which served as a “gateway” to lecture notes.

6

The Tests and the Quizzes

• 2006/07– Every test was worth 30% of the final mark;– The questions were drawn out of 15 question

from 10 sets previously used for quizzes;– The quizzes did not contribute towards the final.

• 2007/08– Every test was worth 25% of the final mark;

• 0.75% for 20 questions from quizzes;• 1% for 10 new true/false questions.

– 10% for all quizzes completion within one week of the material presentation;

2006/07 vs. 2007/08 Tests and Quizzes

Quiz Test2006/07 2007/08 2006/07 2007/08

No of questions 10 10 30 30From quizzes N/a N/a 30 for 1% 20 for 0.75%

new N/a N/a 0 10 for 1% each

Total value towards the final mark

0% Max* 1%** 30% 25%

Drawn out of 15 20 75(15 per unit)

100(20 per unit)

Pass mark 5 questions 5 questions None None

No of attempts Unlimited Unlimited 2 2

7

The Results

• Passing rate: out of 260 and 140 participants respectively in the first and second trimesters of 2007/08

89.62% and 91.43% passed. • This was a substantial increase compared

to 83.70% and 83.79% in 2006/07. • On the other hand, the average final mark decreased from

64.61% and 61.13% for trim. 1 and 2 of 2006/07 to 56.70% and 54.10% in 2007/08.

• Also standard deviation of the final marks decreased.

8

The Research

• The structure introduced seems to have been effective: – the amount of revision effort was fairly considerable;– systematic work was induced and strengthened by the usage of

the simple economic concepts of utility and discounting.

• Open issue: whether the expected formative character of the assessment regime materialised or we just landed with another form of summative assessment.

9

10

Research Stages

• First stage: – retesting sample of students with exactly the same tests

which they had done in the course of the module. – counting the number of all quizzes completed in the first

and following weeks for every student.

• Second stage:– correlate different parts of the assessment with the initial

tests results and the results for knowledge retention.

The Sample

• Invitation sent to 100 students resulted in 10% response: only 10 students retested in the same supervised environment and 50 voluntary completion of the retests from home after the invitation sent to all 685 students.

• The sample mean of final mark was 69.41 with standard deviation of 12.21 and median 70.50 (for the population 64.98 and 12.28 respectively).

• The sample dispersion similar to the population but the mean and median around 5 points higher, what suggests that the sample included better students.

• The 60 sample out of the population of 685 gave us a confidence interval of a little bit more than 12% (95% level)

11

The Data

• Independent variables:– Numbers of quiz completion done within the first week and later;

separately for material covered by test 1 and test 2 (NQF1, NQL1, NQF2, NQL2).

– the fact whether the student was under the initial (0) or adjusted regime (1) of assessment; (dummy variable NAS).

– Essay mark (ESS), – Number of trimesters from the end of module delivery (NTR),

• Independent variable:– for the initial tests results regression: tests results (TST1 and TST2)– for the results for knowledge retention regression: retention of

knowledge (RTN1 and RTN2) which was constructed as the retest’ score (HEA1 and HEA2) divided by the results of the original test’s scores (TST1 and TST2) 12

Initial Tests Results Regression

TST1 TST2Coefficient Standard

Error

t stat Coefficient Standard

Error

t stat

NQF 5.30 0.34 15.51 6.57 0.34 19.35NQL 0.13 0.01 15.30 0.13 0.01 13.76NAS 0.10 0.02 4.39 0.06 0.02 2.88CON 13.91 0.28 49.22 14.17 0.27 51.72

R square 51.33% R square 54.12%Standard error of estimate

4.11 Standard error of estimate

4.25

Degrees of freedom 681 Degrees of freedom 681F statistic 239.43 F statistic 267.8013

After using several sets of dependent variables it turned out that the most meaningful results are generated by using NQF, NQL and NAS as dependent variables in the multiple linear regression with constant

(CON).

Knowledge Retention Regression

14

RTN1 RTN2Coefficient Standard

Error

t stat Coefficient Standard Error

t stat

NQF 18.28 3.05 5.99 16.86 2.43 6.92NQL 0.01 0.25 0.04 1.33 0.39 3.43NTR 1.46 0.42 3.43 -0.08 0.22 0.34

R square 74.10% R square 79.43%Standard error of estimate

38.40 Standard error of estimate

30.43

Degrees of freedom 57 Degrees of freedom 51F statistic 54.37 F statistic 65.65

The most meaningful results we obtained for the regressions without constant (CON) involving NQF, NQL and NTR.

The results confirmed our belief that the regular and continuous work on the material was crucial for the level of final test mark:

– the number of quiz completion within first week of material presentation had the highest influence on the final mark in the case of both tests with very high significance and high R square.

– Although the two other dependent variables, the number of quiz completion after the first week and the influence of the new system of assessment, have also been statistically significant, their positive influence has been much lower, if almost non-existent.

15

Initial Tests Results Regression

The number of quizzes completed in the first week turned out to be the most important factor:

– with relatively high significance it influenced the retention of knowledge for both tests with the R squares being even higher than for the “final mark” regressions.

– This influence was also considerably higher than the “final mark” influence.

– Completing quizzes later was insignificant and had practically no influence on test 1 but a moderately statistically significant modest influence on test 2.

16

Knowledge Retention Regression

• In case of the NTR the results were reversed: statistically significant for test 1 and insignificant for test 2.

• The time passing increased the retention in a similar degree that practising after the first week of module delivery.

• In the case of the test 2 it was slightly negatively related to knowledge retention.

17

Knowledge Retention Regression

• Test 2. results were a surprise; possible explanation:– difference between materials covered by tests 1. (micro-) and 2.

(macroeconomics);– micro aspects of the economy under constant revision in the

modules following Economics 1, – macroeconomic issues are not so often recalled in the course of

later studies. – Therefore, the micro revision in following modules serves as a

similar tool of revision as the quiz completion after the first week of module completion in the case of macroeconomics.

18

Knowledge Retention Regression

• The results confirm the fact that revision within the first week after material presentation is vital in the process learning: in the time of module delivery it increases tests results and after it enhances the knowledge retention.

• We hope that we managed to create a regime of assessment for the students not to the students which according to Biggs (1998) represents the distinction between formative and summative assessment.

19

Conclusions

• Furthermore, the assessment regime led to expected by us action, which was increased continuous and systematic revision which also constitutes a condition for formative assessment (Sadler, 1989).

• This, we hope, induced self assessment skills which students should learn through their undergraduate degrees that, according to Boud (2000) may foster lifelong learning, which is our lecturers’ another important goal.

20

Conclusions

21

Questions and Comments are Welcomed!

22

REFERENCES• Biggs, J. (1999). Teaching for quality learning at university. Buckingham: Open University Press.

• Biggs, John (1998). Assessment and Classroom Learning: a role for summative assessment?. Assessment in Education: Principles, Policy & Practice, 5 (1), 103-110.

• Boud, David (2000). Sustainable Assessment: rethinking assessment for the learning society. Studies in Continuing Education, 22 (2), 151-167.

• Brown, S., & Knight. P. (1994). Assessing learners in higher education. London: Kogan Page.

• Conway, M.A., Cohen, G. & Stanhope, N. (1992). Why is it that university grades do not predict very long-term retention? Journal of Experimental Psychology, 121(3), 382-384.

• Deci E. L., & Ryan R. M. (1985). Intrinsic motivation and self-determination in human behaviour. New York: Plenum.

• Deci E L. & Ryan R. M. (2000). Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemporary Educational Psychology 25, 54-67.

• Entwistle N. (2000). Promoting deep learning through teaching and assessment: conceptual frameworks and educational contexts. Paper presented at TLRP Conference, Leicester, November. http://www.tlrp.org/pub/acadpub/Entwistle2000.pdf

• Gibbs, G. & Simpson, C. (2004). Does your assessment support your students’ learning? Journal of Teaching and Learning in Higher Education, 1, 3–31.

• Jaworski P. (2009) Let Us Teach the Way We Preach: Using Economic Concepts in IT-Assisted Continuous Assessment Scheme for Introductory Economics Module, Manuscript submitted for publication in Advances in Business Education.

23

REFERENCES• Jaworski P. (2008) Enhancing student learning through the introduction of a WebCT-based

continuous assessment scheme. Educational Research Paper Series no. 6, School of Accounting, Economics & Statistics, Napier University Business School

• Jeppson K., (2008) Increasing student examination rate by use of weekly home assignments. European Workshop on Microelectronics Education

• Kniveton, Bromley H. (1996). Student Perceptions of Assessment Methods. Assessment & Evaluation in Higher Education, 21 (3), 229-237.

• Marton F. & Saljo R. (1976). On qualitative differences in learning, outcome and process, British Journal of Educational Psychology 46, 4-11.

• Miller, C.M.I. & Parlett, M. (1974). Up to the mark: A study of the examination game. Guildford: Society for Research into Higher Education.

• Ramsden, P. (1992). Learning to teach in higher education. London: Routledge.

• Snyder, B.R. (1971). The Hidden Curriculum. Cambridge, MA: MIT Press.

• Sadler, D.R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18. 119-144.

• Sambell, K. & McDowell, L. (1998). The construction of the hidden curriculum: messages and meanings in the assessment of student learning. Assessment and Evaluation in Higher Education, 23(4), 391-402.