using examsoft to facilitate active retrieval and promote student success

of 35 /35
1

Author: examsoft

Post on 08-Apr-2017

43 views

Category:

Education


0 download

Embed Size (px)

TRANSCRIPT

PowerPoint Presentation

#

Using ExamSoft to Facilitate Active Retrieval and Promote Student SuccessDavid J. Caldwell, PharmD, AAHIVPDirector of Professional Affairs, ULM School of Pharmacy

Elizabeth M. Lafitte, PharmD, BCPSAssistant Professor Clinical Sciences, ULM School of Pharmacy

#

ObjectivesDescribe active retrieval theory.Elucidate how active retrieval can be used as a formative assessment method.Describe the implementation of an active retrieval exercise to promote student success using ExamSoft.Demonstrate the value of ExamSoft in facilitating an active retrieval exercise.

#

Describe the theory of active retrieval as a formative assessment method. Summarize the evidence supporting active retrieval to increase knowledge retention. Demonstrate the value of ExamSoft in facilitating an active retrieval exercise. Describe the implementation of an active retrieval process using ExamSoft to promote student success.

Lets do a Kahoot!

#

Learning is often thought to correlate with encoding new information into memory

Most research has focused on how to ENCODE things, when the ability to USE information and APPLY relies on RETRIEVAL, not encoding

2015 Karpicke and Nunes Retrieval Based Learning Review

#

Learning is often thought to correlate with encoding new information into memory

Most research has focused on how to ENCODE things, when the ability to USE information and APPLY relies on RETRIEVAL, not encoding

2015 Karpicke and Nunes Retrieval Based Learning Review

#

Learning is often thought to correlate with encoding new information into memory

Most research has focused on how to ENCODE things, when the ability to USE information and APPLY relies on RETRIEVAL, not encoding

2015 Karpicke and Nunes Retrieval Based Learning Review

#

Learning is often thought to correlate with encoding new information into memory

Most research has focused on how to ENCODE things, when the ability to USE information and APPLY relies on RETRIEVAL, not encoding

2015 Karpicke and Nunes Retrieval Based Learning Review

Lets do a Kahoot!

#

Learning is often thought to correlate with encoding new information into memory

Most research has focused on how to ENCODE things, when the ability to USE information and APPLY relies on RETRIEVAL, not encoding

2015 Karpicke and Nunes Retrieval Based Learning Review

Myth #1The key to learning is to study, study, study.

#

Learning is often thought to correlate with encoding new information into memory

Most research has focused on how to ENCODE things, when the ability to USE information and APPLY relies on RETRIEVAL, not encoding

2015 Karpicke and Nunes Retrieval Based Learning Review

Active retrievalActive retrieval theoryRetrieval enhances learningEach act of retrieval improves the ability to re-retrieve knowledgeWhat is active retrieval?The active reconstruction of knowledge based on context and retrieval cues

#

Humans reconstruct knowledge rather than reproducing exact copies

Re-retrieval: the act of retrieving tells our minds that the knowledge is MORE important because we may need it again in the future

Reading, listening, traditional studying, etc. are semi-passive means of putting information into our minds. We tend to practice this repeatedly in our typical study sessions. When you take a quiz or test (or even try to apply the information in real life) are you putting in or getting out? When you think about it, it makes sense that we should practice getting the information out of our minds and using it (retrieval) because thats ultimate goal of learning it the first place.

Studying versus testing and effects on retentionRoediger HL, Karpicke JD. Psychological Science, 2006; 17:249255.

#

Left: Mean proportion of idea units recalled from a prose passage after a 5-min, 2-day, or 1-week retention interval as a function of whether subjects studied the passages twice or studied them once before taking an initial test. Error bars represent standard errors of the means. From Roediger and Karpicke (2006).

Right: Mean proportion of idea units recalled on a final test 5 min or 1 week after learning as a function of learning condition. The shorthand condition labels indicate the order of study (S) and test (T) periods. Error bars represent standard errors of the means. From Roediger and Karpicke (2006).

Studying versus testing and effects on retentionButler AC. Journal of Experimental Psychology: Learning, Memory, and Cognition. 2010; 36:11181133.

#

Accuracy on final tests that consisted of inference-based transfer questions tapping key facts or concepts, administered 1 week after a learn- ing session that involved either practice tests or restudy, in Butler (2010). Error bars represent standard errors.

Similarly, Butler (2010) presented students with expository texts for initial study, which was followed either by repeated restudy or by repeated practice short-answer tests (with feedback) tapping key facts and concepts from the texts. One week later, performance on new inference-based short-answer questions tapping the key facts and concepts was better following practice testing than following restudy (see Fig. 8). The outcomes of a follow-up experiment are particularly striking, given that the criterion test involved far transfer, in that questions required the concepts from one domain to be applied in a novel domain (e.g., students had to apply information learned about bat wings to make inferences about the development of new kinds of aircraft).

Active retrieval as formative assessmentThe only way to assess knowledge is by engaging in the act of retrieval We cant examine what a student has stored; we can only examine what they reconstructUse self-quizzing for retrieval practiceKarpicke JD. Current Directions in Psychological Science. 2012;21(3):157-163.

#

The goal of formative assessment is to provide feedback for students to improve their learning

Formative assessment with feedback

Roediger HL and Butler AC. Trends in Cognitive Sciences. 2011;31:20-27.

#

From Roediger/Butler

providing the correct answer after a retrieval attempt increases the mnemonic benefits of testing the critical mechanism in learning from tests is SUCCESSFUL retrieval because learners are exposed to wrong information in recognition tests (T/F, MC)... Recent research has shown that if students select a wrong answer in a practice, they often repoduce the incorrect information on future test

Pg 23 Research shows that DELAYED feedback may be even more helpful (Maybe make immediate vs. Delayed feedback a slide with results from this and show functionality for delayed exam review in ExamSoft) this could be based on the fact that retrieval practice that occurs in situations where information can be easliy accessed (short-term memory) doesnt lead to beneift for long-term retention (see pag 24 bottom right)

Lets do a Kahoot!

#

Learning is often thought to correlate with encoding new information into memory

Most research has focused on how to ENCODE things, when the ability to USE information and APPLY relies on RETRIEVAL, not encoding

2015 Karpicke and Nunes Retrieval Based Learning Review

Myth #2Learners are good at knowing when they know something.

#

Active retrieval as a metacognitive activity

Nunes LD, Karpicke JD. Retrieval-Based Learning: Research at the Interface between Cognitive Science and Education. In: Scott R and Kosslyn S, eds. Emerging Trends in the Social and Behavioral Sciences. Hoboken, NJ: John Wiley and Sons; 2015.

#

Getting practical:Implementation and utility of ExamSoft

#

The ULM experience: overview

#

We wanted an evidence-based strategy to improve study skills and, ultimately, promote success in our professional pharmacy students. There were two first year classes (drug action I and pathophysiology I) that had traditionally been challenging for our students and had been responsible for MOST of the progression issues in our professional curriculum. Progression is one of the standards that the doctor of pharmacy program is measured on, and we knew that the success of our students and our program were dependent upon finding strategies to promote student success and improve progression. Since David is the director of professional affairs, he had direct access to students who may be more likely to experience academic hardship. We performed a thorough literature search to determine what pre-pharmacy factors influenced non-progression in doctor of pharmacy curricula and found that the two that were most often cited were < 3.0 ____ GPA and multiple Ds or Fs in pre-pharmacy coursework. Using that information, we identified a subset of students who were at a higher risk of academic difficulty and designed a set of interventions to help promote early and continued success in these individuals. These students met with David (director of professional affairs) and signed a study contract that advised them to take part in MULTIPLE helpful resources in order to promote success. The active retrieval/self-quizzing intervention was among the interventions that we used in the first professional year for the class matriculating in 2015.19

The ULM experience: preparing the quizzesIncentivized weekly quiz for each classPosted for download the Monday after a week of material completeOpen until corresponding exam date

Rho Chi (honor society) students wrote questionsCreated template for ease of importStudents used rationale feature to reference notes/textbook pages

Allowed 5 downloads for each weekly quiz

#

We did offer bonus points for the students participating in the weekly quizzes how many bonus points and how many points overall in the class?

Rho Chi students received service hours for participation in question-writing

The ULM experience: question writing

Provided written instructionsProvided RTF template~ for rationale* for correct answer

#

RTF template Peer quizzing instructions

We found that this was a good way to not require extra faculty time for creation of quiz items. Was a challenge in that students werent being quizzed at the same level of difficulty or using the same testing strategies as if faculty had written the questions, but David did challenge the peer question writers to use a variety of questions from different knowledge domains.

ExamSoft feature: import questions

#

ExamSoft feature: bulk question coding

#

From the Question tab, you can choose an entire folder of questions and use the Question Bulk Edit Options Change category assignments to add a category to all questions in that folder.

Weekly question coding and bulk category assignment to allow for longitudinal reporting

ExamSoft feature: bulk question coding

#

The next option will appear to either add or remove categories its helpful to be able to use both in the event that you make an error in coding

We utilized this to code the questions weekly in order to use the coding on longitudinal reports

ExamSoft feature: rationale

Rationale provides feedback for self-quizzing

Easily imports from RTF file with ~ after the question stem

#

Each student referenced the presentation and slide number or a textbook/article reading when writing the question, so students could cross-reference when reviewing.

David referenced the importance of feedback in self-quizzing earlier to ensure that students arent reinforcing wrong answers. The rationale feature allowed the students writing the questions to reference the material from which the question was written for students to reference if they missed or got the question right ** ASK DAVID!

Quiz setupCategorized questions by week and courseEnabled multiple downloadsEnabled secure review

#

Categorizing questions would lend to longitudinal report

ExamSoft feature: edit/save postEasy to update without re-creating the assessmentSettingsSecure ReviewAdditional Security Options

#

From the assessment screen, can choose the pencil to edit a post

We found this very useful if the number of downloads was accidentally set as 1, but we intended to allow for 5

Will also allow for updates to other settings, such as start/finish for downloads and uploads.

Easy to fix mistakes when doing multiple posts per week

ExamSoft feature: exam taker results

#

Graded each quiz each week and released exam taker results since more students would participate closer to the exam

Used this weekly to ensure that attempts were recorded prior to running the longitudinal report

ExamSoft feature: longitudinal reporting

Longitudinal report allowed for easy creation of progress reportRan and downloaded longitudinal reportExported as Excel spreadsheetUsed conditional formatting to highlight empty cellsUploaded to view only GoogleSheetStudents could see their attempts in one place using a unique student ID

#

HAVE TO GRADE BEFORE REPORTS ARE RUN!

After running the longitudinal report, downloaded a Excel file and then used conditional formatting to fill blank cells. Uploaded the updated sheet each week for students to view on Google Drive this allowed them to see if they had missed a quiz attempt, since they were happening so frequently and it was easy to lose track.

Longitudinal report for participation screenshot is in the google docs folder (cant find) - need Davids help

ExamSoft feature: elapsed time report

#

Planned to use the elapsed time report to look at student effort on the quizzes, but werent able to tell much statistically speaking

From assessment window scroll over to the reporting/scoring column and select assessment performance reports, then the elapsed time reports, then select posting, then view report

From here, you can download to an excel spreadsheet so that you can manipulate the data very easily

ExamSoft feature: elapsed time report

#

The ULM experience: student successWhat changed?

School-wide implementation of new early intervention and remediation policies

Bonus points for Boot camp exam (DA, Patho and Pharmaceutics)Practice quizzes (DA and Patho)P1s20142015PHRD 4002 - DA 1Passed post-remediationFailed post-remediationFailed outright540100PHRD 4012 - Patho 1Passed post-remediationFailed post-remediationFailed outright040000Total P1s modified8*0

#

At baseline the classes were similar, so what changed?

The reason the numbers dont add up is because there were 2 other classes that contributed to remediation in 2014 that werent included in the active retrieval exercise, but here we highlight the classes that we implemented active retrieval in

The bonus points resulted in changing a letter grade for 26/89 total students in DA 1and saved only 1 student from having a non-progressing grade InPatho, changed letter grade for 20 students, but none changed from non-progressing to progressing

The ULM experience: post-survey feedback

83/91 (91%) used provided self-quizzes

80/91 (88%) would like to see quizzes provided again

40/91 (44%) used active retrieval as primary study method

#

If you buy into the data, then we need to do a better job of convincing students to use it

The ULM experience: student commentsThese questions were very useful because it allowed me to think about concepts in different ways from experienced students. I often used the practice quizzes generated by older students to help me to think of other ways that a question may be posed for the material, and what other students felt was an important thing to focus on.I thought the quizzes were great and helped me gauge how well I was remembering the material.The quizzes were very helpful in studying because I find testing myself to be a great way to learn.Re-taking the quizzes helped me learn the information quicker than my previous strategies.

#

Some negative feedback about the accuracy of the questions, but overall most students liked the quizzes

Using ExamSoft to Facilitate Active Retrieval and Promote Student SuccessDavid J. Caldwell, PharmD, AAHIVPDirector of Professional Affairs, ULM School of Pharmacy

Elizabeth M. Lafitte, PharmD, BCPSAssistant Professor Clinical Sciences, ULM School of Pharmacy

#