assessment in education
DESCRIPTION
Assessment in Education. Patricia O’Sullivan Office of Educational Development UAMS. Last Month Dr. Allen described a situation about an individual asking her how to get to Freeway Medical Center. How would you determine how successful Dr. Allen was?. Objectives. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/1.jpg)
Assessment in EducationAssessment in Education
Patricia O’SullivanOffice of Educational Development
UAMS
![Page 2: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/2.jpg)
Last Month Dr. Allen described a situation about an individual asking her how to get to Freeway Medical Center. How would you determine how successful Dr. Allen was?
![Page 3: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/3.jpg)
ObjectivesObjectives Provide terminology and principles
fundamental to assessment Introduce various assessment
approaches Review tips for a common assessment:
Multiple Choice Questions Review elements for a less common
assessment: Portfolio
![Page 4: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/4.jpg)
What is Assessment?What is Assessment?
Appraising performance through measurement to make an evaluation
For what purpose? Formative Summative
![Page 5: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/5.jpg)
What Assessments are Used at What Assessments are Used at UAMS?UAMS?
![Page 6: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/6.jpg)
What Makes a Good What Makes a Good Summative Assessment?Summative Assessment?
Fair and practical Based on appropriate criteria that
are shared with the learners Valid Reliable
![Page 7: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/7.jpg)
ValidityValidity Key to the interpretability and
relevance of scores Do the scores have a plausible
meaning? What are the implications of
interpreting the scores? What is the utility of the scores? What are the actual social
consequences?
![Page 8: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/8.jpg)
Validity Evidence for Proposed Validity Evidence for Proposed InterpretationInterpretation
Content/Construct Evidence Content matches the objectives &
teaching Score is shown to be meaningful
Criterion/Predictor Evidence Score relates to future outcomes
![Page 9: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/9.jpg)
ReliabilityReliability Assessment data must be
reproducible Written tests
• Internal consistencyCronbach’s alpha or KR-20 (values
range 0-1)• Test-retest
Correlation ranging from 0-1
![Page 10: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/10.jpg)
Reliability (cont.)Reliability (cont.)
Rater-based assessments Interrater consistency or agreement
(generally correlation coefficient or intraclass correlation)
OSCE & performance assessments Generalizability Theory
![Page 11: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/11.jpg)
Improve ValidityImprove Validity
Match assessment with objectives Increase the sample of objectives
and content on the assessment Review blueprint (or produce one) Use test methods appropriate to
objectives Ensure test security
![Page 12: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/12.jpg)
Improve ReliabilityImprove Reliability
Clear questions Appropriate time allotted Simple, clear and unambiguous
instructions High quality scoring Increase number of observations or
questions
![Page 13: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/13.jpg)
Norm or Criterion ReferencedNorm or Criterion Referenced
Norm-referenced (relative standard) Use the results of the assessment to set the
standards (e.g. proportion to receive each grade); performance judged by comparison with others
Criterion-referenced (absolute standard) Learners achieve some minimal standard of
competency
![Page 14: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/14.jpg)
Standard Setting ProceduresStandard Setting Procedures
Judgments based on Holistic impressions of the exam or
item pool Content of individual test items Examinee’s test performance
![Page 15: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/15.jpg)
Anghoff Standard Setting Anghoff Standard Setting ProcedureProcedure
Judges are instructed to think of a group of “minimally acceptable” persons
For each item estimate the proportion of this group that will answer item correctly
Sum up proportions to get the minimum passing score for a single judge
Average across judges
![Page 16: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/16.jpg)
Assessment MethodsAssessment Methods ACGME Toolbox Additional methods
Other written assessments• Essay• Short answer and computation• Patient management problems• Modified essay questions
Create a game Self-assessment
![Page 17: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/17.jpg)
Matching Assessment Method Matching Assessment Method and Objectivesand Objectives
Examine ACGME matrix
![Page 18: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/18.jpg)
Alternative Testing ModesAlternative Testing Modes
Take-home tests Open-book tests Group exams Paired testing
![Page 19: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/19.jpg)
Reducing Test AnxietyReducing Test Anxiety
Make the first exam relatively easier
Give more than one exam Give advice on how to study Encourage study groups Give a diagnostic test early
![Page 20: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/20.jpg)
Reducing Test AnxietyReducing Test Anxiety
Before a test explain format Provide sample items Provide a pool of final items with
syllabus
![Page 21: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/21.jpg)
Technology and AssessmentTechnology and Assessment
Computer based testing Use of images and video clips
Computer Adaptive Testing (CAT)
![Page 22: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/22.jpg)
Take a TestTake a Test
![Page 23: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/23.jpg)
Multiple Choice Questions: Multiple Choice Questions: Writing TipsWriting Tips
http://www.nbme.org/about/itemwriting.asp Simple suggestions:
Design for one best answer Place hand over stem. Can you
guess the answer? Use a template Attend to Graveyard of Test Items
![Page 24: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/24.jpg)
Template ExamplesTemplate Examples A (patient description) is unable to
(functional disability). Which of the following is most likely to have been injured?
Following (procedure), a (patient description) develops (symptoms and signs). Laboratory findings show (findings). Which of the following is the most likely cause?
![Page 25: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/25.jpg)
GraveyardGraveyard B-type: matching heading with
words or phrases (can use heading more than once)
D-type: complex matching—two step process
K-type: stem with four options and choices are A=1,2,3 only B=1,3 only C=2,4 only D=4 only, E=all are correct
![Page 26: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/26.jpg)
PortfoliosPortfolios
![Page 27: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/27.jpg)
Case Records
Logs
Conferences
Evaluations
![Page 28: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/28.jpg)
PortfoliosPortfolios
Definition A purposeful collection of student
work that exhibits to the student (and/or other) the student’s efforts, progress, or achievement in (a) given area(s).
![Page 29: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/29.jpg)
This collection must include student participation in Selection of portfolio content The criteria for selection The criteria for judging merit Evidence of student
reflection
![Page 30: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/30.jpg)
PortfoliosPortfolios
Learning Showing progress Showing effort Showing
remediation
Assessment Showing
competence
![Page 31: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/31.jpg)
Case Records
Presentations
Evaluations
Logs
![Page 32: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/32.jpg)
Criteria__________________
Criteria__________________
![Page 33: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/33.jpg)
Criteria
![Page 34: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/34.jpg)
Reflection
Portfolio
• Why selected
• How reflects skills
and abilities
![Page 35: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/35.jpg)
Using Portfolios the Learner Using Portfolios the Learner Engages in AssessmentEngages in Assessment
Making choices Critically self-examining Explaining what they are doing
![Page 36: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/36.jpg)
Well-structured PortfolioWell-structured Portfolio
Menu Selection criteria Scoring rubric Benchmarks Trained raters
![Page 37: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/37.jpg)
Determine Reliability and ValidityDetermine Reliability and Validity Reliability: Generalizability theory
Found needed 6 entries with 2 raters Validity: number of studies
Resident performance across years Resident performance in areas Resident perception Resident performance when program made
an intervention Correlation of portfolio performance and
other areas
![Page 38: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/38.jpg)
10-Step Approach10-Step Approach
1. Define the purpose
2. Determine competencies to be assessed
3. Select portfolio materials
4. Develop an evaluation system
5. Select and train examiners
![Page 39: Assessment in Education](https://reader035.vdocuments.mx/reader035/viewer/2022070401/5681372c550346895d9eb942/html5/thumbnails/39.jpg)
10-Step Approach10-Step Approach
6. Plan the examination process
7. Orient learners
8. Develop guidelines for decisions
9. Establish reliability and validity evidence
10. Design evaluation procedures