advancing assessment of quantitative and scientific reasoning

Download Advancing Assessment of  Quantitative and  Scientific Reasoning

Post on 15-Jan-2016




0 download

Embed Size (px)


Advancing Assessment of Quantitative and Scientific Reasoning. Donna L. Sundre Amy D. Thelk Center for Assessment and Research Studies (CARS) James Madison University Overview of talk. Current NSF Research project History of the test instrument - PowerPoint PPT Presentation


  • Advancing Assessment of Quantitative and Scientific Reasoning

    Donna L. SundreAmy D. ThelkCenter for Assessment and Research Studies (CARS)James Madison

  • Overview of talkCurrent NSF Research projectHistory of the test instrumentPhase I: Results from JMUPhase II: Future directionsResults from some of our partners:Michigan StateTruman StateVirginia State

  • Current NSF Project3-year grant funded by National Science Foundation: Advancing assessment of scientific and quantitative reasoningHersh & Benjamin (2002) listed four barriers to assessing general education learning outcomes: confusion; definitional drift; lack of adequate measures, and misconception that general education cannot be measuredThis project addresses all of these concerns with special emphasis on the dearth of adequate measures

  • Objective of NSF project Exploring the psychometric quality and generalizability of JMUs Quantitative and Scientific Reasoning instruments to institutions with diverse missions and serving diverse populations.

  • Partner InstitutionsVirginia State University: State-supported; Historically Black institution

    Michigan State University: State-supported; Research institution

    Truman State University: State-supported; Midwestern liberal arts institution

    St. Marys University (Texas): Independent; Roman-Catholic; Hispanic Serving institution

  • Project phasesPhase I: First Faculty institute (conducted July 2007 at JMU); followed by data collection, identification of barriers, and reporting of resultsPhase II: Validity studies (to be developed and discussed during second faculty institute, July 2008), dissemination of findings and institutional reports

  • History of the instrumentNatural World test, developed at JMU, currently in 9th versionSuccessfully used for assessment of General Education program effectiveness in scientific and quantitative reasoningGenerates two subscores: SR and QRSummary of results since 2001Table of Results -- 5 Test Versions.doc

  • Adaptation of an instrument

    JMU instrument has been carefully scrutinized for over 10 yearsThe QR and SR is currently administered at over 25 institutions across the nationNSF decided to fund this CCLI project to further study procedures for adoption and adaptation of instruments and assessment models

  • Evaluating the generalizability of the instrument

  • Step 1: Mapping Items to ObjectivesRelating test items to stated objectives for each institutionIn the past back translation method was used (Dawis, 1987) ..\..\JMU\NSF Grant\Truman\Blank ObjectiveGrid_truman.docParticipants at the NSF Faculty Institute used a new content alignment method that was reported on at NCME (Miller, Setzer, Sundre & Zeng, 2007)Forms were custom made for each institutionExample Content Alignment form.doc

  • Early content validity evidenceResults strongly support generalizability of test itemsTruman State: 100% of items mapped to their objectivesMichigan State: 98% (1 item not mapped)Virginia State: 97% (2 items unmapped)St. Marys: 92% (5 items not mapped)Mapping of items alone is not sufficientBalance across objectives must be obtainedTeams then created additional items to cover identified gaps in content coverage14 for MSU; 11 for St. Marys; 10 for Truman State; 4 for VSU

  • Step 2: Data Collection and AnalysisDuring Fall 2007 semester, test was administered to students at 3 of the 4 partner institutions Spring 2008 data collection from students at sophomore level or aboveResults so farMeans not given: This activity is not intended to promote comparison of students across institutionsAt this stage, reliabilities provide the most compelling generalizability evidence; of course, the upcoming validity studies will be informative

  • Research at JMU

    Standard Setting to aid in interpretation

    Validity evidence: Instrument aligns with curriculum

  • Standard SettingUsed Angoff Method to set standardsOur process was informal, uniqueResults look meaningful but well reevaluate as we collect more data in upcoming administrations

  • Faculty Objective Standards


    Objective 10.2030.5

    Objective 20.1840.365

    Objective 30.1870.34

    Objective 40.1420.256

    Objective 50.1470.449

    Objective 60.1170.263

    Objective 70.1990.487

    Objective 80.1420.34


    NW-9 Total0.1350.474

    Faculty Standard*

    Freshmen (no CL3 experience)

    CL3 Package completers

    Proportion of students meeting standard

    Proportion of students meeting faculty objective standards


    ObjectiveFaculty Standard*Freshmen (no CL3 experience)CL3 Package completers

    Objective 1-0.800.200.50

    Objective 2-0.730.180.37

    Objective 3-0.760.190.34

    Objective 4-0.790.140.26

    Objective 5-0.750.150.45

    Objective 6-0.760.120.26

    Objective 7-0.780.200.49

    Objective 8-0.750.140.34


    NW-9 Total-0.760.140.47



  • Validity evidence for instrument and curriculum at JMU

  • Validity evidence for instrument and curriculum at JMU -- 2

  • Phase II studiesSamples of Upcoming Studies:Correlational Studies: Is there a relationship between scores on the QR/SR and other standardized tests? and other academic indicators?Comparison of means or models: Is there a variation in the level of student achievement based upon demographic variables? Is there a relationship between scores on the QR/SR and declared majors? Can this instrument be used as a predictor for success and/or retention for specific majors?Qualitative Research: Will institutional differences be reflected in the results of a qualitative interview that accompanies the administration of QRSR?

  • ReferencesDawis, R. (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.Hersh, R. H., & Benjamin, R. (2002). Assessing selected liberal education outcomes: A new approach. Peer Review, 4 (2/3), 11-15.Miller, B. J., Setzer, C., Sundre, D. L., & Zeng, X. (2007, April). Content validity: A comparison of two methods. Paper presentation to the National Council on Measurement in Education. Chicago, IL.

  • History of back translation; problemsBenefits of new methodShow example form of eachKeep in mind that although each schools objectives were related to general education science/math, each institution had a different set of objectives!


View more >