collecting, analyzing, interpreting, and faculty ... analyzing, interpreting, and using ... •...

Download Collecting, Analyzing, Interpreting, and Faculty ...  Analyzing, Interpreting, and Using ... • Steps to analyzing data • Using evaluation data ... • Item analysis

Post on 24-Mar-2018

221 views

Category:

Documents

9 download

Embed Size (px)

TRANSCRIPT

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 1

    1

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    April19,2011

    MichaelTheallYoungstownStateUniversity

    mtheall@ysu.edu

    2

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    LEARNING OBJECTIVESYouwillbeabletoidentifythemostproductivewaystousefacultyevaluationdata:

    *formativelyforimprovementand*summatively fordecisionmaking

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 2

    3

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Agenda Evaluationpurposesanddatatypes Collectingdatafrommultiplesources Stepstoanalyzingdata Usingevaluationdata Guidelinestogoodpractice

    4

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    1.EvaluationPurposesandDataTypes

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 3

    5

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    AllEvaluationandDevelopmentareLocal!

    6

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    EvaluationwithoutDevelopmentisPunitive

    DevelopmentwithoutEvaluationisGuesswork!

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 4

    7

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Evaluationanddevelopmentsystemswillnotbecompleteuntiltheyarebasedonanunderstandoftheworkthatfacultyareexpectedtodo,theskillsthatarerequiredtodothatwork,andthecriteriatobeappliedinevaluation!

    8

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Summarymatrixhttp://www.cedanet.com/meta

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 5

    9

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Data Purposes datatypes Datauses Research[Ratings;otherdata] Decisionmaking Instrumentation Analysis Interpretation

    10

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    BasicIssuesaffectingdatadecisions

    Validity Reliability Generalizability Feasibility Skulduggery

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 6

    11

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Whatinformationdostakeholdersneed?

    12

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    EvaluationInformationMatrix:DevelopingaSynergyforImprovedPractice

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 7

    13

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    EvaluationPurposesandDataTypes

    14

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    UsesofDataPersonnelDecisions TeachingImprovement

    Assessablemodifiablebehaviors

    Qualitativeprocessoriented

    Informativedatabase

    Comprehensivedetailedsuggestive

    Specificlowinferenceitems

    Confidential

    Overallperformance

    Quantitativeoutcomeoriented

    Comparativedatabase

    Empiricaldirectunambiguous

    Globalitems

    Public

    Supportingdata

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 8

    15

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    2.CollectingDataFromMultipleSources

    16

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    16

    Sourcesofdata: StudentOpinions&StudentLearning PeerEvaluation(internal&external) AdministratorandSelfEvaluation Service(department;college;university;national;professional,community)

    ScholarlyWork(pubs;presents;citations;interviews;media;invitedopinion;testimony;products,performances;exhibits)

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 9

    17

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    17

    Sourcesofdata: Awards;Honors;Grant$;Donation$ ProfessionalActivities(consulting&probono) Administration(committees;grants;serviceorGenEdcoursecoordination;GA/TAsupervision;interimroles)

    Advising;Recruiting;Admissions;Retention AlumniRatings&EmployerOpinions MediaDocumentation

    18

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    18

    ResearchFindingsandRecommendations:

    Studentratings Othersourcesofevaluationdata

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 10

    19

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    19

    Whatareratings? Multidimensional ReliableandStable Primarilyafunctionoftheinstructor Relativelyvalidasevidenceofeffectiveteaching

    Relativelyunaffectedbyanumberofvariablesposedasbiases

    UsefulasteachingfeedbackMarsh,2007

    20

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    20

    Additionalfindings: Classsize:slightnegative(curvelinear) Priorinterestinsubject:positive Electivevs.requiredcourses:morepositiveforelectives

    Disciplinaryarea:consistentdifferences Work/difficulty:slightpositive(curvelinear) Courselevel:slightpositiveforupperdivision&grad

    Anonymity:ratingsmorepositiveifviolated

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 11

    21

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    21

    Additionalfindings: Purposeofeval:morepositiveifmanipulated Instructorrank:none Teacher/studentgender:none Teacherethnicity/race:none Researchproductivity:none Studentlocus&performanceattributions:none Student/teacherpersonality:none

    22

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    22

    ITEMS RANKED CORRELATION

    WITH ACHIEVEMENT RANKED CORRELATION

    WITH EVALUATIONS 1. preparation and organization 1 6 2. clarity and understandableness 2 2 3. perceived outcome or impact 3 3 4. stimulation of interest in content 4 1 5. encouragement and openness 5-6 11 6. availability and helpfulness 5-6 16 7. presentation and speaking skills 7-8 10 8. clarity of objectives and requirements 7-8 7 9. subject knowledge 9 9 Dimensions of teaching table adapted by Michael Theall with permission from Kenneth Feldman. Source: Feldman, K. A. (1989). The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education, 30, 583-645.

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 12

    23

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    23

    ITEMS RANKED CORRELATION

    WITH ACHIEVEMENT RANKED CORRELATION

    WITH EVALUATIONS 10. concern for student progress 10 5 11. teacher enthusiasm for subject 11 8 12. teacher's fairness 12 14-15 13. intellectual challenge 13 4 / 14. concern / respect for students 14-15 12 15. feedback quality & frequency 14-15 17 16. nature / value of course material 16 13 17. nature / usefulness of supplements/aids 17 14-15 Dimensions of teaching table adapted by Michael Theall with permission from Kenneth Feldman. Source: Feldman, K. A. (1989). The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education, 30, 583-645.

    24

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    24

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 13

    25

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Othersourcesofdata: PeerEvaluation

    (usuallydepartment,butsometimesinstitutionalcolleagues)

    Bestforteacherknowledge,certaincourseorcurricularissues,assessmentissues,currency/accuracyofcontent,(esp.whenusedalongwithstudentratings).Ifonteaching,lessreliableandhigheronaveragethanstudentratings.

    26

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Othersourcesofdata: AdministratorEvaluation(departmentchair)

    Necessaryaspartofprocess,butsameproblemsaspeersonteaching(criteria,process,instruments,validation,etc.)

    Self Evaluation(e.g.,inaportfolio)Providesthemostcompletepictureofteacherthinking&instructionaldecisions/practices,butdifficulttoreliablyinterpret&use

  • Collecting,Analyzing,Interpreting,andUsingFacultyEvaluationData

    AcademicImpressions 14

    27

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Othersourcesofdata:

    StudentLearningOutcomesUsefulforformative(individual)orprogrampurposes(ifaggregatedforassessment);notrecommendedforsummativedecisions.Testscoresarenotasreliableasratingsfromavalidatedinstrument.Criteriavaryconsiderably(e.g.,WhatdoesAllherstudentsgotAsmean?)

    28

    COLLECTING,ANALYZING,AND USING FACULTY EVALUATION DATA

    Othersourcesofdata: ExternalExpertEvaluation(almostalwaysbyanexpertin

    thesamediscipline;sometimesbyanexpertinteaching)

    Useful,butrequireprocesscautionsandcarefuluse/interpretation;havingapurposeisimportant

    AlumniRatings(atvariouscareerstages&times)Canbeusefulbutgenerallythesameasstudentratingsgivensameinstrument;canshedlightonteachingintermsofcontent,process,orcurricularissuesforformativepurposes.

  • Collecting,Ana

Recommended

View more >