development of evaluation instruments based on ict skills standards for teachers (isst) 7th may,...
TRANSCRIPT
Development of Evaluation Instruments based on ICT Skills Standards for Teachers (ISST)
7th May, 2014
CONTENTS
I Request of MOE
II Proposal for Evaluation
III Development Strategy & Schedule
IV Results & Expectation
I Request of MOE
Request of MOE
Mixed Model(method)
CategoryEvaluation
Set 1 Set 2 Set 3
Category A Likert Scale
Category B Multiple (a) Multiple(b) Multiple(c)
Category C Multiple (a) Multiple(b) Multiple(c)
Category D Likert Scale
… … … …
Definition of Mixed Methods Evaluation
Qualitative Data
Mixed Method
Quantitative Data
• Instruments• Checklist• Records
• Interviews• Observations• Documents• Audio-visual materials
Mixed methodology is a design for collecting, analyzing, and mixing both quantitative and qualitative data in a single study or series for studies to understand an evaluation problem(Creswell and Plano Clark, 2007).
II Proposal for Evaluation
Proposal for Objective and Valid Evaluation
Multiple ChoiceEstimate scientific, directive, and objective performance.Take long time to develop the test sheet and also difficult
to develop identical test sheets. Respondents’ position, difficult to respond.
Before developing the test times, it is necessary to check the characteristics of respondents
Likert ScaleEasy to verify the reliability and validity of test items.
Respondents’ position, Easy to respond. The same test items can be used.
It’s difficult to estimate real ability because it is based on the respondent’s perception.
Respondents can misrepresent about their ability
EvaluationInstruments
Likert Scale
Multiple Choice
• Master Teacher• Leader Teacher
• Leader Teacher
Evaluation Framework of ISST
CompetenciesCompetency Factors
Accessibility Capability Practical Use Satisfaction
Technology
Communicative
Pedagogical
Research /Investigative
Management
Design
Domain of ICT competence for teachers is technology, communicative, pedagogical, research/investigative, management and design.
Teacher Competencies for ICT Use in Education
According to the MEN (2013), there are three levels of all teachers’ competencies: Explorer, Integrator, and Innovator.
CompetenciesLevels
Explorer Integrator Innovator
Technology
Communicative
Pedagogical
Research /Investigative
Management
Design
III Development Strategy & Schedule
Development Strategies and Schedule
Develop & Confirm Indicators• 5.10 ~ 5. 20
Expert review• 5.20 ~ 5. 30
Preliminary test with 300 samples • 6. 1 ~ 6. 7
Analysis and set up the tests • 6. 8 ~ 6. 15
Development oftaxonomy• 6. 1 ~ 6. 10
Item development• 6. 10 ~ 6. 25
1st Preliminary test With 300 samples • 6. 25 ~ 7. 5
Analysis and revise the tests • 7. 5 ~ 7. 10
Setting the criteria6. 15 ~ 6. 25
2st Preliminary test With 300 samples • 7. 10 ~ 7. 20
Analysis and set up the tests • 7. 20 ~ 7. 25
Setting the criteria7. 25 ~ 7. 31
Multiple Choice
Likert Scale
Development Methodology
Validation of Evaluation Instruments
Content validity (through experts’ review) Construct validity (through statistic program) Item response theory (through statistic program)
Method for Setting the Criteria ( Standard-setting methods)
Angoff & Jaeger method
IV Results & Expectation
Representation of Evaluation Result
Radial Chart
Technology
Communicative
Pedagogical
Research / Investigative
Management
Design
0
2
4
Average
Investigative
Setting the Level of Teachers based on Criteria
InnovatorIntegratorExplorer
Score
Score
Score
Output
Likert Scale At least 72 items
Multiple Choice At least 100 items
ISST & Evaluation Items Development
ISST & Evaluation Items Development
Draft ISST Test Item
( 4. 23 ~ 5. 8 )
Development of ISST
( 3.4~4.30 )Conduct Expert
Review( 5. 8 ~ 5. 15 )
Validate Test Items( 5. 15 ~ 5. 30 )
Construct Test Sets
( 5. 30 ~ 6. 5 )
Review and Confirm Test Sets
( 6. 5 ~ 6. 11 )
Pilot Test and Port the Online Test
System ( 6. 11 ~ 6. 17 )
Submit Test sets
( 6. 17 )
Prepare Draft Report
( 6. 17 ~ 7. 25 )
Review by MOE and Revise the
Report ( 7. 25 ~ 8. 18 )
Submit Final ISST Research
Report( 8. 18 )
ISST & Evaluation Items DevelopmentConstruct Test
Sets( 4. 30 ~ 5. 5 )
Development of ISST
( 3.4~4.30 )Review and
Confirm Test Sets( 5. 5 ~ 5. 10 )
Draft Likert Scale Item
( 5. 10 ~ 5. 20 )
Conduct Expert Review
( 5. 20 ~ 5. 30 )
Preliminary Test with 300 samples
( 6. 1 ~ 6. 7 )
Analysis and Set Up the Test
( 6. 8 ~ 6. 15 ) Submit Likert Scale Test set
( 6. 17 )
Prepare Report & Revise
( 6. 17 ~ 8. 18 )
Submit Final ISST Research
Report( 8. 18 )
Setting the Criteria ( 6. 15 ~ 6. 25 )
Development of Taxonomy
( 6. 1 ~ 6. 10 )
Item Development ( 6. 10 ~ 6. 25 )
1st Preliminary Test ( 6. 25 ~ 7. 5 )
Analysis and Revise the Test ( 7. 5 ~ 7. 10 )
2st Preliminary Test ( 7. 10 ~ 7. 20 )
Analysis and Set Up the Test
( 7. 20 ~ 7. 25 )
Setting the Criteria ( 7. 25 ~ 7. 31 )
Submit Multiple Choice Test set
( 7. 25 )
Question & Answers