evidence-centered approach to online assessment of students’ digital competence
Post on 15-Apr-2017
99 Views
Preview:
TRANSCRIPT
Evidence-Centered Approach to Online Assessment of Students’
Digital Competence
Mart Laanpere, Kai Pata (Tallinn University)Mario Mäeots , Leo Siiman (University of Tartu)
Problem space• Need to assess digital competence: a newly
defined key competences in Estonian national curriculum for primary and secondary schools
• Requirements: – Compliant with DigComp framework– Implemented online using e-exam system EIS – Scalability– Automated assessment– Reliability, validity
Digital competence
• Three competing approaches to digital competence/competencies:– Theory-based knowledge, derived from the academic
discipline of computer science (computational thinking)– Pragmatic skills, needed at the workplace (ECDL)– Key competence, contextualised, closer to “learning to
learn” skill• DigComp follows the second approach, Estonian
curriculum combines 2nd and 3rd
Teaching and assessing ICT skills in Estonia
• 1986: Computer science as a compulsory theoretical subject, no exam, teacher-created tests
• 1996: Informatics as an elective subject, learning generic ICT skills for office jobs
• 2000: ICT and media as cross-curricular theme, pilot exam on national level (test + practical task)
• 2011: informatics as an elective subject, digital competence as a key competence
Research design and organisation
• Two teams of researchers from two different universities (University of Tartu & Tallinn University)
• Combination of two approaches: – Inductive/descriptive, bottom-up (Tartu team):
collecting and analysing the best practice from large-scale comparative and national studies
– Deductive/prescriptive, top-own (Tallinn team): building a new framework from scratch, deducing the new type of practice
ECD framework (Almond, Mislevy et al. 2015) inspired by e-exams of Cisco Network Academy;Has been applied in various contexts: Assessing languagecompetence,Inquiry skills in STEMEtc.
Layered assessment architecture of ECDLayer Role Key entities Examples
Domain analysis
Collect info about domain, teaching practice, frames
Concepts; terminology; tools; representation forms
Curricula, rubrics, test
Domain modelling
Express assessmentargument in narrative form
KSAs; potential work products;potential observations
Design patterns
Assessment framework CAF
Express assessmentargument as blueprints for tasks or items
Student, evidence, and task models; student-model, rubrics etc
Task templates
Imple-mentation
Implement assessment,presenting tasks or items and gathering andanalyzing responses
Task materials (incl.all materials, tools, affordances); work products
Rendering protocols for tasks; IMS/QTIrepresentation of materials & scores;
Delivery Coordinate interactions of students and tasks; task and test-level scoring; reporting.
Tasks as presented; workproducts as created; scores as evaluated.
summaries forindividual and group-level reports;
Assessment framework (CAF)
What?
How?
Where?
Extent?
Domain analysis: example
Performance taxonomy
Fact Concept Procedure Rule
Knows
Understands
Applies
HOTS
Based on: Merrill ja Feisel-Schmitz
Bayesian network
Information & Data Literacy
Digital content creation
Communication & collaboration
Digital safety
Problem solving
KSA (Knowledge-Skills-Abilities)P_Info P_Comm P_Content P_Safety P_Probl
T_Info
T_Comm
T_Content
T_Safety
T_Probl
Example: practical task
• Create a presentation that compares two countries: Estonia and Slovenia
• Criteria: – At least 4 slides/screens– Diagrams based on reliable and recent data– Images (IPR need to be followed, licensing)
Piloting
• Not completed, planned in 6-8 schools this October
• Bayesian data analysis, evidence accumulation• Scenario-based design of new task templates• Exploring the solutions for simulation tasks
(inspired by examples collected by Tartu group)
Questions for discussion
• Any experiences with ECD?• How to combine inductive and deductive
approaches?• Semi-automated assessment of practical
tasks?
top related