hr lsa trends
DESCRIPTION
trensdsTRANSCRIPT
![Page 1: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/1.jpg)
Advancements in Large-‐scale Assessment
Heiko Rölke & Krisz:na Toth DIPF -‐ Deutsches Ins:tut für Interna:onale
Pädagogische Forschung Frankfurt am Main
TAO Days | 10.09.2012 Seite 1
![Page 2: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/2.jpg)
Overview
• Trends in LSA • Closer Look at Trends • Example: Making Use of Data • Challenges in LSA
TAO Days| 10.09.2012 Seite 2
![Page 3: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/3.jpg)
Trends
Main trend: CBA
Sub-‐trends/effects of CBA: • CAT
• towards forma:ve assessment • Complex items
• Simula:on, interconnec:on • Interweaving
• different tests, ques:onnaires, etc. • Big data
TAO Days | 10.09.2012 Seite 3
![Page 4: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/4.jpg)
Trend: CBA
Computer-‐based Assessment has to serve a purpose • Costs? • Time! • Validity!
TAO Days | 10.09.2012 Seite 4
![Page 5: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/5.jpg)
CAT
• Also not prac:ced for its own sake • Not as important in pure summa:ve assessment • But:
• Strict :me limits (e.g. PIAAC) • Growing demand for forma:ve aspects in summa:ve assessment
TAO Days | 10.09.2012 Seite 5
![Page 6: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/6.jpg)
Complex Items
• Closer at reality • Simula:on Most important: • Ability to assess new domains
TAO Days | 10.09.2012 Seite 6
![Page 7: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/7.jpg)
Examples of Complex Items § Simula+on of an “Email / Web Scenario”
The test person receives an email and should book cinema :ckets online.
§ Dynamic Model (MicroDYN)
The test person should explore and master a dynamic system with input (exogenous) variables influencing output (endogenous) variables.
§ Automaton with Finite State Machine (MicroFIN) The test person should interact with a mobile phone and set :me to summer:me.
![Page 8: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/8.jpg)
Interweaving of Tests and Ques:onnaires • Framing
• test only if precondi:ons are fulfilled • Double-‐check
• findings of ques:onnaire
TAO Days | 10.09.2012 Seite 8
![Page 9: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/9.jpg)
Big Data
• Log data, not only results • Find out what is going on • Make use of data
• E.g. par:al scoring
• -‐> Examples from PIAAC PS-‐TRE
TAO Days | 10.09.2012 Seite 9
![Page 10: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/10.jpg)
Simula:on-‐based assessment
• Modern assessments: • real life situa:ons: e.g. web environments • complexity of instrument – various ways
• Monitoring students‘ ac:ons -‐ log files • Analysis of test-‐taking paths is a rela:vely new field in educa:on
(reason: printed vs. computer-‐based data collec:on) • Process data requires to consider and evaluate methods
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 10
![Page 11: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/11.jpg)
Aim • to inves:gate how log data can be integrated into the process
of educa+onal assessment and evalua:on to support researchers and prac::oners in making use of the data assembled in computer-‐based test delivery
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 11
![Page 12: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/12.jpg)
Aim • to inves:gate how log data can be integrated into the process
of educa+onal assessment and evalua:on to support researchers and prac++oners in making use of the data assembled in computer-‐based test delivery
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 12
![Page 13: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/13.jpg)
Hypertext Item
13
![Page 14: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/14.jpg)
14
![Page 15: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/15.jpg)
15
829709 15 {"sender":"15","type":"loading","data":""} 833463 15 {"sender":"15","type":"loaded","data":""} 848153 15 {"sender":"15","type":"user_interac+on","":"<?xml version=\"1.0\">
\u000d\u000a<cbaloggingmodel:EmbeddedLinkLogEntry xmlns:cbaloggingmodel=\"htp://www.soucon.de/cba/cbaloggingmodel\" id=\„-‐ b576759f1:-‐7fed\" sourcePageId=\"Item15_linklist\" targetPageId=\"Item15_website1\" textFieldId=\"cbaTextField_71"/>\u000d\u000a"} 849094 15 {"sender":"15","type":"variable_change","data":{"name":
"snapshot_url„,"value":"htp://localhost:8101/cba-‐run:me/itemjsessionid= LB1?custom_servicehandler=downloadService&file= C:\\...snapshot5537437125227002523.xml"}}
855852 15 {"sender":"15","type":"user_interac+on","data":"<?xmlversion= \"1.0\“encoding=\"UTF8\"?>\u000d\u000a <cbaloggingmodel:BuRonLogEntryxmlns:cbaloggingmodel=\"htp://www.sou con.de/cba/cbaloggingmodel\id=\"cbaBackBuRon_14_13454937565947\"/> \u000d\u000a„}":"15","type":"unloaded","data":""}
Example log
![Page 16: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/16.jpg)
Promising methods • Sta:s:cs and visualisa:on • Clustering • Classifica:on
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 16
![Page 17: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/17.jpg)
Process Measures
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 17
1. Number of page visits 2. Number of different page visits 3. Visit of relevant page 4. Time spent on the relevant page 5. Ra:o of :me spent on the relevant page 6. Ra:o of :me spent on the opening screen 7. Comple:on :me
![Page 18: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/18.jpg)
Sta:s:cs
Process measures Mean Standard Devia+on
Number of page visits 5.18 3.13 Number of different page visits 2.38 1.38 Time spent on the relevant page 10.87 9.77 Ra:o of :me spent on the relevant page .14 .11 Ra:o of :me spent on the opening screen .64 .18 Comple:on :me 67.74 27.49
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 18
![Page 19: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/19.jpg)
Distribu:on of different page visits
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 19
![Page 20: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/20.jpg)
Distribu:on of different page visits
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 20
![Page 21: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/21.jpg)
Visualisa:on
21 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-‐Scale Assessments
![Page 22: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/22.jpg)
Visualisa:on
22 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-‐Scale Assessments
![Page 23: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/23.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits
Nr. of different page visits
Comple:on :me
Relevant page visited (Y/N)
Ra:o of :me on relevant page
Ra:o of :me on star:ng page
Distribu:on of sequences (%) 23% 16% 34% 27%
Ra:o of correct responses
23 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 24: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/24.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits
Nr. of different page visits
Comple:on :me
Relevant page visited (Y/N)
Ra:o of :me on relevant page
Ra:o of :me on star:ng page
Distribu:on of sequences (%) 23% 16% 34% 27%
Ra:o of correct responses 25.0% 1.7% 92.6% 84.4%
24 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 25: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/25.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41
Nr. of different page visits .36 2.54
Comple:on :me 28.38 62.08
Relevant page visited (Y/N) No No
Ra:o of :me on relevant page 0 0
Ra:o of :me on star:ng page .93 .60
Distribu:on of sequences (%) 23% 16%
Ra:o of correct responses 25.0% 1.7%
25 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 26: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/26.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits 4.73 9.57
Nr. of different page visits 2.29 4.53
Comple:on :me 59.60 91.60
Relevant page visited (Y/N) Yes Yes
Ra:o of :me on relevant page .20 .17
Ra:o of :me on star:ng page .62 .45
Distribu:on of sequences (%) 34% 27%
Ra:o of correct responses 92.6% 84.4%
26 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 27: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/27.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits 5.41 4.73
Nr. of different page visits 2.54 2.29
Comple:on :me 62.08 59.60
Relevant page visited (Y/N) No Yes
Ra:o of :me on relevant page 0 .20
Ra:o of :me on star:ng page .60 .62
Distribu:on of sequences (%) 16% 34%
Ra:o of correct responses 1.7% 92.6%
27 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 28: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/28.jpg)
Cluster Analysis
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41 4.73 9.57
Nr. of different page visits .36 2.54 2.29 4.53
Comple:on :me 28.38 62.08 59.60 91.60
Relevant page visited (Y/N) No No Yes Yes
Ra:o of :me on relevant page 0 0 .20 .17
Ra:o of :me on star:ng page .93 .60 .62 .45
Distribu:on of sequences (%) 23% 16% 34% 27%
Ra:o of correct responses 25.0% 1.7% 92.6% 84.4%
28 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 29: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/29.jpg)
Summary and Conclusions
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 29
![Page 30: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/30.jpg)
Summary and Conclusions
Features Cluster 1 Cluster 2 Cluster 3 Cluster 4
Nr. of page visits .72 5.41 4.73 9.57
Nr. of different page visits .36 2.54 2.29 4.53
Comple:on :me 28.38 62.08 59.60 91.60
Relevant page visited (Y/N) No No Yes Yes
Ra:o of :me on relevant page 0 0 .20 .17
Ra:o of :me on star:ng page .93 .60 .62 .45
Distribu:on of sequences (%) 23% 16% 34% 27%
Ra:o of correct responses 25.0% 1.7% 92.6% 84.4%
30 Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments
![Page 31: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/31.jpg)
Future work • Pilot study – sample size • Valida:on
• Other types of items require new process measures
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 31
![Page 32: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/32.jpg)
32
![Page 33: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/33.jpg)
Future work • Pilot study – sample size • Valida:on
• Other types of items require new process measures • We have a lot of work to do:
• Souware developer • Test developer • Psychometricans
Berlin, 10.9.2012 | H. Rölke & K. Tóth | TAO Days 2012 | Advancements in Large-Scale Assessments 33
![Page 34: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/34.jpg)
(Selected) Challenges
• Authoring and Management • Delivery • Re-‐use and exchange
TAO Days | 10.09.2012 Seite 34
![Page 35: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/35.jpg)
(Selected) Challenges
• Authoring and Management • Delivery • Re-‐use and exchange
TAO Days | 10.09.2012 Seite 35
![Page 36: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/36.jpg)
(Selected) Challenges
• Authoring and Management • Delivery • Re-‐use and exchange
TAO Days | 10.09.2012 Seite 36
![Page 37: Hr Lsa Trends](https://reader033.vdocuments.mx/reader033/viewer/2022051019/5695d2f71a28ab9b029c5a77/html5/thumbnails/37.jpg)
(Selected) Challenges
• Authoring and Management • Delivery • Re-‐use and exchange
TAO Days | 10.09.2012 Seite 37