2 – 6 september 2013 shape (mons) belgium petya georgieva georgi rakovski national defense academy...

21
2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA BULGARIAN STANAG 6001 TESTING TEAM HEAD 2 – 6 September 2013 SHAPE (Mons) Belgium

Upload: clementine-lucas

Post on 21-Jan-2016

232 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

COMPUTER-ASSISTED TESTING

PETYA GEORGIEVA

BULGARIAN STANAG 6001 TESTING TEAM HEAD

2 – 6 September 2013SHAPE (Mons) Belgium

Page 2: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

till 1 March, 2012 – approbation of the CAT “writing” module;

till 1 July, 2012 – analysis of the results; since exam session Autumn 2012 – optional CAT “writing”

module; March, 2013 – approbation of CAT modules for “reading”

and “listening”; till 1 July, 2013 – analysis of the results; since exam session Autumn 2013 – optional CAT “writing”,

“reading”, and “listening” modules; since exam session Autumn 2014 – all candidates are to be

tested on computers for the modules “writing”, “reading”, and “listening”(Action Plan 1 iaw Ministerial Order № 857/10.11.2011).

since exam session Autumn 2013 – parallel pen-and-paper and CAT in the three skills.

As of Spring 2015 - End of pen-and-paper STANAG 6001 tests (Action Plan 2 iaw Ministerial Order № 219/19.03.2013 ).

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

*BG CAT Background

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 3: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Administrative Module main menus:ListsProtocolsTasksTestsEvaluation and Analysis

Page 4: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 5: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 6: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

Candidates’ module: Reading Test Listening Test Writing Test

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 7: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 8: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 9: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

follows the current Specifications for STANAG 6001 exam for

the Bulgarian Armed Forces; starts with items for L1, then L2, then L3 ; a minimum of 12 and a maximum of 18 questions per

level; level is given for a minimum of 70% correct answers; a candidate can terminate the exam when a level is achieved or decide to go up; 50%-69% correct answers of the next level → (+)

level.

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 10: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

Paper-based

withdrawals -16.67%

Failed – Reading (1.5%) Listening (2.5%) Writing (1%)

CAT

withdrawals – 17.16%

Failed – Reading (2.56%) Listening (16.21%) Writing (0%)

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 11: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 12: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 13: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

CAT – 33% of candidates improved their results (R, L, or both).

Reading – 30.76% showed lower results;

33.33% confirmed the results;

35.91% scored higher.

Listening – 27.92% showed lower results;

39.63% confirmed the results;

32.45% scored higher.

Data refers to March 2013 CAT piloting session (R and L).

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 14: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 15: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 16: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

PROS:

candidates can manage their time;decreased candidates’ frustration;time saving (for both candidates and testers);anonymity criteria unconditionally complied with;immediate feedback;highly objective feedback;fewer items are compromised;new items can be piloted and statistics is facilitated.

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 17: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

Cons: interface problems for the test-takers

(computer screen clarity, font, size, color of the letters)

due to test-takers’ characteristics (computer literacy, motor skills needed to respond to test items, danger of bad time-management);

no option for test administrators to troubleshoot during the test process.

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 18: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

RECOMMENDATIONS FUTURE ACTION PLANS

(based on candidates retrospective written and oral feedback

and test administrators’ impressions);

Several groups:

technical (software perfection, interface layout);

methodological (revision of test components and test administration)

facility use and administrative support

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 19: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

Instructions – reviewed;

Layout of the reading material – black font; screen allowing for parallel viewing of the text and answer sheet;

Increase time allotted for looking through the options for L1 and L2 candidates at the Listening;

Reducing number of items if more than 3 per text (R and L).

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 20: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA

Page 21: 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA

2 – 6 September 2013SHAPE (Mons) Belgium Petya Georgieva

? ? ? ?

? ?

GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA

BULGARIA