iatefl(16 apr 2016) - · pdf file4/16/2016 · 16/07/06 1 developing a...
TRANSCRIPT
16/07/06
1
Developingacomputer-basedspeakingtestofEnglishasaLinguaFranca
IATEFLAnnualConferenceBirmingham,April2016
YumiHato,KyotoIns0tuteofTechnology,JapanKatsunoriKanzawa,KyotoIns0tuteofTechnology,Japan
YasushiTsubota,KyotoIns0tuteofTechnology,Japan
HaruhikoMitsunaga,ShimaneUniversity,Japan
NicUnderhill,educa0onandassessmentconsultant,UK
2012 Oct. launchedprojectatKyotoIns0tuteofTechnology(KIT)
2013 Jan. heldsymposium“ExploringthePossibilityofIntroducingaSpeakingTestinUniversityEntranceExamina0ons”
Feb. startedtodrawuptestspecifica0on
Apr. startedtodevelopspeakingtestsystemwithKITresearchersininforma0ontechnology
May firstna0onalgrant,Grant-in-AidforScien0ficResearch,awarded
2014
Jul. startedtodevelopCBTadministra0onsystemandonlinera0ngsystemwithE-communica0ons,Inc.(CBTdeveloperandprovider)
2015 testadministra0on
Sep.
startedtoworkwithQQEnglish(onlineEnglishlessonproviderbasedinthePhilippines)inra0ng
2016 Apr. secondna0onalgrantawarded
Date NumberofExaminees Version
1stadministra0on
20-22Jan2015 596(551first-yearstudentsand45anchorexaminees) Ver.1,2,3
2ndadministra0on
20Dec2015 644(575first-yearstudentsand69anchorexaminees) Ver.4,5
OutlineofpresentaFon
- Testspecifica0onPurposeTasktypesRa0ngscalesMarking
- Preliminaryresults- Futureplansandchallenges
Testspecifica0onhcps://kitspeakee.wordpress.com/testspec/Sampletesthcps://kitspeakee.wordpress.com/sampletest/
Testpurpose1
ToassesstowhatextenteachstudenthasacquiredthespeakingabilityinEnglishrequiredtoac0velypar0cipatein21stcenturyglobalsociety
Testpurpose2
Tosendposi0vewashbackmessages:- Speakingisasimportantastheothercorelanguageskills.Itis
notjustlimitedtodailyconversa0onbutisaseriouscommunica0onmodeessen0altosucceedinthe21stcentury.
- StudentsarelearningEnglishasalinguafrancaforcommunica0onbetweennon-na0vespeakersaswellaswithna0vespeakers.
- Studentsmustlearntobeconfidentusersoftheirexis0nglanguageresourcesatanystageoftheirlanguagedevelopment,freefromanexcessiveconcerntoconformtoNSnorms.
- Studentsmustbepreparedtospeakwithoutprepara0on,asspokeninterac0onisnormallyspontaneous.
Testpurpose3
Toevaluatethefeasibilityofincorpora0ngaspeakingtestintothecurrentEnglisheduca0onprogramme,anddevelopingaspeakingcomponentforanEnglishlanguageadmissionstesttoJapaneseuniversitypostgraduatecourses,withapossiblefutureapplica0ontoundergraduateentranceexamina0ons
16/07/06
2
Constructs
- Theabilitytoachieveagiventasku0lisingspokenlanguageproficiencyand21stcenturyskills(crea0vityandinnova0on,cri0calthinking,problemsolving,globalawareness)
- Confidentandfluentuseoflanguage
Tasktypes
KITtestSpec160402.docx page3
6. Tasktypes
Qno. Name Speech
sample Prompt 21stcenturyskills:Learnersshould…
Spokenlanguageproficiency:Learnersshould… Sampletask
Part11&2 Imagine 45
seconds Photo thinkcreativelyanddemonstrateoriginality speakcoherentlyandclearly Imaginewhotheowneroftheseshoesisand
whytheyarehere.3
Compare 45seconds
Twophotos
analysealternativesanddrawreasonableconclusion compare,decideandjustify
Whichofthesecarswouldyouprefertohave?Comparethecarsandexplainyourreasons.
Part24 Identify
differentvalues
45seconds
Audiodialogue&photos
understanddiversevaluesandperspectives
summariseandcontrastdifferentpointsofview
HowareSusan’sandKenji’sopinionsdifferent?[joinbigcompany/securityvsstartingownbusiness/excitinglife]
5 Takeposition
60seconds
Sameastask4
evaluateargumentsanddecideownposition stateandjustifyownposition Whichwayofthinkingdoyousupport?
Explainyourposition,givingexamples.6
Identifyproblem
45seconds
Audiodialogue&photos
interpretinformationtoIdentifyproblem describeproblem WhatistheproblemBillisfacing?[One
memberofthelabteamisn’tcollaborating]
7 Problemsolving
60seconds
Sameastask6 findsolutiontoproblem proposesolutiontoproblem IfyouwereBill,whatwouldyoudotosolve
theproblem?Part38
Planandorganise
60seconds,plusonerehearsal
identifyandorganizecomponentpartstomakeaplan
suggestaplanandseriesofstepstoachievegoal
Youwanttoorganizeavolunteergrouponcampustohelphomelesspeople.Identifythedifferentstepsyouwouldtakeandexplainhowyouwouldorganizethem.
9
Persuade
60seconds,plusonerehearsal
promoteandinfluence persuadebypresentingapositiveimageandmessage
Inaninterviewforascholarshipprogram,youareaskedtoexplainwhyyoushouldbeselected.Talkaboutyourpersonalachievementsandstrongpoints.
Part1
Ques0onno. 1&2 3
Name Imagine Compare
Speechsample 45seconds 45seconds
Prompt Photo Twophotos
21stcenturyskills:Learnersshould…
thinkcrea0velyanddemonstrateoriginality
analysealterna0vesanddrawreasonableconclusion
Spokenlanguageproficiency:Learnersshould…
speakcoherentlyandclearly compare,decideandjus0fy
Sampletask Imaginewhotheowneroftheseshoesisandwhytheyarehere.
Whichofthesecarswouldyouprefertohave?Comparethecarsandexplainyourreasons.
QuesFon1
Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.
QuesFon1
Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.
QuesFon1
Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.
16/07/06
3
QuesFon1
Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.
QuesFon1
Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.
QuesFon1
Imaginewhotheowneroftheseshoesisandwhytheyarehere.Talkfor45seconds.
Recording
Part2Ques0onno. 4 5 6 7
Name Iden0fydifferentvalues Takeposi0on Iden0fyproblem Problemsolving
Speechsample 45seconds 60seconds 45seconds 60seconds
Prompt Audiodialogue&photos Sameastask4
Audiodialogue&photos Sameastask6
21stcenturyskills:Learnersshould…
understanddiversevaluesandperspec0ves
evaluateargumentsanddecideownposi0on
interpretinforma0ontoIden0fyproblem
findsolu0ontoproblem
Spokenlanguageproficiency:Learnersshould…
summariseandcontrastdifferentpointsofview
stateandjus0fyownposi0on describeproblem proposesolu0onto
problem
Sampletask
HowareSusan’sandKenji’sopinionsdifferent?[joinbigcompany/securityvsstarEngownbusiness/exciEnglife]
Whichwayofthinkingdoyousupport?ExplainyourposiEon,givingexamples.
WhatistheproblemBillisfacing?[Onememberofthelabteamisn’tcollaboraEng]
IfyouwereBill,whatwouldyoudotosolvetheproblem?
Part3
Ques0onno. 8 9
Name Planandorganise Persuade
Speechsample 60seconds,plusonerehearsal 60seconds,plusonerehearsal
Prompt - -
21stcenturyskills:Learnersshould…
iden0fyandorganizecomponentpartstomakeaplan promoteandinfluence
Spokenlanguageproficiency:Learnersshould…
suggestaplanandseriesofstepstoachievegoal
persuadebypresen0ngaposi0veimageandmessage
Sampletask
Youwanttoorganizeavolunteergrouponcampustohelphomelesspeople.IdenEfythedifferentstepsyouwouldtakeandexplainhowyouwouldorganizethem.
Inaninterviewforascholarshipprogram,youareaskedtoexplainwhyyoushouldbeselected.Talkaboutyourpersonalachievementsandstrongpoints.
16/07/06
4
Tasktypes
KITtestSpec160402.docx page3
6. Tasktypes
Qno. Name Speech
sample Prompt 21stcenturyskills:Learnersshould…
Spokenlanguageproficiency:Learnersshould… Sampletask
Part11&2 Imagine 45
seconds Photo thinkcreativelyanddemonstrateoriginality speakcoherentlyandclearly Imaginewhotheowneroftheseshoesisand
whytheyarehere.3
Compare 45seconds
Twophotos
analysealternativesanddrawreasonableconclusion compare,decideandjustify
Whichofthesecarswouldyouprefertohave?Comparethecarsandexplainyourreasons.
Part24 Identify
differentvalues
45seconds
Audiodialogue&photos
understanddiversevaluesandperspectives
summariseandcontrastdifferentpointsofview
HowareSusan’sandKenji’sopinionsdifferent?[joinbigcompany/securityvsstartingownbusiness/excitinglife]
5 Takeposition
60seconds
Sameastask4
evaluateargumentsanddecideownposition stateandjustifyownposition Whichwayofthinkingdoyousupport?
Explainyourposition,givingexamples.6
Identifyproblem
45seconds
Audiodialogue&photos
interpretinformationtoIdentifyproblem describeproblem WhatistheproblemBillisfacing?[One
memberofthelabteamisn’tcollaborating]
7 Problemsolving
60seconds
Sameastask6 findsolutiontoproblem proposesolutiontoproblem IfyouwereBill,whatwouldyoudotosolve
theproblem?Part38
Planandorganise
60seconds,plusonerehearsal
identifyandorganizecomponentpartstomakeaplan
suggestaplanandseriesofstepstoachievegoal
Youwanttoorganizeavolunteergrouponcampustohelphomelesspeople.Identifythedifferentstepsyouwouldtakeandexplainhowyouwouldorganizethem.
9
Persuade
60seconds,plusonerehearsal
promoteandinfluence persuadebypresentingapositiveimageandmessage
Inaninterviewforascholarshipprogram,youareaskedtoexplainwhyyoushouldbeselected.Talkaboutyourpersonalachievementsandstrongpoints.
IniFalraFngscales(Usedinthe1stadministra0on,Jan2015)
Scorera0ng
Taskachievement(45%weigh0ng)
Communica0velanguageproficiency(45%weigh0ng)
Antudeandeffort(10%weigh0ng)
5
-taskisachievedeffec0vely-contribu0onsarefullyrelevantbuttheremaybesomerepe00on
-communicateseffec0vely-producesextendeducerancesusingarangeofgrammarandvocabularydespiteapossiblelackoffluencyoraccuracy-demonstratesappropriateuseofcohesivedevices
-speaksconfidentlyandwillingly(generallycalmand/orforceful,ac0ve,posi0ve,engaged,enthusias0c,expressive,firmandasser0ve)-makesabigefforttousehis/herlanguageresources
4 Between3and5 Between3and5 Between3and5
3
-taskisachievedonlypar0allyorinadequately-contribu0onsaremostlyrelevant
-communicatessuccessfullybutmaybelimitedbylackoffluencyoraccuracy-constructsuceranceslongerthanshortphrasesusingbasicgrammarandvocabularyonly-demonstrateslimiteduseofcohesivedevices
-speakswithsomeconfidencebutissome0mesnervous,hesitantorunwilling-makessomeefforttousehis/herlanguageresources
2 Between1and3 Between1and3 Between1and3
1 -taskisnotachieved-contribu0onsaremostlynotrelevant.
-ooenfailstocommunicateduetolackoffluencyoraccuracy-producesshortuceranceswithwordsorphrasesonly-doesnotusecohesivedevices
-lacksconfidenceandunengaged-makeslicleefforttousehis/herlanguageresources
0 Nocontribu0onatall doesnotcommunicateatall Noeffortatall
RevisedraFngscales(Usedinthe2ndadministra0on,Dec2015)
Scorera0ng Taskachievement(80%weigh0ng) Taskdelivery(20%weigh0ng)
5 -taskisachievedwithsa0sfactorysuppor0ngdetail
-speaksfluentlyenoughtobecomprehensibleandwithsomeconfidence-given0meiswelluseddespitesomehesita0onorrepe00on
4 Between3and5 Between3and5
3 -taskispar0allyachieved,orisachievedwithminimalsuppor0ngdetail
-justfluentenoughtobecomprehensiblemostofthe0mebutmaylackconfidence-given0meisnoteffec0velyusedbecauseoffrequenthesita0onorrepe00on
2 Between1and3 Between1and3
1 -somerelevantwordsbuttaskisnotachieved
-isnotcomprehensiblemostofthe0me
0 -norelevantcontribu0on -nocomprehensiblecontribu0on
Stepsforensuringwashback
- No0fiedstudentsofadministra0onofspeakingtestatfirst-yearstudentorienta0on
- VisitedeverycompulsoryEnglishclasstoexplainpurposeofthetest,tasktypes,ra0ngscales,andhowtoprepareforthetest
- DistributedJapanesetransla0onofinforma0onaboutthetestandmadeitavailableonwebsite
16/07/06
5
Marking
1stadministra0on 2ndadministra0on
Testdate 20-22Jan2015 19Dec2015
Numberofresponses 9ques0ons×670examinees=6,030responses
9ques0ons×713examinees=6,417responses
Raters7na0ve-speakers7non-na0vespeakers(Japanese-speakers)
9na0ve-speakers9non-na0vespeakers(inthePhillipines)
Ra0ngprocess*Doublemarking
Oneratermarkedallthenineresponsesgivenbyoneexaminee.
Eachratermarkedalltheresponsesforonetaskonly.
Dura0onofmarking 2weeks 2weeks
!Ver0calscaleforoveralldata
Exactmatches EsFmatedmatches
35.6% 33.3%
Exactmatches EsFmatedmatches
41.3% 39.7%
Inter-rateragreement(NSandJapanese)
Inter-rateragreement(PhilippinesandJapanese)
(Es0matedbyFacets(Linacre,1989))
11-1
Standardizedscoredistribu0onbasedonMany-facetItemResponseTheory(MIRT)
11-2
KeyfeaturesofKITSpeakingTest
- ELF- Focusonwashback- Tailoredtothestudentcontext- Sta0s0calvalida0on- CBT(testadministra0onandra0ngsystems)
ELFfeatures
- Purpose- Constructs- Tasktypes- Dialoguerecordings- Ra0ngscales- Ra0ngprocess
Futurechallenges
- Ra0ngscales- Ratertraining- Feedbackform,scorereport- Linkteaching/learningandtes0ng- Opera0onalissuesforentranceexams
16/07/06
6
Thankyou!
Gettheslideshere:hcps://goo.gl/6E5KsE＊
YumiHato:[email protected]:[email protected]
KatsunoriKanzawa:[email protected]:[email protected]