işın demirşahin, İhsan yalçınkaya and deniz...
TRANSCRIPT
IşınDemirşahin,İhsanYalçınkayaandDenizZeyrekMiddleEastTechnicalUniversity,Ankara,Turkey
TurkishDiscourseBank(TDB) 500000‐word‐subcorpusofMetuTurkishCorpus(Sayetal.,2002)
AnnotatedfordiscourserelaFonsinthestyleofPennDiscourseTreeBank(Prasadetal.,2008) DiscourseconnecFves Arguments‐>Abstractobjects(Asher,1993)
7/12/12 26thLinguisFcAnnotaFonWorkshop
TurkishDiscourseBank(TDB) DiscourseAnnotaFonToolforTurkish(Aktaş,etal.,2010)
Version1.0: 147connecFves,8482relaFons hZp://medid.ii.metu.edu.tr
7/12/12 6thLinguisFcAnnotaFonWorkshop 3
AnnotaEonProcedureintheTDB
IndependentAnnotaFon
Searchfortokens
Eliminatenon‐discourseconnecFves
Annotatespans(Conn,Arg1,Arg2,Mod,Shared,Supps)
7/12/12 46thLinguisFcAnnotaFonWorkshop
IniEalAnnotaEonProcedureintheTDB
Agreementprocedure
LocaFngdisagreements
AdjudicaFon
Groupdiscussion
7/12/12 76thLinguisFcAnnotaFonWorkshop
IniEalAnnotaEonProcedureintheTDB
AnnotaFonguidelines
IndependentannotaFons
Agreementprocedure
ModificaFonofguidelines
Proofprocedure
GoldStandard7/12/12 86thLinguisFcAnnotaFonWorkshop
CommonDivergencesamongIndependentAnnotaEons
Physicalerrors Wordboundaries
SpacesandpunctuaFon PairsofpunctuaFonmarks
7/12/12 6thLinguisFcAnnotaFonWorkshop 9
Başkakimseolmadığındanikikadınındayüzüaçık,.
Sincetherewasnooneelse,thefacesofbothwomenwereunveiled.
“Gidelim”dedim,amadışarıdanyeniseslergeldi.
“Let’sgo”Isaid,butnewvoicescamefromoutside.
CommonDivergencesamongIndependentAnnotaEons
Ambiguouscases SamerelaFon,withsomedifferences
DifferentrelaFons
7/12/12 6thLinguisFcAnnotaFonWorkshop 10
Saltgülmekiçingelmişlerdi.
Theycametojustlaugh.
[Salt]gülmekiçingelmişlerdi.
Theycame[only]tolaugh.
Usumi için ilk tören, Türkiye Gazeteciler CemiyeG (TGC) önünde düzenlendi.TGCBaşkanıOrhanErinç,konuşmasındaUsumi'ninyokluğunuhissedeceklerinivurguladı.Usumi'nincenazesi[daha]sonraSultanahmetCamii'negötürüldü.
ThefirstceremonyforUsumiwasarrangedinfrontoftheAssociaGonoftheJournalistsofTurkey(TGC).OrhanErinç,thechairmantheTGC,emphasizedthatUsumiwouldbemissed.Then,theUsumi’sfuneralwasmovedtotheSultanAhmedMosque.
CommonDivergencesamongIndependentAnnotaEons
Missing/invalidannotaFons Simpleoverlook
DisagreementaboutthediscoursefuncFon
7/12/12 6thLinguisFcAnnotaFonWorkshop 11
Yarındemedim,ancakhafabaşınahazırolur.
Ididn’tsayitwouldbetomorrow,itwon’tbereadyunFlthefirstdayoftheweek.
Ididn’tsayitwouldbetomorrow,butitwillbereadyonthefirstdayoftheweek.
NotD‐Conn
D‐Conn
KızınsaçlarısiyahvekıvırcıkU.
Thegirl’shairwasblackandcurly.
PairProgramming Twoprogrammersworkasone(Williams,etal,2000) Driver Navigator
Jointownership Credit Responsibility
Therolesareswitchedperiodically
7/12/12 6thLinguisFcAnnotaFonWorkshop 12
PairProgramming
7/12/12 6thLinguisFcAnnotaFonWorkshop 13
PROS CONSMoreefficientuseofcodinghours
Increasedprogramminghours(15%‐60%)Increasedperformance
(PairJelling)
DecreasedcommunicaFonFmebetweenunits
FasterproducFon Increasedman‐hours
PairAnnotaEonandPairProgramming
SimilariFes Thedriverandthenavigator Switchingroles
Difference:Mixedapproach Oneindependentannotator Onepairofannotators
7/12/12 6thLinguisFcAnnotaFonWorkshop 14
ObservedBenefitsofPairAnnotaEon
Higherinter‐annotatoragreement Lessphysicalerrors
FasterannotaFonofambiguouscases IncreaseddiscussionduringannotaFon‐>decreaseddiscussionduringagreement
UFlizaFonofnotesfield
IncreasedmoFvaFon
7/12/12 6thLinguisFcAnnotaFonWorkshop 15
AgreementStaEsEcs IndependentannotaFonvs.PairannotaFon
Pair‐wiseaveragedKfor3individualannotatorsinIA–individualagainstindividualVSKforpairvs.individualinPA
7/12/12 6thLinguisFcAnnotaFonWorkshop 16
ConnecEve IA–Arg1 PA–Arg1 IA–Arg2 PA–Arg2
ama‘but’ 0.832 0.956 0.901 0.969sonra‘afer,later’ 0.820 0.889 0.902 0.953ve‘and’ 0.692 0.945 0.791 0.964yada‘or’ 0.843 0.939 0.974 0.973
AgreementStaEsEcs IndividualannotatorvsPairannotator
Individualannotatorvs.goldstandardKVSPairannotatorvs.goldstandardK
7/12/12 6thLinguisFcAnnotaFonWorkshop 17
ConnecEve Ind–Arg1 Pair–Arg1 Ind–Arg2 Pair–Arg2
aslında‘actually’ 0.766 0.937 0.889 0.984halde‘despite’ 0.834 0.973 0.898 1.000nedeniyle‘becauseof‘ 0.905 0.937 0.984 0.984nedenle‘forthisreason’ 0.952 1.000 0.987 1.000ötürü‘dueto’ 1.000 1.000 0.907 0.953yüzden‘becauseofthis’ 0.916 0.992 0.983 1.000
IssuesinPairAnnotaEon LesssetsofindependentannotaFons Concernaboutnavigator’scontribuFons
Switchingroles Periodicalfeedback
BiasedannotaFons KeepingrecordofiniFalintuiFonsanddisagreementwithinpair
Biasedagreementprocess TreaFngthepairasasingleenFty
AllsoluFonsareprojectspecific
7/12/12 6thLinguisFcAnnotaFonWorkshop 18
PairAnnotaEon
7/12/12 6thLinguisFcAnnotaFonWorkshop 19
PROS CONSSignificantincreaseofinter‐annotatoragreements
Possibilitythatthepair’sannotaFoncouldbebiased
Significantincreaseofannotatorvsgoldstandardagreements
Possibilitythatthepaircoulddominatetheagreementprocess
Fasteragreementprocess OnelesssetofannotaFons
FutureWork QuanFtaFveevaluaFonoftheincreaseinannotaFonhours
Inter‐annotatoragreementwithdifferentpairs
Inter‐annotatoragreementbetweentwopairsofannotators.
7/12/12 6thLinguisFcAnnotaFonWorkshop 20
Acknowledgements WegratefullyacknowledgethesupportoftheTurkishScienFficandTechnologicalResearchCouncilofTurkey(TUBITAK)andMETUScienFficResearchFund(no.BAP‐07‐04‐2011‐005)whiledevelopingtheTurkishDiscourseBank.
WethanktheAsianFederaFonofNaturalLanguageProcessing(AFNLP)andtheNaFonalScienceFoundaFon(NSF)forsupporFngIşınDemirşahinwithstudentgrantsfortravel,registraFonandaccommodaFon.
7/12/12 6thLinguisFcAnnotaFonWorkshop 21
References NicholasAsher.1993.ReferencetoAbstractObjectsinDiscourse.KluwerAcademicPublishers. BerfinAktaş,CemBozşahin,DenizZeyrek.2010.DiscourseRelaFonConfiguraFonsinTurkishandanAnnotaFon
Environment.LAWIV‐TheFourthLinguisGcAnnotaGonWorkshop.Uppsala,Sweden,July2010. KentBack.2000.ExtremeProgrammingExplained:EmbrassChange.AddisonWesleyLongman,Reading,Mass. Frederick,P.J.Brooks.1975.TheMythicalMan‐Month.Addison‐Wesley,Reading,Mass. TomDeMarcoandTimothyLister.1977.Peopleware.DorsetHouse,NewYork. EleniMiltsakaki,RashmiPrasad,AravindJoshi,BonnieWebber.2004.AnnotaFngDiscourseConnecFvesand
TheirArguments.HLT/NAACLWorkshoponFronGersinCorpusAnnotaGon.Boston,MA.May2004. BeataBeigmanKlebanovandEyalBeigman.2008.FromAnnotatorAgreementtoNoiseModels.ComputaGonal
LinguisGcs,34(3):495–503. RashmiPrasad,NikhilDinesh,AlanLee,EleniMiltsakaki,LivioRobaldo,AravindJoshi,BonnieWebber.2008.The
PennDiscourseTreebank2.0.LREC’08‐ThesixthinternaGonalconferenceonLanguageResourcesandEvaluaGon.Marrakech,Morocco,May2008.
DennisReidsmaandJeanCarleZa.2008.Reliabilitymeasurementwithoutlimit.ComputaGonalLinguisGcs,34(3):319–326.
BilgeSayandDenizZeyrekandKemalOflazerandUmutOzge.2002.DevelopmentofaCorpusandaTreebankforPresent‐dayWriZenTurkish.ICTL2002‐11thInternaGonalConferenceonTurkishLinguisGcs.Famagusta,TRNC,August2002.
LaurieWilliams,RobertR.Kessler,WardCunningham,RonJeffries.2000.StrengtheningtheCaseforPairProgramming.IEEESohware,July/August2000:19–25.
LaurieWilliamsandRobertR.Kessler.2000.AllIReallyNeedtoKnowaboutPairProgrammingILearnedInKindergarten,CommunicaGonsoftheACM,43(5):108–114.
LaurieWilliamsandRobertR.Kessler.2003.PairProgrammingIlluminated.AddisonWesley,Reading,MassachuseZs.
DenizZeyrek,IşınDemirşahin,AyışığıSevdik‐Çallı,HaleÖgelBalaban,İhsanYalçınkayaandÜmitDenizTuran.2010.TheAnnotaFonSchemeoftheTurkishDiscourseBankandanEvaluaFonofInconsistentAnnotaFons.LAWIV‐TheFourthLinguisGcAnnotaGonWorkshopUppsala,Sweden,July2010.
7/12/12 6thLinguisFcAnnotaFonWorkshop 22
AddiEonalReference Yalçınkaya,İhsan.2010.AnInter‐AnnotatorAgreementMethodologyfortheTurkishDiscourseBank.UnpublishedMSThesis,MiddleEastTechnicalUniversity.
7/12/12 6thLinguisFcAnnotaFonWorkshop i
IniEalAnnotaEonProcedureintheTDB
Onesearchtoken‐>MulFpleconnecFves Halde
‐DIğIhalde‘although,despite’ ohalde‘then,inthatcase’ şuhalde‘inthecurrentsituaFon,inthisspecificcase’
aksihalde,‘otherwise’ birhalde‘insuchamannerthat’
7/12/12 ii6thLinguisFcAnnotaFonWorkshop
IniEalAnnotaEonProcedureintheTDB
Ogünakşamakadarneyapacağımıbilmezbirhaldedolaş}meviniçinde.
IwalkedaroundthehouseFlleveningthatday,notknowingwhattodo.
DoğuBeyazıt'dagecelediğimizhaldebirdünyaşaheseriolanİshakPaşamedresesinigöremedenAnkara'yadöndük.
AlthoughwespentthenightinDoğuBeyazıt,wereturnedtoAnkarawithoutseeingtheİshakPaşaMedresseh,whichisamasterpiece.
FeyziBeyböylebirdurumdamebusluktanisGfaedeceğini,aksihalde[de]FalihRı`ıBey'inisEfaetmesigerekEğinibelirtmiş.
Mr.FevzistatedthatinsuchasituaGonhewouldresignfromparliamentmembership,otherwiseMr.FalihRı`ıwouldhavetoresign.
Notannotated!
7/12/12 iii6thLinguisFcAnnotaFonWorkshop