işın demirşahin, İhsan yalçınkaya and deniz...

IşınDemirşahin,İhsanYalçınkayaandDenizZeyrekMiddleEastTechnicalUniversity,Ankara,Turkey

disin@metu.edu.tr,yalcinkaya.ihsan@gmail.com,dezeyrek@metu.edu.tr

TurkishDiscourseBank(TDB) 500000‐word‐subcorpusofMetuTurkishCorpus(Sayetal.,2002)

 AnnotatedfordiscourserelaFonsinthestyleofPennDiscourseTreeBank(Prasadetal.,2008) DiscourseconnecFves Arguments‐>Abstractobjects(Asher,1993)

7/12/12 26thLinguisFcAnnotaFonWorkshop

TurkishDiscourseBank(TDB) DiscourseAnnotaFonToolforTurkish(Aktaş,etal.,2010)

 Version1.0:  147connecFves,8482relaFons  hZp://medid.ii.metu.edu.tr

7/12/12 6thLinguisFcAnnotaFonWorkshop 3

AnnotaEonProcedureintheTDB

 IndependentAnnotaFon

Searchfortokens

Eliminatenon‐discourseconnecFves

Annotatespans(Conn,Arg1,Arg2,Mod,Shared,Supps)

7/12/12 46thLinguisFcAnnotaFonWorkshop

IniEalAnnotaEonProcedureintheTDB

 Agreementprocedure

LocaFngdisagreements

AdjudicaFon

Groupdiscussion

7/12/12 76thLinguisFcAnnotaFonWorkshop

IniEalAnnotaEonProcedureintheTDB

AnnotaFonguidelines

IndependentannotaFons

Agreementprocedure

ModificaFonofguidelines

Proofprocedure

GoldStandard7/12/12 86thLinguisFcAnnotaFonWorkshop

CommonDivergencesamongIndependentAnnotaEons

 Physicalerrors Wordboundaries

 SpacesandpunctuaFon PairsofpunctuaFonmarks

7/12/12 6thLinguisFcAnnotaFonWorkshop 9

Başkakimseolmadığındanikikadınındayüzüaçık,.

Sincetherewasnooneelse,thefacesofbothwomenwereunveiled.

“Gidelim”dedim,amadışarıdanyeniseslergeldi.

“Let’sgo”Isaid,butnewvoicescamefromoutside.

CommonDivergencesamongIndependentAnnotaEons

 Ambiguouscases SamerelaFon,withsomedifferences

 DifferentrelaFons

7/12/12 6thLinguisFcAnnotaFonWorkshop 10

Saltgülmekiçingelmişlerdi.

Theycametojustlaugh.

[Salt]gülmekiçingelmişlerdi.

Theycame[only]tolaugh.

Usumi için ilk tören, Türkiye Gazeteciler CemiyeG (TGC) önünde düzenlendi.TGCBaşkanıOrhanErinç,konuşmasındaUsumi'ninyokluğunuhissedeceklerinivurguladı.Usumi'nincenazesi[daha]sonraSultanahmetCamii'negötürüldü.

ThefirstceremonyforUsumiwasarrangedinfrontoftheAssociaGonoftheJournalistsofTurkey(TGC).OrhanErinç,thechairmantheTGC,emphasizedthatUsumiwouldbemissed.Then,theUsumi’sfuneralwasmovedtotheSultanAhmedMosque.

CommonDivergencesamongIndependentAnnotaEons

 Missing/invalidannotaFons Simpleoverlook

 DisagreementaboutthediscoursefuncFon

7/12/12 6thLinguisFcAnnotaFonWorkshop 11

Yarındemedim,ancakhafabaşınahazırolur.

Ididn’tsayitwouldbetomorrow,itwon’tbereadyunFlthefirstdayoftheweek.

Ididn’tsayitwouldbetomorrow,butitwillbereadyonthefirstdayoftheweek.

NotD‐Conn

D‐Conn

KızınsaçlarısiyahvekıvırcıkU.

Thegirl’shairwasblackandcurly.

PairProgramming Twoprogrammersworkasone(Williams,etal,2000) Driver Navigator

 Jointownership Credit Responsibility

 Therolesareswitchedperiodically

7/12/12 6thLinguisFcAnnotaFonWorkshop 12

PairProgramming

7/12/12 6thLinguisFcAnnotaFonWorkshop 13

PROS CONSMoreefficientuseofcodinghours

Increasedprogramminghours(15%‐60%)Increasedperformance

(PairJelling)

DecreasedcommunicaFonFmebetweenunits

FasterproducFon Increasedman‐hours

PairAnnotaEonandPairProgramming

 SimilariFes Thedriverandthenavigator Switchingroles

 Difference:Mixedapproach Oneindependentannotator Onepairofannotators

7/12/12 6thLinguisFcAnnotaFonWorkshop 14

ObservedBenefitsofPairAnnotaEon

 Higherinter‐annotatoragreement Lessphysicalerrors

 FasterannotaFonofambiguouscases  IncreaseddiscussionduringannotaFon‐>decreaseddiscussionduringagreement

 UFlizaFonofnotesfield

 IncreasedmoFvaFon

7/12/12 6thLinguisFcAnnotaFonWorkshop 15

AgreementStaEsEcs  IndependentannotaFonvs.PairannotaFon

 Pair‐wiseaveragedKfor3individualannotatorsinIA–individualagainstindividualVSKforpairvs.individualinPA

7/12/12 6thLinguisFcAnnotaFonWorkshop 16

ConnecEve IA–Arg1 PA–Arg1 IA–Arg2 PA–Arg2

ama‘but’ 0.832 0.956 0.901 0.969sonra‘afer,later’ 0.820 0.889 0.902 0.953ve‘and’ 0.692 0.945 0.791 0.964yada‘or’ 0.843 0.939 0.974 0.973

AgreementStaEsEcs  IndividualannotatorvsPairannotator

  Individualannotatorvs.goldstandardKVSPairannotatorvs.goldstandardK

7/12/12 6thLinguisFcAnnotaFonWorkshop 17

ConnecEve Ind–Arg1 Pair–Arg1 Ind–Arg2 Pair–Arg2

aslında‘actually’ 0.766 0.937 0.889 0.984halde‘despite’ 0.834 0.973 0.898 1.000nedeniyle‘becauseof‘ 0.905 0.937 0.984 0.984nedenle‘forthisreason’ 0.952 1.000 0.987 1.000ötürü‘dueto’ 1.000 1.000 0.907 0.953yüzden‘becauseofthis’ 0.916 0.992 0.983 1.000

IssuesinPairAnnotaEon LesssetsofindependentannotaFons Concernaboutnavigator’scontribuFons

  Switchingroles  Periodicalfeedback

 BiasedannotaFons  KeepingrecordofiniFalintuiFonsanddisagreementwithinpair

 Biasedagreementprocess TreaFngthepairasasingleenFty

 AllsoluFonsareprojectspecific

7/12/12 6thLinguisFcAnnotaFonWorkshop 18

PairAnnotaEon

7/12/12 6thLinguisFcAnnotaFonWorkshop 19

PROS CONSSignificantincreaseofinter‐annotatoragreements

Possibilitythatthepair’sannotaFoncouldbebiased

Significantincreaseofannotatorvsgoldstandardagreements

Possibilitythatthepaircoulddominatetheagreementprocess

Fasteragreementprocess OnelesssetofannotaFons

FutureWork QuanFtaFveevaluaFonoftheincreaseinannotaFonhours

 Inter‐annotatoragreementwithdifferentpairs

 Inter‐annotatoragreementbetweentwopairsofannotators.

7/12/12 6thLinguisFcAnnotaFonWorkshop 20

Acknowledgements WegratefullyacknowledgethesupportoftheTurkishScienFficandTechnologicalResearchCouncilofTurkey(TUBITAK)andMETUScienFficResearchFund(no.BAP‐07‐04‐2011‐005)whiledevelopingtheTurkishDiscourseBank.

 WethanktheAsianFederaFonofNaturalLanguageProcessing(AFNLP)andtheNaFonalScienceFoundaFon(NSF)forsupporFngIşınDemirşahinwithstudentgrantsfortravel,registraFonandaccommodaFon.

7/12/12 6thLinguisFcAnnotaFonWorkshop 21

References  NicholasAsher.1993.ReferencetoAbstractObjectsinDiscourse.KluwerAcademicPublishers.  BerfinAktaş,CemBozşahin,DenizZeyrek.2010.DiscourseRelaFonConfiguraFonsinTurkishandanAnnotaFon

Environment.LAWIV‐TheFourthLinguisGcAnnotaGonWorkshop.Uppsala,Sweden,July2010.  KentBack.2000.ExtremeProgrammingExplained:EmbrassChange.AddisonWesleyLongman,Reading,Mass.  Frederick,P.J.Brooks.1975.TheMythicalMan‐Month.Addison‐Wesley,Reading,Mass.  TomDeMarcoandTimothyLister.1977.Peopleware.DorsetHouse,NewYork.  EleniMiltsakaki,RashmiPrasad,AravindJoshi,BonnieWebber.2004.AnnotaFngDiscourseConnecFvesand

TheirArguments.HLT/NAACLWorkshoponFronGersinCorpusAnnotaGon.Boston,MA.May2004.  BeataBeigmanKlebanovandEyalBeigman.2008.FromAnnotatorAgreementtoNoiseModels.ComputaGonal

LinguisGcs,34(3):495–503.  RashmiPrasad,NikhilDinesh,AlanLee,EleniMiltsakaki,LivioRobaldo,AravindJoshi,BonnieWebber.2008.The

PennDiscourseTreebank2.0.LREC’08‐ThesixthinternaGonalconferenceonLanguageResourcesandEvaluaGon.Marrakech,Morocco,May2008.

  DennisReidsmaandJeanCarleZa.2008.Reliabilitymeasurementwithoutlimit.ComputaGonalLinguisGcs,34(3):319–326.

  BilgeSayandDenizZeyrekandKemalOflazerandUmutOzge.2002.DevelopmentofaCorpusandaTreebankforPresent‐dayWriZenTurkish.ICTL2002‐11thInternaGonalConferenceonTurkishLinguisGcs.Famagusta,TRNC,August2002.

  LaurieWilliams,RobertR.Kessler,WardCunningham,RonJeffries.2000.StrengtheningtheCaseforPairProgramming.IEEESohware,July/August2000:19–25.

  LaurieWilliamsandRobertR.Kessler.2000.AllIReallyNeedtoKnowaboutPairProgrammingILearnedInKindergarten,CommunicaGonsoftheACM,43(5):108–114.

  LaurieWilliamsandRobertR.Kessler.2003.PairProgrammingIlluminated.AddisonWesley,Reading,MassachuseZs.

  DenizZeyrek,IşınDemirşahin,AyışığıSevdik‐Çallı,HaleÖgelBalaban,İhsanYalçınkayaandÜmitDenizTuran.2010.TheAnnotaFonSchemeoftheTurkishDiscourseBankandanEvaluaFonofInconsistentAnnotaFons.LAWIV‐TheFourthLinguisGcAnnotaGonWorkshopUppsala,Sweden,July2010.

7/12/12 6thLinguisFcAnnotaFonWorkshop 22

AddiEonalReference Yalçınkaya,İhsan.2010.AnInter‐AnnotatorAgreementMethodologyfortheTurkishDiscourseBank.UnpublishedMSThesis,MiddleEastTechnicalUniversity.

7/12/12 6thLinguisFcAnnotaFonWorkshop i

IniEalAnnotaEonProcedureintheTDB

 Onesearchtoken‐>MulFpleconnecFves Halde

 ‐DIğIhalde‘although,despite’ ohalde‘then,inthatcase’ şuhalde‘inthecurrentsituaFon,inthisspecificcase’

 aksihalde,‘otherwise’ birhalde‘insuchamannerthat’

7/12/12 ii6thLinguisFcAnnotaFonWorkshop

IniEalAnnotaEonProcedureintheTDB

Ogünakşamakadarneyapacağımıbilmezbirhaldedolaş}meviniçinde.

IwalkedaroundthehouseFlleveningthatday,notknowingwhattodo.

DoğuBeyazıt'dagecelediğimizhaldebirdünyaşaheseriolanİshakPaşamedresesinigöremedenAnkara'yadöndük.

AlthoughwespentthenightinDoğuBeyazıt,wereturnedtoAnkarawithoutseeingtheİshakPaşaMedresseh,whichisamasterpiece.

FeyziBeyböylebirdurumdamebusluktanisGfaedeceğini,aksihalde[de]FalihRı`ıBey'inisEfaetmesigerekEğinibelirtmiş.

Mr.FevzistatedthatinsuchasituaGonhewouldresignfromparliamentmembership,otherwiseMr.FalihRı`ıwouldhavetoresign.

Notannotated!

7/12/12 iii6thLinguisFcAnnotaFonWorkshop