AVENUE Mapudungun

Download AVENUE Mapudungun

Post on 13-Jan-2016

33 views

Category:

Documents

8 download

Embed Size (px)

DESCRIPTION

Data Collection and Analysis of Mapudungun Morphology for Spelling Correction. Christian Monson, Lori Levin, Rodolfo Vega, Ralf Brown, Ariadna Font Llitjos, Alon Lavie, Jaime Carbonell, Eliseo Caulef, Rosendo Huisca. - PowerPoint PPT Presentation

TRANSCRIPT

<ul><li><p>Data Collection and Analysis of Mapudungun Morphology for Spelling Correction</p><p>Christian Monson, Lori Levin, Rodolfo Vega, Ralf Brown, Ariadna Font Llitjos, Alon Lavie, Jaime Carbonell, Eliseo Caulef, Rosendo Huisca</p></li><li><p>AVENUE MapudungunInstituto de Estudios IndgenasUniversidad de La Frontera, Temuco, ChilePrograma de Educacin Intercultural BilingeMinistry of Education (Mineduc), ChileLanguage Technologies Institute Carnegie Mellon University, USA</p></li><li><p>Goals of AVENUE MapudungunMulticultural and Bilingual Education, Mineduc, ChileAVENUE Project, CMU, USANLP tools for bilingual education: On-line dictionaryBilingual CorpusSpelling checkerBasic skills taught in Spanish and mother tongueUse of technology and networking even in rural areasNLP tools for languages with low resourcesMachine learning of morphology and translation rules</p></li><li><p>OutlineOverview of Mapudungun languagePlan for on-line dictionaryProgress on dictionaryPlan for spelling checkerProgress on spelling checker</p></li><li><p>MapudungunMapuche peopleAround 900,000 Chile and ArgentinaAgglutinative/PolysyntheticUp to 36 suffix slots (Smeets, 1989)Typical verb has five or six suffixesNoun incorporationNoun goes immediately after the verb stemVstem+(noun)+(suffixes)+last-suffixLast suffix for finite verb is mood and person/number of agent or patient, Last suffix for non-finite verb is nominalization or adverbializationOther suffixes include aspect, negation, inversive, etc. </p></li><li><p>Examples of Mapudungun verbsAmu -ke-yngngo -habitual-3plIndicThey (usually) go</p><p>Ngtrmtu -a-lucall -fut-adverbWhile calling (tomorrow), </p><p>nentu-ma-nge-ymiextract-mal-pass-2sgIndicYou were extracted (on me)</p><p>ngtramka-me-a-fi-tell-loc-fut-3obj-1sgIndicI will tell her (away)</p></li><li><p>Plans for Dictionary (Mineduc)Tri-lingual (Spanish-Mapudungun-English); Pronunciation for each word for each languageExample of use for each Mapudungun wordSpecific users can exchange suggestions and alternate pronunciationsTeachers and students of schools in the PEIB/Orgenes program www.origenes.cl Web-based, using Flashbased on shared lessons plans and network communicationsVocabularyFrom the come from the corpus of spoken MapudungunFrom the Chilean curriculum for first four years of schoolFrom the informatics domainUser interface will be designed by Mineduc</p></li><li><p>Corpus of spoken Mapudungun170 hours of speech120 hours: Nguluche dialect30 hours: Lafkenche dialect20 hours: Pewenche dialect0 hours: Williche dialectDifferent and more endangeredMapuche interviewer and intervieweesDialogues about health problems treated by doctor or traditional healer.Recorded with DAT recorder Some recordings are poor qualitySome high enough quality for training a speech recognizerTranscribed using TransEdit Translated into Spanish by native speaker of Mapudungun</p></li><li><p>Examples from Mapudungun-Spanish corpusnmlch-nmjm1_x_0405_nmjm_00:M:no ptokovilu kay koC:no, si me lo tomaba con agua</p><p>M:chumgechi ptokoki femuechi ptokon pu C:como se debe tomar, me lo tom pus</p><p>nmlch-nmjm1_x_0406_nmlch_00:M:ChengewerkelafuymirkeC:Ya no estabas como gente entonces!</p></li><li><p>Progress on DictionaryAround 3000 Mapudungun words (stems and fully inflected forms) Spanish translation of the wordSentence from the corpus of spoken Mapudungun containing the word formSpanish translation of the sentence, and Reference into the corpus of spoken Mapudungun identifying the specific cited sentenceFor 1600 wordssegmentation of the word into morphemesgloss for each morphemeStored as a Word file with delimiters between fields.Can be easily converted to other formats</p></li><li><p>Examples from DictionaryLichi: .? . / /. leche. translationFeychi lichi, chem lichingey? example(Esta leche qu leche es?) translationnmlch-nmfhp1_x_0051_nmlch_00. Ec/Rh/Fc. Ec/ Rh02-01-03. index</p></li><li><p>Examples from DictionaryKmeknueymu: kme-knu-eymu. segmentationbien-quedar-l(ella).a.ti .? . / /. glosste ha dejado muy bien. translationKa kmeknueymu tati. example(Y te ha dejado muy bien). translationnmlch-nmpll1_x_0070_nmlch_00. EC/RH03-02-03. index</p></li><li><p>Examples from DictionaryMongeperkelayan: monge-pe-rke-la-y-a-n. segmentationsanar-tal.vez-acaso-no-0-futuro-yo .? . / /. glossno mejorar tal vez. translationFeytfachi operalayaymi, operaeliyu l'ayaymi" pieneu. "Mongeperkelayan may" pin. Fey l'awen'tueneu, l'awen'tueneu; fey ka tripantun. example("Esta vez no te vas a operar, si te opero te vas a morir" me dijo. "No mejorar tal vez, entonces", dije. Entonces me medicin, me medicin; entonces tambin estuve un ao). translationnmlch-nmpll1_x_0042_nmpll_00. Ec/Rh/Fc. Ec/ Rh23-12-02 index</p></li><li><p>Plans for spelling checkerGoal: identify misspellings even for morphologically complex words. We dont have a morphological analyzer Mapugungun speakers dont know computational linguisticsWe dont know MapudungunCurrently training a field linguist from Argentina (Roberto Aranovich) in computational linguisticsResearch on automated morphology learning (Christian Monson)We want the spelling checker to be compatible with a major word processor.Using MySpell and OpenOffice</p></li><li><p>MySpellOpen-source, standalone version of OpenOffice.org spell-checkerFunctional equivalent of Unix 'ispell'Data files specify stems and classes of affixeseach base-form word specifies valid affix classescan condition applicability based on characters in base-form worde.g. English plurals formed with -es if word ends in -chcan modify base form prior to adding affixe.g. change -y to -ie before adding -sLimitation: at most one prefix and one suffix can be applied to each base form</p></li><li><p>Plans for Spelling CheckerMySpell for MapudungunExample of full segmentationMongeperkelayan monge-pe-rke-la-y-a-n. no mejorar tal vez.Example of segmentation for MySpellmonge stemperkelayan suffix string</p></li><li><p>Progress on Spelling CheckerStep 1: Devise spelling conventionsThere are competing standards for Mapudungun spellingFirst version of spelling checker: AVENUE Mapudungun spelling standards by Caulef, Huisca, Painequeo, and CarrascoStep 2: Get a list of correctly spelled words, according to the conventions.Currently have correct spelling for the 70,000 most frequent words from the corpus</p></li><li><p>Progress on Spelling Checkermost frequent 70,000 words corrected by handFrequency RankTranscribed Word FormSpelling Corrected Word Form..103 feli feley pichikeche pichikeche kmey kmey10,001 chumkunual chumknuael10,002 puedelafuy puedelafuy10,003 tulayin tulayi10,004 kimngepelay kimngepelayCan we use this list instead of stemming?</p></li><li><p>The bad news </p></li><li><p>Progress on Spelling CheckerStep 3: Iteration of stem/suffix boundariesStart with 1600 segmented words from the dictionaryIdentify the suffix stringsFor the next most frequent 1000 wordsIf the word ends in a known suffix string, insert a stem/suffix boundaryOversegments because we dont check that the remaining stem is known after the suffix string is removedNative speakers correct the boundaries333 had to be correctedTwo more iterationsNext most frequent 3000 words (579 were wrong)Next most frequent 5000 words (1175 were wrong)Results in 9000 words with correct stem/suffix boundaries</p></li><li><p>Effect of stemming on number of typesIf the suffix string and the stem are in the list of 9000 correctly segmented words, treat it as an instance of the stem. Otherwise, treat it as a new type. </p><p>Chart1</p><p>11</p><p>22</p><p>33</p><p>44</p><p>55</p><p>66</p><p>77</p><p>88</p><p>99</p><p>1010</p><p>1111</p><p>1212</p><p>1212</p><p>1313</p><p>1414</p><p>1515</p><p>1616</p><p>1616</p><p>1717</p><p>1818</p><p>1818</p><p>1919</p><p>2020</p><p>2121</p><p>2222</p><p>2323</p><p>2424</p><p>2525</p><p>2626</p><p>2626</p><p>2727</p><p>2828</p><p>2828</p><p>2929</p><p>2929</p><p>3030</p><p>3131</p><p>3131</p><p>3131</p><p>3131</p><p>3232</p><p>3232</p><p>3333</p><p>3434</p><p>3535</p><p>3535</p><p>3636</p><p>3737</p><p>3737</p><p>3838</p><p>3939</p><p>3939</p><p>4040</p><p>4040</p><p>4040</p><p>4040</p><p>4141</p><p>4141</p><p>4141</p><p>4242</p><p>4242</p><p>4242</p><p>4343</p><p>4444</p><p>4545</p><p>4646</p><p>4747</p><p>4747</p><p>4848</p><p>4949</p><p>5050</p><p>5050</p><p>5151</p><p>5151</p><p>5252</p><p>5353</p><p>5353</p><p>5454</p><p>5555</p><p>5556</p><p>5556</p><p>5556</p><p>5657</p><p>5657</p><p>5657</p><p>5657</p><p>5758</p><p>5859</p><p>5960</p><p>6061</p><p>6162</p><p>6263</p><p>6364</p><p>6364</p><p>6364</p><p>6364</p><p>6364</p><p>6364</p><p>6364</p><p>6364</p><p>6465</p><p>6465</p><p>6465</p><p>6566</p><p>6566</p><p>6667</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6768</p><p>6869</p><p>6970</p><p>7071</p><p>7071</p><p>7071</p><p>7172</p><p>7273</p><p>7374</p><p>7475</p><p>7475</p><p>7475</p><p>7475</p><p>7576</p><p>7677</p><p>7778</p><p>7778</p><p>7879</p><p>7879</p><p>7879</p><p>7879</p><p>7879</p><p>7879</p><p>7879</p><p>7879</p><p>7980</p><p>8081</p><p>8182</p><p>8182</p><p>8283</p><p>8283</p><p>8283</p><p>8283</p><p>8384</p><p>8485</p><p>8485</p><p>8485</p><p>8485</p><p>8485</p><p>8586</p><p>8687</p><p>8788</p><p>8889</p><p>8889</p><p>8990</p><p>8990</p><p>8990</p><p>9091</p><p>9192</p><p>9192</p><p>9192</p><p>9192</p><p>9293</p><p>9293</p><p>9293</p><p>9294</p><p>9395</p><p>9395</p><p>9395</p><p>9496</p><p>9597</p><p>9597</p><p>9698</p><p>9698</p><p>9799</p><p>9799</p><p>9799</p><p>9799</p><p>98100</p><p>99101</p><p>100102</p><p>101103</p><p>102104</p><p>102104</p><p>102104</p><p>103105</p><p>104106</p><p>105107</p><p>106108</p><p>106108</p><p>107109</p><p>107109</p><p>108110</p><p>108110</p><p>108110</p><p>109111</p><p>110112</p><p>110112</p><p>110112</p><p>110113</p><p>110113</p><p>110113</p><p>110113</p><p>111114</p><p>111114</p><p>111114</p><p>112115</p><p>113116</p><p>113116</p><p>113116</p><p>114117</p><p>115118</p><p>115119</p><p>115119</p><p>116120</p><p>116120</p><p>116120</p><p>116120</p><p>116120</p><p>116120</p><p>117121</p><p>117121</p><p>117121</p><p>118122</p><p>118122</p><p>118122</p><p>119123</p><p>119123</p><p>119123</p><p>120124</p><p>120124</p><p>120124</p><p>120124</p><p>120124</p><p>121125</p><p>121125</p><p>121125</p><p>121125</p><p>121125</p><p>121125</p><p>121125</p><p>122126</p><p>123127</p><p>124128</p><p>125129</p><p>126130</p><p>126130</p><p>127131</p><p>127131</p><p>127131</p><p>128132</p><p>129133</p><p>129133</p><p>129133</p><p>129133</p><p>129133</p><p>129133</p><p>130134</p><p>130134</p><p>131135</p><p>132136</p><p>133137</p><p>133137</p><p>134138</p><p>135139</p><p>135139</p><p>135140</p><p>135141</p><p>136142</p><p>136142</p><p>136142</p><p>137143</p><p>137143</p><p>137143</p><p>137144</p><p>138145</p><p>138145</p><p>139146</p><p>139146</p><p>140147</p><p>141148</p><p>142149</p><p>142149</p><p>142149</p><p>142149</p><p>142149</p><p>142149</p><p>143150</p><p>144151</p><p>145152</p><p>145152</p><p>146153</p><p>147154</p><p>148155</p><p>148155</p><p>148155</p><p>148155</p><p>148155</p><p>148155</p><p>149156</p><p>149156</p><p>150157</p><p>150157</p><p>150157</p><p>150157</p><p>150157</p><p>151158</p><p>151159</p><p>152160</p><p>153161</p><p>153161</p><p>153162</p><p>153162</p><p>154163</p><p>155164</p><p>156165</p><p>156165</p><p>157166</p><p>157166</p><p>157166</p><p>157166</p><p>157167</p><p>157167</p><p>158168</p><p>158168</p><p>159169</p><p>160170</p><p>160170</p><p>161171</p><p>161171</p><p>161171</p><p>161171</p><p>161171</p><p>161171</p><p>161171</p><p>161172</p><p>162173</p><p>162173</p><p>162173</p><p>162173</p><p>162173</p><p>162174</p><p>163175</p><p>164176</p><p>164176</p><p>164176</p><p>165177</p><p>165177</p><p>166178</p><p>167179</p><p>168180</p><p>168180</p><p>168181</p><p>168181</p><p>168181</p><p>169181</p><p>169181</p><p>169182</p><p>169182</p><p>169182</p><p>169182</p><p>170183</p><p>170183</p><p>170183</p><p>170183</p><p>170183</p><p>171184</p><p>172185</p><p>172185</p><p>172185</p><p>173186</p><p>174187</p><p>174187</p><p>174187</p><p>175188</p><p>176189</p><p>177190</p><p>178191</p><p>178191</p><p>178191</p><p>178191</p><p>178191</p><p>178191</p><p>178191</p><p>178191</p><p>178191</p><p>179192</p><p>180193</p><p>181194</p><p>181195</p><p>181195</p><p>181195</p><p>181195</p><p>182196</p><p>183197</p><p>183197</p><p>183198</p><p>183198</p><p>183199</p><p>183199</p><p>183199</p><p>183199</p><p>183199</p><p>183199</p><p>183199</p><p>183199</p><p>184200</p><p>185201</p><p>185201</p><p>185201</p><p>185201</p><p>186202</p><p>186202</p><p>186202</p><p>186202</p><p>187203</p><p>188204</p><p>188204</p><p>188204</p><p>188204</p><p>188204</p><p>188204</p><p>189205</p><p>190206</p><p>191207</p><p>191207</p><p>191207</p><p>192208</p><p>192208</p><p>193209</p><p>194210</p><p>195211</p><p>195211</p><p>195211</p><p>196212</p><p>196212</p><p>197213</p><p>197213</p><p>198214</p><p>199215</p><p>199215</p><p>200216</p><p>200216</p><p>200216</p><p>200216</p><p>200216</p><p>200216</p><p>200216</p><p>201217</p><p>202218</p><p>203219</p><p>204220</p><p>205221</p><p>206222</p><p>207223</p><p>208224</p><p>209225</p><p>209225</p><p>209225</p><p>209225</p><p>209225</p><p>210226</p><p>211227</p><p>212228</p><p>212228</p><p>213229</p><p>214230</p><p>215231</p><p>215231</p><p>215231</p><p>215231</p><p>215231</p><p>215231</p><p>215231</p><p>216232</p><p>216232</p><p>216232</p><p>216232</p><p>216232</p><p>216232</p><p>216232</p><p>216232</p><p>217233</p><p>218234</p><p>219235</p><p>220236</p><p>220236</p><p>221237</p><p>221237</p><p>222238</p><p>222238</p><p>223239</p><p>224240</p><p>225241</p><p>226242</p><p>227243</p><p>228244</p><p>228244</p><p>228245</p><p>228245</p><p>228245</p><p>228245</p><p>228245</p><p>228245</p><p>228245</p><p>228245</p><p>229246</p><p>229246</p><p>230247</p><p>230247</p><p>230247</p><p>230247</p><p>230247</p><p>231248</p><p>231249</p><p>231249</p><p>231250</p><p>232251</p><p>232251</p><p>233252</p><p>234253</p><p>235254</p><p>236255</p><p>236255</p><p>237256</p><p>237256</p><p>237256</p><p>237256</p><p>238257</p><p>238257</p><p>239258</p><p>240259</p><p>240259</p><p>240260</p><p>241261</p><p>241261</p><p>241261</p><p>241261</p><p>241261</p><p>242262</p><p>243263</p><p>244264</p><p>245265</p><p>246266</p><p>246266</p><p>246266</p><p>247267</p><p>248268</p><p>249269</p><p>250270</p><p>251271</p><p>252272</p><p>252272</p><p>253273</p><p>254274</p><p>254274</p><p>254274</p><p>254274</p><p>254274</p><p>255275</p><p>255275</p><p>255275</p><p>255275</p><p>256276</p><p>256276</p><p>257277</p><p>257277</p><p>257277</p><p>258278</p><p>259279</p><p>260280</p><p>261281</p><p>262282</p><p>262282</p><p>263283</p><p>264284</p><p>264284</p><p>265285</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>266286</p><p>267287</p><p>267287</p><p>267287</p><p>268288</p><p>268288</p><p>268288</p><p>268288</p><p>268288</p><p>269289</p><p>269289</p><p>269289</p><p>270290</p><p>270290</p><p>270290</p><p>270290</p><p>270290</p><p>270290</p><p>271291</p><p>272292</p><p>273293</p><p>274294</p><p>274294</p><p>274294</p><p>274294</p><p>275295</p><p>275295</p><p>275295</p><p>276296</p><p>276296</p><p>277297</p><p>277297</p><p>278298</p><p>278298</p><p>278298</p><p>279299</p><p>279299</p><p>280300</p><p>280300</p><p>280300</p><p>280300</p><p>281301</p><p>282302</p><p>282302</p><p>283303</p><p>283303</p><p>283303</p><p>283303</p><p>284304</p><p>285305</p><p>286306</p><p>287307</p><p>288308</p><p>289309</p><p>289309</p><p>289309</p><p>289309</p><p>289309</p><p>290310</p><p>290310</p><p>290310</p><p>291311</p><p>292312</p><p>292312</p><p>292312</p><p>293313</p><p>294314</p><p>294314</p><p>294314</p><p>295315</p><p>295315</p><p>296316</p><p>296316</p><p>296316</p><p>296317</p><p>296317</p><p>296317</p><p>297318</p><p>297318</p><p>297318</p><p>297318</p><p>297318</p><p>297318</p><p>297318</p><p>298319</p><p>298319</p><p>298319</p><p>299320</p><p>299320</p><p>299320</p><p>299320</p><p>299320</p><p>299320</p><p>300321</p><p>300321</p><p>300321</p><p>300321</p><p>301322</p><p>302323</p><p>302323</p><p>303324</p><p>304325</p><p>305326</p><p>305327</p><p>306328</p><p>306328</p><p>306329</p><p>306329</p><p>306329</p><p>307330</p><p>307330</p><p>307330</p><p>307330</p><p>308331</p><p>308331</p><p>308331</p><p>308331</p><p>309332</p><p>310333</p><p>311334</p><p>311334</p><p>312335</p><p>313336</p><p>314337</p><p>315338</p><p>315338</p><p>315338</p><p>315339</p><p>315339</p><p>316340</p><p>316340</p><p>316340</p><p>316340</p><p>316341</p><p>317342</p><p>318343</p><p>318343</p><p>318343</p><p>318343</p><p>318343</p><p>318343</p><p>319344</p><p>319344</p><p>319344</p><p>319344</p><p>319344</p><p>319344</p><p>319344</p><p>320345</p><p>320345</p><p>320346</p><p>320347</p><p>321348</p><p>322349</p><p>322349</p><p>322349</p><p>322349</p><p>322349</p><p>322349</p><p>323350</p><p>323350</p><p>324351</p><p>324351</p><p>325352</p><p>325352</p><p>325352</p><p>325352</p><p>326353</p><p>326353</p><p>326353</p><p>327354</p><p>327354</p><p>327354</p><p>328355</p><p>329356</p><p>330357</p><p>330357</p><p>330357</p><p>330357</p><p>331358</p><p>331358</p><p>332359</p><p>333360</p><p>334361</p><p>335362</p><p>335362</p><p>335362</p><p>336363</p><p>336363</p><p>336363</p><p>336363</p><p>336363</p><p>336363</p><p>336363</p><p>337364</p><p>337364</p><p>338365</p><p>338365</p><p>339366</p><p>340367</p><p>341368</p><p>341368</p><p>341368</p><p>342369</p><p>343370</p><p>344371</p><p>344371</p><p>344371</p><p>344371</p><p>344371</p><p>344372</p><p>344372</p><p>345373</p><p>345373</p><p>345373</p><p>346374</p><p>346374</p><p>347375</p><p>348376</p><p>349377</p><p>350378</p><p>350378</p><p>350378</p><p>350378</p><p>351379</p><p>351379</p><p>351379</p><p>351379</p><p>351379</p><p>352380</p><p>352380</p><p>353381</p><p>353382</p><p>354383</p><p>354383</p><p>354383</p><p>355384</p><p>355384</p><p>355384</p><p>355384</p><p>356385</p><p>357386</p><p>357386</p><p>358387</p><p>359388</p><p>360389</p><p>361390</p><p>362391</p><p>363392</p><p>363392</p><p>363392</p><p>363392</p><p>363393</p><p>363393</p><p>364394</p><p>364394</p><p>365395</p><p>366396</p><p>366396</p><p>366396</p><p>367397</p><p>367397</p><p>367397</p><p>368398</p><p>368398</p><p>368398</p><p>369399</p><p>369399</p><p>369399</p><p>370400</p><p>371401</p><p>371401</p><p>371401</p><p>371401</p><p>372402</p><p>372402</p><p>373403</p><p>373403</p><p>373403</p><p>374404</p><p>374404</p><p>374404</p><p>374404</p><p>375405</p><p>375405</p><p>376406</p><p>377407</p><p>378408</p><p>378408</p><p>378408</p><p>378408</p><p>378408</p><p>379409</p><p>379409</p><p>379409</p><p>379409</p><p>379409</p><p>380410</p><p>380410</p><p>381411</p><p>381411</p><p>381411</p><p>381411</p><p>381411</p><p>382412</p><p>382412</p><p>382412</p><p>382412</p><p>382412</p><p>383413</p><p>383413</p><p>383413</p><p>383413</p><p>383413</p><p>383413</p><p>384414</p><p>384414</p><p>384414</p><p>385415</p><p>385415</p><p>385415</p><p>385415</p><p>385415</p><p>385415</p><p>385415</p><p>386416</p><p>387417</p><p>387417</p><p>387417</p><p>387417</p><p>387417</p><p>387418</p><p>387418</p><p>388419</p><p>388419</p><p>388419</p><p>388419</p><p>389420</p><p>389420</p><p>389420</p><p>389420</p><p>389420</p><p>389420</p><p>390421</p><p>390421</p><p>391422</p><p>391422</p><p>392423</p><p>392423</p><p>392423