computational linguistics for low-resource languages · palmer, icl, u heidelberg 8 cl4lrl, 4 may...
TRANSCRIPT
![Page 1: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/1.jpg)
Computational Linguistics for
Low-Resource Languages
4 May 2016 Alexis Palmer
![Page 2: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/2.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 20162
• topics and schedule posted on Thursday or Friday
• May 11: LRAs from Mayumi Ohta, Juri Opitz
• May 11: active learning for morpheme glossing
• don’t forget to send questions
Organizational
![Page 3: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/3.jpg)
The Human Language Project: Building a Universal Corpus of the
World’s Languages
Steven Abney and Steven BirdACL 2010, position/challenge paper
![Page 4: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/4.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 20164
Aim • corpus of world’s languages: cf. HGP
• consistent, machine-readable format
• fixed set of data types
• enable massive cross-linguistic processing
Human Language Project - the vision
Ainu Nahuatl German …LexiconNaBitextSwadesh…
![Page 5: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/5.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 20165
Aim • corpus of world’s languages: cf. HGP
• consistent, machine-readable format
• fixed set of data types
• enable massive cross-linguistic processing
Human Language Project - the vision
Ainu Nahuatl German …LexiconNaBitextSwadesh…
![Page 6: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/6.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 20166
Urgency
Motivations
![Page 7: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/7.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 20167
Urgency • critical period is NOW
• expertise in computational linguistics needed
“The next generation will forgive us for the most egregious shortcomings in theory construction and technology development, but they will not forgive us if we fail to preserve vanishing primary language data in a form that enables future research.”
Motivations
![Page 8: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/8.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 20168
In the name of linguistics (and CL)
Urgency • critical period is NOW
• expertise in computational linguistics needed
“The next generation will forgive us for the most egregious shortcomings in theory construction and technology development, but they will not forgive us if we fail to preserve vanishing primary language data in a form that enables future research.”
Motivations
![Page 9: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/9.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 20169
In the name of linguistics (and CL) • “complete digitization of every human language”
• universal theory/understanding of human language
• enable data-lean approaches via cross-linguistic modeling
• support automatic processing for all (?) languages
Urgency • critical period is NOW
• expertise in computational linguistics needed
“The next generation will forgive us for the most egregious shortcomings in theory construction and technology development, but they will not forgive us if we fail to preserve vanishing primary language data in a form that enables future research.”
Motivations
![Page 10: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/10.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201610
What counts as sufficient digitization of a language? • one view: enough to enable language learning in the
absence of native speakers
• another view: enough to enable linguistic analysis in the absence of native speakers
• yet another view: enough to provide an appropriate cultural record (moving target...)
Digitizing a language
![Page 11: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/11.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201611
Abney and Bird’s view: • use case: machine translation, data-lean methods
• MT as proxy for language learning, semantic interpretation
• for each language: primary texts w/ translation to reference language
Digitizing a language
![Page 12: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/12.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201612
Anticipated “pyramidal” structure of resources
unannotated text translation from target lg. to reference lg.
sentence and word alignments morphological analysis interlinear glossed text
Digitizing a language
![Page 13: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/13.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 2016
Producing interlinear glossed text (IGT) starts from:
Transcription of recorded speech
Translation of transcribed text (may be literal or free)
13
(a) xelch li. (b) Salio entonces. (b’) Then he left.
Interlinear glossed text (IGT)
![Page 14: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/14.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 2016
Producing interlinear glossed text (IGT) involves:
morphological segmentation
stem translation
morpheme glossing
14
(a) xelch li. (b) x- el -ch li (c) COM- salir -DIR DEM (d) Salio entonces.
Interlinear glossed text (IGT)
![Page 15: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/15.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 2016
Producing interlinear glossed text (IGT) involves:
morphological segmentation
stem translation
morpheme glossing
POS-tagging of stems (often derived automatically from lexicon)
15
(a) xelch li. (b) x- el -ch li (c) COM- salir -DIR DEM (d) TAM- VI -DIR PART (e) Salio entonces.
Interlinear glossed text (IGT)
![Page 16: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/16.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 2016
Producing interlinear glossed text (IGT) involves:
morphological segmentation
stem translation
morpheme glossing
POS-tagging of stems (often derived automatically from lexicon)
16
(a) xelch li. (b) x- el -ch li (c) COM- salir -DIR DEM (d) TAM- VI -DIR PART (e) Salio entonces.
Interlinear glossed text (IGT)
![Page 17: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/17.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201617
• Metadata - Dublin Core, TEI
• Provenance
• Source citation (beyond bib. entry)
• Spontaneous vs. elicited speech
• Slow speech “audio transcriptions”
• Orthographic vs. phonetic transcription
• Attested forms
• ISO 639 language codes
Terminology and background
![Page 18: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/18.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201618
1. Title 2. Creator 3. Subject 4. Description 5. Publisher 6. Contributor 7. Date 8. Type
Dublin Core Metadata Elements
9. Format 10. Identifier 11. Source 12. Language 13. Relation 14. Coverage 15. Rights
![Page 19: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/19.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201619
Essential components of the Universal Corpus • Metadata
• Written text:
• primary data
• transcriptions
• Spoken text:
• audio recordings
• audio transcriptions
• written transcriptions
What to include - 1 of 2
![Page 20: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/20.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201620
Essential components of the Universal Corpus • Both written and spoken text:
• translations into reference language
• sentence-level segmentation & translation
• word-level segmentation & glossing
• morpheme-level segmentation & glossing
• Secondary resources
• lexicon with glosses in reference language
• paradigms and phonological information sufficient to build a morphological analyzer
What to include - 2 of 2
![Page 21: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/21.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201621
ID: europarl/swedish/ep-00-01-17/18LANGS: swd engSENT: det gäller en ordningsfråga TRANS: this is a point of order ALIGN: 1-1 2-2 3-3 4-4 4-5 4-6 PROVENANCE: pharaoh-v1.2, ... REV: 8947 2010-05-02 10:35:06 leobfld12RIGHTS: Copyright (C) 2010 Uni...; CC-BY
Example: sentence alignment
![Page 22: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/22.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201622
ID: example/001LANGS: engSENT: the dogs are barkingLEX: the dog be barkAFF: - PL PL ING
ID: swedishlex/v3.2/0419 LANGS: swd eng LEX: ordningsfr ̊aga TRANS: point of order
Examples
ID: swadesh/47LANGS: fraLEX: chien
ID: swadesh/47LANGS: engLEX: dog
![Page 23: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/23.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201623
Important foundational principles/goals • Universality
• Machine readability and consistency
• Community effort
• Availability
• Utility - lightweight vs. rich annotation
• Centrality of primary data - analyses ideally reference primary data
Guiding principles
![Page 24: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/24.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201624
Problems with some extant text collections • Traditional language archives
• potentially broad coverage *but*
• restricted access
• not amenable to machine processing
• Large-scale data collection efforts
• e.g. LDC (Linguistic Data Consortium), ELRA (Europear Language Resources Association)
• minimal coverage
• General problems
• discoverability
• standardization
What’s wrong with current resources?
![Page 25: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/25.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201625
Roles • Editors
• CL research
• Tool builders
• Volunteer annotators
• Documentary linguists
• Data agencies
• Language archives
• Funding agencies
The Universal Corpus ecosystem
![Page 26: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/26.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201626
Early tasks • Seed corpus
• Resource discovery
• Resource classification
• Acquisition
• Text collection
• Audio protocol
• Corpus readers
Where do we begin?
![Page 27: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/27.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201627
• Active learning to improve quality of alignments and bilingual lexicons
• Automatic language identification for low-resource languages
• Automatic morphology learning for low-resource languages
• Leaner methods for building MT systems
• New algorithms for cross-linguistic bootstrapping via multiple paths
• More effective techniques for leveraging human effort
• …
Potential research directions
![Page 28: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/28.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201628
• Tools for various stages in the process
• Pilot studies to determine documentation cost for different language categories
• Development of corpus readers, format converters, etc.
Potential development directions
![Page 29: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/29.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 2016
Already in progress
✦ BOLD:PNG and larger (100-lg) project
✦ All Languages Wiki (alpha version)
✦ Language Commons portal
✦ OLAC
✦ Alliance for Linguistic Diversity: endangeredlanguages.com
✦ Relevant open-source code: https://github.com/RichardLitt/endangered-languages
29
Outgrowths
![Page 30: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/30.jpg)
Seven Dimensions of Portability for Language Documentation and
Description
Steven Bird and Gary SimonsLanguage 2003
![Page 31: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/31.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201631
Foundational paper for digital documentation/description • Language documentation v. language description
• Dated in some ways, utterly relevant in others
• Sidetrip: Summer Institute for Linguistics (SIL)
The context
![Page 32: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/32.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201632
• General purpose tools
• office software - WYSIWYG
• hypertext
• database packages
• presentation vs. underlying data structure
• Specialized tools
• Shoebox/Toolbox, Fieldworks Language Explorer
• Praat, ELAN, etc.
• Digital technologies
• Digital archives
Tools and technologies
![Page 33: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/33.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201633
• General purpose tools
• office software - WYSIWYG
• hypertext
• database packages
• presentation vs. underlying data structure
• Specialized tools
• Shoebox/Toolbox, Fieldworks Language Explorer
• Praat, ELAN, etc.
• Digital technologies
• Digital archives
Tools and technologies
“digital detritus”
![Page 34: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/34.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201634
Content • Coverage - weaknesses in coverage limit interpretability
of data
• Accountability - allow reference to original linguistic event
• Terminology - GOLD
General Ontology for Linguistic Description (Farrar & Lewis 2007)
The seven dimensions
![Page 35: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/35.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201635
GOLD - Linguistic examples
linguistics-ontology.org
![Page 36: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/36.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201636
Format • Openness - the problem of proprietary formats
• Encoding - despite significant work in this area, a perpetual problem when working with language data
• Markup - need for machine-readable markup
• Rendering - need for human-accessible display (1) a. chien n dog. b. chien: [n] dog. (2) \ent chien \pos n
\def dog
The seven dimensions
![Page 37: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/37.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201637
The seven dimensions
a. <p><font size=+1> <i>chien</i></font> <b>n</b> <font color=blue>dog.</font></p>
b. <entry> <headword>chien</headword> <pos>n</pos> <definition>dog</definition>
</entry>
![Page 38: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/38.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201638
Discovery • Existence
• Relevance - role of controlled vocabulary
Access • Scope of access
• Process for access
• Ease of access - importance of printed versions, access via DVD/etc. (i.e. without internet connection), importance of both online and offline tool versions
The seven dimensions
![Page 39: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/39.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201639
Citation • Bibliography
• Persistence
• Immutability/versioning
• Granularity - locating the precise reference/instance/lexical item
Preservation • Longevity
• Safety
• Media
The seven dimensions
![Page 40: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/40.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201640
Rights • Terms of use - complex ethical issues
• Benefit
• Sensitivity - different perspectives on data
• Balance
The seven dimensions
![Page 41: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/41.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201641
Many of the recommendations have become standards • XML
• Unicode
• Archiving and metadata standards (OLAC)
• Citation persistence
• Open source
• Movement within ACL/LSA/etc. to encourage (sometimes even require) submission of data with publications
Impact
![Page 42: Computational Linguistics for Low-Resource Languages · Palmer, ICL, U Heidelberg 8 CL4LRL, 4 May 2016 In the name of linguistics (and CL) Urgency • critical period is NOW • expertise](https://reader033.vdocuments.mx/reader033/viewer/2022042104/5e817dd1a2cff24b8b55a274/html5/thumbnails/42.jpg)
Palmer, ICL, U Heidelberg CL4LRL, 4 May 201642
• rights re: crawled data?
• HLP: list of numbers given as one way of representing data - what do these mean? [9347, 3053, 0038, …]
• what about all of the pragmatic and world knowledge that gets lost if we consider translation-capability a measure of an adequate digitization?
• is primary text always written in Latin letters? what about other/no writing systems?
Questions from you