lingvis: motivation and use cases · •linguists are making more and more use of newly available...
TRANSCRIPT
![Page 1: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/1.jpg)
LingVis: Motivation and Use Cases
Miriam Butt & Mennatallah El-Assady
LingVis: Visual Analytics for LinguisticsDGfS 2018 | 6.3.2018
![Page 2: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/2.jpg)
Tutorial schedule
• 10:00 am – 11:30 am: Motivation and Use Cases• Examples of existing visualizations relevant for LingVis.
• 11:45 am – 1:00 pm: Visualization Theory• Goals and principles of Visual Analytics
• Cognitive foundations
• 2:00 pm – 5:00 pm: Hands-on• Intro to existing visualization tools/frameworks
• Work with existing visualizations
2
![Page 3: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/3.jpg)
Before we start....
3
![Page 4: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/4.jpg)
Personal Questions
• Who are we?
• Who are you?• Programming Background
• What types of linguistic questions interest you?
• What is the interest in LingVis?
4
![Page 5: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/5.jpg)
Motivation and backgroundSlides based on the DGfS 2016 and Konvens 2016 Tutorials on LingVis and HistoBankVis 2017 talk.
Contributors: Miriam Butt, Oliver Deussen, Mennatallah El-Assady, Annette Hautli-Janisz, Wolfgang Jentner, Daniel Keim, Thomas Mayer, Frans Plank, Christian Rohrdantz, Dominik Sacha, Christin Schätzle, Rita Sevastjanova
5
![Page 6: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/6.jpg)
Overall Goals:
• Integrate methods from visual analytics into domains of linguistic inquiry.
• Explore challenges based on the needs of linguistic analysis for
visualization methods.
linguistic inquiry
linguistic analysis
visual analytics
visualization
Linguistics Computer Science
6
LingVis
![Page 7: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/7.jpg)
Visual Analytics
• Interactive, exploratory access to data
• Iterations of hypothesis formation and hypothesis testing
• Overview first – details on demand• Holistic picture
• But can drill down to individual data points
• Abstract representation of multifactorial, multidimensional data.
• Good for understanding complex interactions in the data.
7
![Page 8: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/8.jpg)
Existing Visualization: Syntax
• Syntactic Analysis with Lexical-Functional Grammar (LFG)
• http://clarino.uib.no/iness/xle-web (Web Interface for LFG Grammars)
• Grammar developed at PARC (Palo Alto Research Center)5
![Page 9: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/9.jpg)
Existing Visualization: Semantics
• Semantic Analysis with Discourse Representation Theory (Boxer)
• http://gmb.let.rug.nl/webdemo/demo.php (web interface for CCG/DRT)
• Grammar/Semantics developed by Johan Bos and colleagues (Groningen)
(Discourse Representation Structure)
![Page 10: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/10.jpg)
Existing Visualization: Phonetics
• aap=ne mara kis se thaa
![Page 11: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/11.jpg)
11
![Page 12: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/12.jpg)
Sample Visualizations12
![Page 13: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/13.jpg)
• Linguists are making more and more use of newly available technology to detect distributional patterns in language data.
• Ever increasing availability of digital corpora (synchronic and diachronic).
• Increasing interest in language output produced in social media.
• Ever better query and search tools (CQP, COSMAS, DWDS, ANNIS).
• Programming languages suitable for text processing, statistical analysis and visualization (e.g., Python, R).
13
LingVis – Motivation
![Page 14: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/14.jpg)
Making Sense of Numbers
• Current linguistics often includes corpus work.
• Linguists try to determine patterns, interactions and usage preferences within a language but also across different languages.
• This work generates a lot of numbers (statistics).
• Numbers are difficult for humans to process.
• Solution: translate numbers into visual properties.
Human visual apparatus can process this easily.
14
![Page 15: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/15.jpg)
Visualization in Linguistics
• Visualization is part and parcel of linguistics.
• However, very little advantage is taken of new visualization techniques.
• Newest ones borrowed from statistical analysis (bar charts, scatter plots, etc.), but often do not do justice to complexity of linguistic data.
• Multifactorial/Multidimensional (including temporal)
• Complex interactions
• Different data types
• Metadata
• Our Work: Identify and explore good use cases within linguistics
![Page 16: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/16.jpg)
Data / Language Resources Domain Expert
Research Question
16
Interdisciplinary Collaboration:LingVis
![Page 17: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/17.jpg)
Data / Language Resources Domain Expert
task modelling,
algorithmic processing,
statistical analyses
(Numerical) Features
Research Question
17
Interdisciplinary Collaboration:LingVis
![Page 18: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/18.jpg)
Data / Language Resources Domain Expert
(Numerical) Features
investigate
interactively
mapping to visual
variables, design,
layout algorithms
Visual Representation
Research Question
task modelling,
algorithmic processing,
statistical analyses
18
Interdisciplinary Collaboration:LingVis
![Page 19: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/19.jpg)
Example: Pixel-Based Visualizations
Two Use Cases• N-V Complex Predicates
• Vowel Harmony
19
![Page 20: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/20.jpg)
N-V Complex Predicates
• N-V complex predicates occur very frequently in Urdu.
• Examples: phone-do, memory-do, memory-become, resolution-do, resolution-be, ...
• Problem: would be nice if one knew which nouns were likely to co-occur with which verbs.
20
![Page 21: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/21.jpg)
21Butt, Miriam, Tina Bögel, Annette Hautli, Sebastian Sulger & Tafseer Ahmed. 2012. Identifying Urdu Complex Predication via Bigram Extraction. In Proceedings of the 24th International Conference on Computational Linguistics (COLING), 409–424. Mumbai, India.
• Goal: identify sequences of Noun+Verbfor understanding complex predicate patterns
• phone-do, use-do, memory-come, begin-do/come
• Data: 7.9 million word raw (unannotated) corpus of Urdu (BBC Urdu)
Example: N-V Complex Predicates in Urdu
![Page 22: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/22.jpg)
Example: Pixel Visualization Statistical Data:
Color Scale
kar, ho, hu, rakh
'achievement'
'announcement'
'talk'
'beginning'
do, be, become, put
![Page 23: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/23.jpg)
Pixel plus Cluster Visualization
• Performed k-means clustering combined with a pixel visualization.
• Advantages: – can inspect clusters visually and detect patterns
– Outliers spotted easily (mostly errors – “kyA” is not a noun, it is a wh-word and was included by mistake).
23
do be bec. put
![Page 24: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/24.jpg)
Example: Identifying N-V complex predicates in Hindi/IUrdu
Outliers/Errors are easily identified (Clustering Algorithm has applied)
Tool facilitates zooming and mousing over to see the underlying data set
![Page 25: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/25.jpg)
Vowel Harmony (VH)
• Phenomenon (simplified): Vowels in affixes change according to vowels found in stems.
• (Famous) Example: Turkish
27
![Page 26: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/26.jpg)
Goal: Try to determine automatically whether a given language
contains patterns indicative of vowel harmony.
Basic Computational Approach:
• Use written corpus (caveat: only approximates actual phonology).
• Count which vowels succeed which other vowels in VC+V sequences
(within words — again an approximation)
• Through statistical analysis find out the association strength between
vowels: normalized association strength value ϕ.
• Results show that Turkish and Hungarian, for example, pattern similarly.
Languages like Spanish or German pattern differently.
Vowel Harmony28
![Page 27: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/27.jpg)
Turkish
Spanish
German Hungarian
Results — Standard Methods: Can you detect a pattern?
29
![Page 28: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/28.jpg)
• Matrix visualization of association strengths between vowels (deviation from
statistical expectation).
• Vowels are sorted alphabetically.
• More saturated colors show greater association strength.
• Blue is for more frequently than expected, red for less.
• The +/– are redundant encodings.
First Simplistic Visualization: Can you detect a pattern?
Turkish Spanish GermanHungarian
30
![Page 29: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/29.jpg)
Turkish Spanish GermanHungarian
Vowels sorted according to similarity (note: not a trivial process)
Can even see the type of Vowel Harmony involved.
Sorted Visualization: Can you detect a pattern now?
31
T. Mayer, C. Rohrdantz, M. Butt, F. Plank and D. A. Keim. Visualizing Vowel Harmony. Linguistic Issues inLanguage Technology, 4(Issue 2):1-33, 2010.
![Page 30: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/30.jpg)
Visualizing Vowel Harmony
Counting Vowel Successions in all Bible Types
Example: Finnish
Statistics & Visualization
Sorting
[9]
Sorting done according to feature vectors of
each of the rows.
![Page 31: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/31.jpg)
33
Results –
Sorted Visualization:
• Automatic Visual Analysis of
vowel successions
for 42 languages – sorted for
effect strength.
33
![Page 32: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/32.jpg)
Hungarian
Breton Ukrainian
Tagalog Finnish
Indonesian
Turkish Maori Warlpiri
• In VH languages,
crucially there are some
vowels which never co-
occur.
• This can be seen via a
calculation of
succession
probabilities.
• Maori is not a VH
language.
Vowel Harmony vs. Reduplication34
![Page 33: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/33.jpg)
Even though Umlaut (raising of vowel in stem before high vowel in affix) is no longer a
productive process in German, the Umlaut harmony pattern is still visible in the matrices.
35
Historical Fingerprint: German Umlaut
![Page 34: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/34.jpg)
Example: Droplet Visualizations
• Different Types of Visualizations can be used to look at the same data.
• Example: Droplets for Vowel Harmony
• This droplet technique was originally used for rendering geospatial information (an item moving from one place to the next).
37
![Page 35: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/35.jpg)
kaşık-lar-ım-aspoon-Pl-1SgPoss-Dat‘my spoons’
kedi-ler-im-ecat-Pl-1SgPoss-Dat‘my cat’
aka
şık
lar
ım
kaşık-lar-ım-a
kedi-ler-im-e
kedi
ler
im
e
Vowel Harmony via Droplets38
![Page 36: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/36.jpg)
Language Comparison via Droplets
Norwegian shows language change a e in comparison to Swedish.
![Page 37: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/37.jpg)
• Another way to compare features across languages is via a sunburst visualization.
• The following visualization combines sunburst with a link to the geographical location of the language.
• The visual analysis is heavily interactive. • One can feed in one’s own data.
• One can also use the WALS (World Atlas of Language Structures; http://wals.info).
40
Christian Rohrdantz, Michael Hund, Thomas Mayer, Bernhard Wälchli and Daniel A. Keim. 2012. The World’s Languages Explorer: Visual Analysis of Language Features in Geneaologica and Areal contexts. Computer Graphics Forum 31(3), 935-944.
Example: Sunburst and Maps
![Page 38: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/38.jpg)
Sunburst and Maps for Language Families41
![Page 39: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/39.jpg)
Each circle segment represents one language, each ring the values of one feature across all languages.
Comparing 126 Languages of Papua New-Guinea based on the New Testament.
World's Language Explorer
![Page 40: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/40.jpg)
Bringing genealogy (left) and areal distributions (right) interactively into context: The values of a selected feature ring are color-coded on a map for exploration.
43
World's Language Explorer
![Page 41: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/41.jpg)
Sorting and Pattern Discovery44
![Page 42: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/42.jpg)
Sorting and Pattern Discovery45
![Page 43: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/43.jpg)
• A version that is tailored to interact with WALS is available on-line
• http://www.th-mayer.de/wals/
WALS Explorer46
Thomas Mayer, Bernhard Wälchli, Christian Rohrdantz and Michael Hund. 2014. From the extraction of continuous features in parallel texts to visual analytics of heterogeneous areal-typological datasets. In B. Nolan and C. Periñán-Pascual (eds.), Language Processing and Grammars: The role of functionally oriented computational models, 13–38. John Benjamins.
![Page 44: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/44.jpg)
Glyph Visualization for Diachronic Corpora
• Visualization of IcePaHC – Diachronic Corpus of Icelandic• Syntactically annotated in Penn Treebank style
• 60 texts
• 12th century CE to 21st century CE
• Two case studies so far (on-going)• V1 in Icelandic
• Dative Subjects in Icelandic
52
![Page 45: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/45.jpg)
V1 (Verb Initial or Verb First)
− Verb initial structures were common in matrix declaratives in Germanic.
− In German (and English) they mostly survive in narrative/joke contexts
Walked a man into a pub...
Questions
− What determines the appearance of V1?
− How did this change over the history of Germanic?
Glyph Visualization for Diachronic Corpora53
Butt, Miriam, Tina Bögel, Kristina Kotcheva, Christin Schätzle, Christian Rohrdantz, Dominik Sacha, Nicole Dehé & Daniel Keim. 2014. V1 in Icelandic: A MultifactoricalVisualization of Historical Data. Proceedings of the LREC 2014 Workshop on Visualization as added value in the development, use and evaluation of LRs (VisLR). Reykjavik, Iceland.
![Page 46: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/46.jpg)
Visual Analytic Access to Data
− Glyph Visualization of likely factors
− Overview of all 60 texts at once
− Can drill down to individual data points interactively
− Shneiderman’s Mantra: Overview First, Show the Important – Details
on Demand
Example: V1 in Icelandic
![Page 47: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/47.jpg)
HistoBankVis
• New Visualization for working with diachronic treebanks:• HistoBankVis
• Accessible On-line
• Interactive and Fast
subva.dbvis.de/histobankvis-v1.0
histobankvis.pdf
55
Schätzle, Christin, Michael Hund, Frederik Dennig, Miriam Butt, Daniel A. Keim. 2017. HistoBankVis: Detecting Language Change via Data Visualization. In G. Bouma and Y. Adesam (eds.): Proceedings of the NoDaLiDa 2017 Workshop on Processing Historical Language, Linköping: Linköping University Electronic Press, pp. 32-39.
![Page 48: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/48.jpg)
Example: Analyzing Political Argumentation (VisArgue)
• Public mediation on S21 (controversy around Stuttgart train station)
• Speakers are either Pro or Contra.
• Mediator is supposed to be neutral
• Data is annotated (rule based)
Gold, Valentin, Mennatallah El-Assady, Tina Bögel, Christian Rohrdantz, Miriam Butt, Katharina Holzinger & Daniel Keim. 2015. Visual Linguistic Analysis of Political Discussions: Measuring Deliberative Quality. Digital Scholarship in the Humanities,DOI: 10.1093/llc/fqv033
![Page 49: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/49.jpg)
VisArgue Framework (http://visargue.inf.uni-konstanz.de)
![Page 50: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/50.jpg)
Content Analysis
Discourse and Argumentation
Speaker Dynamics
L ex i c a l E p i s o d e P l o t s
N a m e d - E nt i t y R e l a t i o n s h i p E x p l o ra t i o n
U s e r - S te e ra b l e To p i c M o d e l i n g
To p i c S p a c e V i e w s
A r g u m e n ta t i o n G l y p h s
F e a t u r e A l i g n m e nt
D i s c o u r s e M a p s
M. El-Assady, R. Sevastjanova, B. Gipp, D. Keim, and C. Collins.
NEREx: Named-Entity Relationship Exploration in Conversational Text Corpora.
Computer Graphics Forum – EuroVis, 2017.
![Page 51: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/51.jpg)
![Page 52: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/52.jpg)
![Page 53: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/53.jpg)
• Person
• Geo-Location
• Organization
• Date-Time
• Measuring Unit
• Measure
• Context-Keyword
• Politeness-Indicator
• Positive-Emotion Indicator
• Negative-Emotion Indicator
Extraction and Categorization
Supervised LearningUse Stanford Named Entity Recognition Library to train a 7-class model to extract: Location, Person, Organization, Money, Percent, Date, and Time.
Unsupervised LearningUse topic modeling and lexical-chaining algos.
Heuristics Use rule-based classifications using word-lists, lemmas, POS-tags, and regular expressions.
![Page 54: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/54.jpg)
Close-Reading Distant-Reading
![Page 55: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/55.jpg)
Named-Entity Relationship Named-Entity Relationship
Grammar Document Structure
Works well for highly-edited texts but not for verbatim conversation transcripts
due to ungrammatical sentences.
![Page 56: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/56.jpg)
Named-Entity Relationship
Grammar
Works well for highly-edited texts but not for verbatim conversation transcripts
due to ungrammatical sentences.
![Page 57: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/57.jpg)
Named-Entity Relationship
Grammar Document Structure
Works well for highly-edited texts but not for verbatim conversation transcripts
due to ungrammatical sentences.
Works well for text summarization but constructs too broad relations.
![Page 58: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/58.jpg)
Distance-Restricted Entity-Relationship Model
“I believe that we can cut taxes by an additional $5 trillion.”
maxDist = 5
![Page 59: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/59.jpg)
Distance-Restricted Entity-Relationship Model
maxDist = 5
![Page 60: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/60.jpg)
Entity Graph
![Page 61: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/61.jpg)
All 2016 US Presidential Debates
minimum entity-pair frequency = 3
![Page 62: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/62.jpg)
Views
[SG] Speaker Graph
[TLV] Text-Level View
[ELV] Entity-Level View
[EG] Entity Graph
[CG] Concept Graph[TG] Temporal Graph
![Page 63: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/63.jpg)
Use Case2016 US Presidential Debates
![Page 64: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/64.jpg)
Overall Exploration of the Dataset
![Page 65: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/65.jpg)
![Page 66: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/66.jpg)
![Page 67: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/67.jpg)
Topic: Taxes
![Page 68: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/68.jpg)
![Page 69: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/69.jpg)
Content Analysis
Discourse and Argumentation
Speaker Dynamics
L ex i c a l E p i s o d e P l o t s
N a m e d - E nt i t y R e l a t i o n s h i p E x p l o ra t i o n
U s e r - S te e ra b l e To p i c M o d e l i n g
To p i c S p a c e V i e w s
A r g u m e n ta t i o n G l y p h s
F e a t u r e A l i g n m e nt
D i s c o u r s e M a p s
M. El-Assady, V. Gold, C. Acevedo, C. Collins and D. A. Keim.
ConToVi: Multi-Party Conversation Exploration using Topic-Space Views.
Computer Graphics Forum – EuroVis, 2016 .
![Page 70: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/70.jpg)
Reason
Conclusion
Topic
Speaker
![Page 71: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/71.jpg)
Reason
Conclusion
Argumentation Glyphs
Topic
Speaker
![Page 72: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/72.jpg)
Argumentation Glyphs
![Page 73: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/73.jpg)
Argumentation Glyphs – Clustered View
Heiner Geißler
Volker Kefer
![Page 74: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/74.jpg)
Micro-Linguistic Discourse Features
Dimensions of Deliberative Communication
1. Participation (equal and inclusive participation)
2. Atmosphere & Respect (equal respect)
3. Argumentation & Justification (reason-giving)
4. Accommodation (consensus-seeking)
![Page 75: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/75.jpg)
>
Discourse Maps
![Page 76: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/76.jpg)
4 Dimensions Quadrants
19 Sub-Dimensions Rows
53 Measures / Features Rectangles
Discourse Maps
![Page 77: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/77.jpg)
• Binary (0/1, e.g. “reason” or not)
• Numerical, continuous (01, e.g. epistemic value)
• Numerical, bipolar (-11, e.g. emotion=“negative/positive”)
Types of measures
0 1
0 1
-1 10
![Page 78: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/78.jpg)
lessfrequent
more frequent
Frequency of the measure
![Page 79: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/79.jpg)
shorter longer
Average utterance length
Speakers and speaker parties
PRO 2 CONTRA 2 NEUTRAL 2 EXPERT 2
PRO 1 CONTRA 1 NEUTRAL 1 EXPERT 1
![Page 80: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/80.jpg)
Participation Argumentation &Justification
Accommodation Atmosphere & Respect
Discourse Maps
![Page 81: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/81.jpg)
Average sentencecomplexity
Emotion
Discourse Maps
Participation Argumentation &Justification
Accommodation Atmosphere & Respect
![Page 82: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/82.jpg)
Argumentation
Neutral: Comparatively little Argumentation
Experts: Justifications/Reasons
Pro & Contra: Conclusions and Justifications
Deliberative Argumentation
Measures shown:
1. Conclusion
2. Justification
1 2
Participation Argumentation
Discussion Respect
![Page 83: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/83.jpg)
Rejection of "Common" Ground
Refusal to provide information
Pro, Contra & Neutral:
Establishment of Common Ground
Pro: Rejection of "Common" Ground,
Refusal to provide information
controversial discussions
Information mainly from discourse particles and speech act verbs
Establishment of Common Ground
![Page 84: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/84.jpg)
Glyphs per Group Type and Topics (Topic-Modelling)
![Page 85: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/85.jpg)
Ca
Can aggregate or look at individually
![Page 86: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/86.jpg)
Who was interrupting the debates?
Firs
t D
eb
ate
Seco
nd
De
bat
e
![Page 87: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/87.jpg)
Summary of all three debates
• High amount of interruptions• Negative sentiment
• Multiple topic shifts towards “previous” topics
• High amount of filler words• Low lexical diversity
![Page 88: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/88.jpg)
Visualization of spoken language
D. Sacha, Y. Asano, C. Rohrdantz, F. Hamborg, D. A. Keim, B. Braun and M. Butt. 2015. Self Organizing Maps for the Visual Analysis of Pitch Contours. Proceedings ofthe 20th Nordic Conference of Computational Linguistics (NoDaLiDa-2015), Vilnius, Lithuana, 2015.
• So far we have been working with textual data. However, one can also work with spoken data.
• For Visual Analytics, all one needs is to have features (or vectors) that can be computed with
![Page 89: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/89.jpg)
Motivation138
“Sumimasen”
“Entschuldigung”
Experiments Recording
Analysis
- Manual Inspection- Statistics- Functional Data Analysis- EXPLORATION
Recording,Landmarks,Utterance,Pitch-Vectors
Speaker Information
![Page 90: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/90.jpg)
Our Approach - Workflow139
A B
Data Input Configuration
C
Training
D
Interactive Visualization
![Page 91: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/91.jpg)
Configuration and Preprocessing140
RAW-Pitches Processed Pitches
Essential: Comparable Vectors & Distance Function
![Page 92: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/92.jpg)
Iterative SOM Training and Interaction141
Interactions
Data Exploration• SOM learning is fast• user can switch among different
perspectives on the data• user can interactively delete or
pin cells• and retrain and re-explore
![Page 93: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/93.jpg)
Interactive Visualizations142
Grid – Cells/CentroidsHistory/Distances
Word Clouds Bar Charts Colored Cells Cell Layout
Centroid& Vectors
Attribute (Value) - Heatmaps
![Page 94: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/94.jpg)
Use Case143
“Sumimasen”
“Entschuldigung”
Experiment - Data Pre-Processing
Native Speakers vs. Learners
Tasks
Are there Differences?
FDA
![Page 95: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/95.jpg)
Use Case – Two Separate SOMs are Trained for Japanese and German Speakers
144
Speakers pronounced "sorry/excuse me" in ever more exasperating circumstances• Japanese natives do not vary the pitch contour (red SOM - B)• German learners do vary the pitch (blue SOM - C)
![Page 96: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/96.jpg)
Identifying Optimal Visualizations
• Understanding which visualizations are optimal is not trivial.
• Are the individual dimensions (color, shape, direction, size, etc.) usefully meaningful?
• Does the visualization allow for at-a-glance understanding, or does it confuse the user?
• This also depends on the user's background• What is the user used to looking at?
• How is the user used to understanding the data?
• How is the user used to interacting with systems?
• Currently evaluations are mainly performed via user studies in Visual Analytics.
• Project (SFB/TRR 161): Establish evaluation metrics.
![Page 97: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/97.jpg)
Distorted Map according to number of languages spoken in area.
Visualization only as good as your data – India massively underrepresented
![Page 98: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/98.jpg)
Outlook
• Further exploration of possibilities offered by Visual Analytics• The systems illustrated here are recent
• Interactive exploratory linguistic analysis is on-going
• Systems are being fine-tuned
• Workflow • Use cases for Digital Humanities /eHumanities are being developed
• Infrastructure Platforms (mix and match the available tools)
• Measuring Success• Development of Evaluation Metrics for LingVis.
• Use cases, work flow and result comparison.
![Page 99: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/99.jpg)
What interests Visualizers?
• Need interesting interactions
• Multiple dimensions
• Time depth
• Cross-modular interactions.
• Not just coloring in bits of text that are of interest for linguists.
155
![Page 100: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/100.jpg)
Summary
• Example of different kinds of visualizations
• These visualizations allow a new approach to linguistic data
• Flexible, interactive, make use of the highly skilled human perceptual system
• More examples to follow.
Now first some design basics!
156
![Page 101: LingVis: Motivation and Use Cases · •Linguists are making more and more use of newly available technology to detect distributional patterns in language data. •Ever increasing](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6045719d7c1b2b099e699452/html5/thumbnails/101.jpg)
THANK YOU!Questions?
157
Action
Finding
Hypothesis
Insight
Knowledge
Human
Exploration
Loop
Verification Loop
Knowledge
Generation Loop
Visualization
Model
Data
Computer