mathematical linguistics in the 21st centuryjeffreyheinz.net/talks/heinz-2020-ml21c-slides.pdf ·...
TRANSCRIPT
![Page 1: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/1.jpg)
Mathematical Linguisticsin the 21st Century
Jeffrey Heinz
New Orleans, LAWorkshop on Formal Language TheorySociety for Computation in Language
January 5, 2020FLT · SCiL | 2020/01/05 J. Heinz | 1
![Page 2: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/2.jpg)
Thesis
Far from being a fossil from a former era, mathematicalthinking about language
1 continues to play an essential role in understanding naturallanguages and
2 continues to make critical contributions to ourunderstanding of how things which compute—both humansand machines—can learn.
FLT · SCiL | 2020/01/05 J. Heinz | 2
![Page 3: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/3.jpg)
Part I
What is mathematics?
FLT · SCiL | 2020/01/05 J. Heinz | 3
![Page 4: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/4.jpg)
What is mathematics?
Marcus Kracht (Los Angeles circa 2005)
“It is a way of thinking.”
Eugenia Cheng How to Bake π (2015) : 8
“Math, like recipes, has both ingredients and method. . . . Inmath, the method is probably even more important than theingredients.”
FLT · SCiL | 2020/01/05 J. Heinz | 4
![Page 5: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/5.jpg)
Abstraction
Eugenia Cheng How to Bake π (2015) : 16/22
• “Math is there to make things simpler, by finding thingsthat look the same if you ignore some small details.”
• “Abstraction can appear to take you further and furtheraway from reality, but really you’re getting closer andcloser to the heart of the matter.”
Noam Chomsky The Minimalist Program (1995) : 6
“Idealization, it should be noted, is a misleading term for theonly reasonable way to approach a grasp of reality.”
I disagree with the word ‘only,’ but I do think abstraction isunderappreciated.
FLT · SCiL | 2020/01/05 J. Heinz | 5
![Page 6: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/6.jpg)
Abstraction
Eugenia Cheng How to Bake π (2015) : 16/22
• “Math is there to make things simpler, by finding thingsthat look the same if you ignore some small details.”
• “Abstraction can appear to take you further and furtheraway from reality, but really you’re getting closer andcloser to the heart of the matter.”
Noam Chomsky The Minimalist Program (1995) : 6
“Idealization, it should be noted, is a misleading term for theonly reasonable way to approach a grasp of reality.”
I disagree with the word ‘only,’ but I do think abstraction isunderappreciated.
FLT · SCiL | 2020/01/05 J. Heinz | 5
![Page 7: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/7.jpg)
Abstraction
image credit: https://computersciencewiki.org/index.php/Abstraction
FLT · SCiL | 2020/01/05 J. Heinz | 6
![Page 8: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/8.jpg)
Abstraction
Many things were at one time considered to be “too abstract”:0, real numbers,
√−1, uncountable infinity, number theory, . . .
image credit: https://computersciencewiki.org/index.php/Abstraction
FLT · SCiL | 2020/01/05 J. Heinz | 6
![Page 9: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/9.jpg)
Goals
Deducing consequences from premises.
Advantages:
1 Can provide complete, verifiable, interpretable &understandable solutions to problems.
2 Can provide fresh insight into reality.
3 Truth is timeless.
realizationabstraction
Problem Solution
Complicated messy system
FLT · SCiL | 2020/01/05 J. Heinz | 7
![Page 10: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/10.jpg)
Goals
Deducing consequences from premises.
Disadvantages:
1 The ‘abstraction’ and ‘realization’ steps take additionalwork and time.
realizationabstraction
Problem Solution
Complicated messy system
FLT · SCiL | 2020/01/05 J. Heinz | 7
![Page 11: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/11.jpg)
Part II
Overview of Rest of Talk
Mathematical Linguistics in the 21st Century
FLT · SCiL | 2020/01/05 J. Heinz | 8
![Page 12: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/12.jpg)
Timeframe
• Of course much is owed to Boole and Frege of the 19thcentury
• Of course much is owed to Russell, Church, Turing, Post,the Polish school of Logic, and others
• Of course there are many others I omit (Montague, Bach,Monnich, Savitch, Johnson, . . . )
• I am going to focus on specific contributions in the latterhalf of the 20th century which directly contributed to myown interests.
FLT · SCiL | 2020/01/05 J. Heinz | 9
![Page 13: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/13.jpg)
Linguistic Questions as ComputationalProblems
What is it that we know when we know a language?
How do we come by this knowledge?
1 A Membership problem.
2 A Learning problem.
3 Variations thereof.
FLT · SCiL | 2020/01/05 J. Heinz | 10
![Page 14: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/14.jpg)
Linguistic Questions as ComputationalProblems
What is it that we know when we know a language?
How do we come by this knowledge?
1 A Membership problem.
2 A Learning problem.
3 Variations thereof.
FLT · SCiL | 2020/01/05 J. Heinz | 10
![Page 15: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/15.jpg)
Part III
Characterizing knowledge of language
FLT · SCiL | 2020/01/05 J. Heinz | 11
![Page 16: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/16.jpg)
Conservativity (Keenan and Stavi 1986)
The theory of Generalized Quantifiers addresses determinerexpressions like
• every, all, some, not one, more than three, fewer thantwenty, most, how many, which, more male than female,less than half, . . .
in utterances like
birds fly south for the winter.
1 What are the (possible) denotations of these expressions?
2 How arbitrary can they be?
FLT · SCiL | 2020/01/05 J. Heinz | 12
![Page 17: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/17.jpg)
Conservativity (Keenan and Stavi 1986)
All birds fly south for the winter.
Denotation of all
• all P Q is true iff P⊆ Q
Conservativity
D is conservative iff D P Q = D P R whenever P∩Q = P∩R.
Informally, this means that in evaluating D P Q we ignore theelements of Q which do not lie in P.
An example of non-conservative D:
• Hartig P Q is true iff |P| = |Q|
FLT · SCiL | 2020/01/05 J. Heinz | 13
![Page 18: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/18.jpg)
Conservativity (Keenan and Stavi 1986)
All birds fly south for the winter.
Denotation of all
• all P Q is true iff P⊆ Q
Conservativity
D is conservative iff D P Q = D P R whenever P∩Q = P∩R.
Informally, this means that in evaluating D P Q we ignore theelements of Q which do not lie in P.
An example of non-conservative D:
• Hartig P Q is true iff |P| = |Q|
FLT · SCiL | 2020/01/05 J. Heinz | 13
![Page 19: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/19.jpg)
Conservativity (Keenan and Stavi 1986)
All birds fly south for the winter.
Denotation of all
• all P Q is true iff P⊆ Q
Conservativity
D is conservative iff D P Q = D P R whenever P∩Q = P∩R.
Informally, this means that in evaluating D P Q we ignore theelements of Q which do not lie in P.
An example of non-conservative D:
• Hartig P Q is true iff |P| = |Q|
FLT · SCiL | 2020/01/05 J. Heinz | 13
![Page 20: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/20.jpg)
Conservativity (Keenan and Stavi 1986)
Logically Possible Generalized Quantifiers
GQs satisfying
conservativity
FLT · SCiL | 2020/01/05 J. Heinz | 14
![Page 21: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/21.jpg)
Conservativity (Keenan and Stavi 1986)
Logically Possible Generalized Quantifiers
GQs satisfying
conservativity
FLT · SCiL | 2020/01/05 J. Heinz | 14
![Page 22: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/22.jpg)
Mathematics of Sequences
• Language unfolds over time.
• We observe sequences of linguistic events.
• What is the mathematics of sequences?
• What is the mathematics of other relational structures liketrees and graphs?
Knowledge of language includes knowledge of which sequencesare licit and which are not.
• John laughed and laughed. X
• John and laughed. 7
FLT · SCiL | 2020/01/05 J. Heinz | 15
![Page 23: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/23.jpg)
Mathematics of Sequences
• Language unfolds over time.
• We observe sequences of linguistic events.
• What is the mathematics of sequences?
• What is the mathematics of other relational structures liketrees and graphs?
Knowledge of language includes knowledge of which sequencesare licit and which are not.
• John laughed and laughed. X
• John and laughed. 7
FLT · SCiL | 2020/01/05 J. Heinz | 15
![Page 24: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/24.jpg)
A Membership problem
M
yes no
s
Logically Possible Strings
Ss ∈ S s 6∈ S
FLT · SCiL | 2020/01/05 J. Heinz | 16
![Page 25: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/25.jpg)
Variations thereof
Functions on the string domain . . .
Function Type Output Type
Σ∗ → {T, F} BooleansΣ∗ → Σ∗ StringsΣ∗ → N Natural NumbersΣ∗ → [0, 1] Reals in the Unit IntervalΣ∗ → P (Σ∗) Stringsets. . .
Mathematics classifies numerical functions according togeneral properties: linear, polynomial, trigonometric,logarithmic, . . .
How can we classify functions like those above?
FLT · SCiL | 2020/01/05 J. Heinz | 17
![Page 26: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/26.jpg)
Variations thereof
Functions on the string domain . . .
Function Type Output Type
Σ∗ → {T, F} BooleansΣ∗ → Σ∗ StringsΣ∗ → N Natural NumbersΣ∗ → [0, 1] Reals in the Unit IntervalΣ∗ → P (Σ∗) Stringsets. . .
Mathematics classifies numerical functions according togeneral properties: linear, polynomial, trigonometric,logarithmic, . . .
How can we classify functions like those above?
FLT · SCiL | 2020/01/05 J. Heinz | 17
![Page 27: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/27.jpg)
Variations thereof
Functions on the string domain . . .
Function Type Output Type
Σ∗ → {T, F} BooleansΣ∗ → Σ∗ StringsΣ∗ → N Natural NumbersΣ∗ → [0, 1] Reals in the Unit IntervalΣ∗ → P (Σ∗) Stringsets. . .
Mathematics classifies numerical functions according togeneral properties: linear, polynomial, trigonometric,logarithmic, . . .
How can we classify functions like those above?
FLT · SCiL | 2020/01/05 J. Heinz | 17
![Page 28: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/28.jpg)
Classifying Membership Problems
Nowak et al. 2002, NatureFLT · SCiL | 2020/01/05 J. Heinz | 18
![Page 29: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/29.jpg)
Classifying Membership Problems
Nowak et al. 2002, NatureFLT · SCiL | 2020/01/05 J. Heinz | 18
![Page 30: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/30.jpg)
Where is natural language?
Computably Enumerable
Context-sensitive
Context-free
Regular
Finite
MSO
FO(prec)
FO(succ)
Prop(succ) Prop(prec)
CNL(succ) CNL(prec)
Finite
1 Morpho-phonology is regular (Johnson 1972, Kaplan andKay 1994, Roark and Sproat 2007, a.o.)
2 Syntax is mildly context sensitive (Joshi 1984, Schieber1985, Joshi, Vijay-Shanker & Weir 1991, Stabler 1997, a.o.)
FLT · SCiL | 2020/01/05 J. Heinz | 19
![Page 31: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/31.jpg)
From the 20th to the 21st Century
FLT · SCiL | 2020/01/05 J. Heinz | 20
![Page 32: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/32.jpg)
Example #1: Stress PatternsRogers and Lambert (2019, JLM)
1 They consider over 100 distinct stress patterns, expressedas regular grammars, from over 700 languages in theStressTyp2 database.
2 They develop methods to factor these grammars intoprimitive constraints. Almost all constraints fall into thesekinds:
1 Strictly Local (no LL)2 Co-Strictly Local (require L)3 Strictly Piecewise (no H. . . L)4 Co-Strictly Piecewise (require H. . . L)
3 See also their 2019 MoL paper.
FLT · SCiL | 2020/01/05 J. Heinz | 21
![Page 33: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/33.jpg)
Example #1: Stress PatternsRogers and Lambert (2019, JLM)
1 They consider over 100 distinct stress patterns, expressedas regular grammars, from over 700 languages in theStressTyp2 database.
2 They develop methods to factor these grammars intoprimitive constraints. Almost all constraints fall into thesekinds:
1 Strictly Local (no LL)2 Co-Strictly Local (require L)3 Strictly Piecewise (no H. . . L)4 Co-Strictly Piecewise (require H. . . L)
3 See also their 2019 MoL paper.
FLT · SCiL | 2020/01/05 J. Heinz | 21
![Page 34: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/34.jpg)
Stress patterns are not just regular, theybelong to distinct sub-regular classes.
Regular Languages
Stress patterns satisfying
SL, coSL, SP, coSP constraints
FLT · SCiL | 2020/01/05 J. Heinz | 22
![Page 35: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/35.jpg)
Example #2: Local and Long-distance stringtransformations
Some facts
1 In phonology, both local and non-local assimilations occur(post-nasal voicing, consonant harmony, . . . )
2 In syntax, both local and non-local dependencies exist(selection, wh-movement, . . . )
3 There is also copying (reduplication). . .
Questions
1 What are (possible) phonological processes? Syntacticdependencies?
2 How arbitrary can they be?
FLT · SCiL | 2020/01/05 J. Heinz | 23
![Page 36: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/36.jpg)
What is Local?
Computably Enumerable
Context-sensitive
Context-free
Regular
Finite
MSO
FO(prec)
FO(succ)
Prop(succ) Prop(prec)
CNL(succ) CNL(prec)
Finite
1 The 20th century gave us local and long-distance dependenciesin (sets of) sequences
2 But it wasn’t until the 21st century that a theory ofMarkovian/Strictly Local string-to-string functions wasdeveloped (Chandlee 2014 et seq.)
FLT · SCiL | 2020/01/05 J. Heinz | 24
![Page 37: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/37.jpg)
Input/Output Strictly Local Functions
u
b a b b a ba aaa b... ...
x
b a b b a ba aaa b... ...
Chandlee 2014 et seq.
FLT · SCiL | 2020/01/05 J. Heinz | 25
![Page 38: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/38.jpg)
Input/Output Strictly Local Functions
u
b a b b a ba aaa b... ...
x
b a b a ba aaa b... ...b
Chandlee 2014 et seq.
FLT · SCiL | 2020/01/05 J. Heinz | 25
![Page 39: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/39.jpg)
How much Phonology is in there?
Regular Functions
Input/Output Strictly Local Functions
Graf (2020, SCiL) extend this notion of locality to treefunctions to characterize subcategorization in syntactic structures
FLT · SCiL | 2020/01/05 J. Heinz | 26
![Page 40: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/40.jpg)
How much Phonology is in there?
Regular Functions
Input/Output Strictly Local Functions
Graf (2020, SCiL) extend this notion of locality to treefunctions to characterize subcategorization in syntactic structures
FLT · SCiL | 2020/01/05 J. Heinz | 26
![Page 41: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/41.jpg)
What is Non-Local?
There are different types of long-distance dependencies oversstrings just like there are different types of non-linear numericalfunctions.
1 Tier-based Strictly Local Functions (McMullin a.o.)
2 Strictly Piecewise Functions (Burness and McMullin, SCiL2020)
3 Subsequential Functions (Mohri 1997 et seq.)
4 Subclasses of 2way FSTs (Dolatian and Heinz 2018 et seq.)
5 . . .
FLT · SCiL | 2020/01/05 J. Heinz | 27
![Page 42: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/42.jpg)
What is Non-Local?
There are different types of long-distance dependencies oversstrings just like there are different types of non-linear numericalfunctions.
1 Tier-based Strictly Local Functions (McMullin a.o.)
2 Strictly Piecewise Functions (Burness and McMullin, SCiL2020)
3 Subsequential Functions (Mohri 1997 et seq.)
4 Subclasses of 2way FSTs (Dolatian and Heinz 2018 et seq.)
5 . . .
FLT · SCiL | 2020/01/05 J. Heinz | 27
![Page 43: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/43.jpg)
Example #3: Parallels betweenSyntax and Phonology
Non Regular
Regular
CNL(X) / QF(X)
(Appropriately Subregular)
strings
P
S
(Chomsky 1957, Johnson 1972, Kaplan and Kay 1994, Roark and
Sproat 2007, and many others)
FLT · SCiL | 2020/01/05 J. Heinz | 28
![Page 44: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/44.jpg)
Example #3: Parallels betweenSyntax and Phonology
Non Regular
Regular
CNL(X) / QF(X)
(Appropriately Subregular)
strings
S
P
(Potts and Pullum 2002, Heinz 2007 et seq., Graf 2010, Rogers et al.
2010, 2013, Rogers and Lambert, and many others)
FLT · SCiL | 2020/01/05 J. Heinz | 28
![Page 45: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/45.jpg)
Example #3: Parallels betweenSyntax and Phonology
Non Regular
Regular
CNL(X) / QF(X)
(Appropriately Subregular)
stringstrees
P
S
(Rogers 1994, 1998, Knight and Graehl 2005, Pullum 2007, Kobele
2011, Graf 2011 and many others)
FLT · SCiL | 2020/01/05 J. Heinz | 28
![Page 46: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/46.jpg)
Example #3: Parallels betweenSyntax and Phonology
Non Regular
Regular
CNL(X) / QF(X)
(Appropriately Subregular)
stringstrees
PS
(Graf 2013, 2017, Vu et al. 2019, Shafiei and Graf 2020, and others)
FLT · SCiL | 2020/01/05 J. Heinz | 28
![Page 47: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/47.jpg)
Example #4: Other Applications
Understanding Neural Networks:
1 Merril (2019) analyzes the asymptotic behavior of RNNs interms of regular languages.
2 Rabusseau et al. (2019 AISTATS) proves 2nd-order RNNsare equivalent to weighted finite-state machines.
3 Nelson et al. (2020) (SCiL) use Dolatian’s analysis ofreduplication with 2way finite-state transducers to betterunderstand what and how RNNs with and withoutattention can learn.
FLT · SCiL | 2020/01/05 J. Heinz | 29
![Page 48: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/48.jpg)
Example #4: Other Applications
Understanding Neural Networks:
1 Merril (2019) analyzes the asymptotic behavior of RNNs interms of regular languages.
2 Rabusseau et al. (2019 AISTATS) proves 2nd-order RNNsare equivalent to weighted finite-state machines.
3 Nelson et al. (2020) (SCiL) use Dolatian’s analysis ofreduplication with 2way finite-state transducers to betterunderstand what and how RNNs with and withoutattention can learn.
FLT · SCiL | 2020/01/05 J. Heinz | 29
![Page 49: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/49.jpg)
Example #4: Other Applications
Understanding Neural Networks:
1 Merril (2019) analyzes the asymptotic behavior of RNNs interms of regular languages.
2 Rabusseau et al. (2019 AISTATS) proves 2nd-order RNNsare equivalent to weighted finite-state machines.
3 Nelson et al. (2020) (SCiL) use Dolatian’s analysis ofreduplication with 2way finite-state transducers to betterunderstand what and how RNNs with and withoutattention can learn.
FLT · SCiL | 2020/01/05 J. Heinz | 29
![Page 50: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/50.jpg)
Summary of this Part
What is it that we know when we know a language?
1 Mathematical linguistics in the 20th century, and so farinto the 21st century, continues to give us essential insightsinto (nearly) universal properties of natural languages.
2 This is accomplished by dividing the logically possiblespace of generalizations into categories and studying wherethe natural language generalizations occur.
3 The properties are not about a particular formalism (likefinite-state vs. regular expressions vs. rules vs. OT) butmore about conditions on grammars. What must/shouldany grammar at least be sensitive to? What can/should beignored?
4 Because it’s math, it is verifiable, interpretable, analyzable& understandable, and is thus used to understandcomplicated systems (like natural languages and NNs).
FLT · SCiL | 2020/01/05 J. Heinz | 30
![Page 51: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/51.jpg)
Summary of this Part
What is it that we know when we know a language?
1 Mathematical linguistics in the 20th century, and so farinto the 21st century, continues to give us essential insightsinto (nearly) universal properties of natural languages.
2 This is accomplished by dividing the logically possiblespace of generalizations into categories and studying wherethe natural language generalizations occur.
3 The properties are not about a particular formalism (likefinite-state vs. regular expressions vs. rules vs. OT) butmore about conditions on grammars. What must/shouldany grammar at least be sensitive to? What can/should beignored?
4 Because it’s math, it is verifiable, interpretable, analyzable& understandable, and is thus used to understandcomplicated systems (like natural languages and NNs).
FLT · SCiL | 2020/01/05 J. Heinz | 30
![Page 52: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/52.jpg)
Summary of this Part
What is it that we know when we know a language?
1 Mathematical linguistics in the 20th century, and so farinto the 21st century, continues to give us essential insightsinto (nearly) universal properties of natural languages.
2 This is accomplished by dividing the logically possiblespace of generalizations into categories and studying wherethe natural language generalizations occur.
3 The properties are not about a particular formalism (likefinite-state vs. regular expressions vs. rules vs. OT) butmore about conditions on grammars. What must/shouldany grammar at least be sensitive to? What can/should beignored?
4 Because it’s math, it is verifiable, interpretable, analyzable& understandable, and is thus used to understandcomplicated systems (like natural languages and NNs).
FLT · SCiL | 2020/01/05 J. Heinz | 30
![Page 53: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/53.jpg)
Summary of this Part
What is it that we know when we know a language?
1 Mathematical linguistics in the 20th century, and so farinto the 21st century, continues to give us essential insightsinto (nearly) universal properties of natural languages.
2 This is accomplished by dividing the logically possiblespace of generalizations into categories and studying wherethe natural language generalizations occur.
3 The properties are not about a particular formalism (likefinite-state vs. regular expressions vs. rules vs. OT) butmore about conditions on grammars. What must/shouldany grammar at least be sensitive to? What can/should beignored?
4 Because it’s math, it is verifiable, interpretable, analyzable& understandable, and is thus used to understandcomplicated systems (like natural languages and NNs).
FLT · SCiL | 2020/01/05 J. Heinz | 30
![Page 54: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/54.jpg)
Part IV
Learning Problems
FLT · SCiL | 2020/01/05 J. Heinz | 31
![Page 55: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/55.jpg)
Questions about learning
Motivating question
How do we come by our knowledge of language?
Questions about learning
1 What does it mean to learn?
2 How can learning be a formalized as a problem and solved(like the problem of sorting lists)?
FLT · SCiL | 2020/01/05 J. Heinz | 32
![Page 56: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/56.jpg)
Some answers from before the 21st century:Computational Learning Theory
1 Identifications in the Limit (Gold 1967)
2 Active/Query Learning (Angluin 1988)
3 Probably Approximately Correct (PAC) Learning (Valiant1984)
4 Optimizing Objective Functions
5 . . .
FLT · SCiL | 2020/01/05 J. Heinz | 33
![Page 57: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/57.jpg)
In Pictures
algorithmlearning
A M
S
D
no
Is x in S?
yes
for any S belonging to a class C?
FLT · SCiL | 2020/01/05 J. Heinz | 34
![Page 58: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/58.jpg)
Many, many methods
1 Connectionism/Associative Learning (Rosenblatt 1959,McClelland and Rumelhart 1986, Kapatsinski 2018, a.o.)
2 Bayesian methods (Bishop 2006, Kemp and Tenenbaum2008, a.o.)
3 Probabilistic Graphical Models (Pearl 1988, Koller andFriedman 2010, a.o.)
4 State-merging (Feldman 1972, Angluin 1982, Oncina et al1992, a.o.)
5 Statistical Relational Learning (De Raedt 2008, a.o.)
6 Minimum Description Length (Risannen 1978, Goldsmitha.o..)
7 Suport Vector Machines (Vapnik 1995, 1998 a.o.)
8 . . .
FLT · SCiL | 2020/01/05 J. Heinz | 35
![Page 59: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/59.jpg)
Many, many methods
Newer methods
1 Deep NNs (LeCun et al. 2015, Schmidhuber 2015,Goodfellow et al. 2016, a. MANY o.)
• encoder-decoder networks• generative adversarial networks• . . .
2 Spectral Learning (Hsu et al 2009, Balle et al. 2012, 2014,a.o.)
3 Distributional Learning (Clark and Yoshinaka 2016, a.o.)
4 . . .
FLT · SCiL | 2020/01/05 J. Heinz | 35
![Page 60: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/60.jpg)
Computational Learning Theory
algorithmlearning
A M
S
D
no
Is x in S?
yes
CLT studies conditions on learning mechanisms/methods!
FLT · SCiL | 2020/01/05 J. Heinz | 36
![Page 61: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/61.jpg)
The main lesson from CLT
There is no free lunch.
1 There is no algorithm that can feasibly learn any patternP, even with lots of data from P.
2 But—There are algorithms that can feasibly learn patternswhich belong to a suitably structured class C.
Gold 1967, Angluin 1980, Valiant 1984,Wolpert and McReady 1997, a.o.
FLT · SCiL | 2020/01/05 J. Heinz | 37
![Page 62: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/62.jpg)
The main lesson from CLT
There is no free lunch.
1 There is no algorithm that can feasibly learn any patternP, even with lots of data from P.
2 But—There are algorithms that can feasibly learn patternswhich belong to a suitably structured class C.
Gold 1967, Angluin 1980, Valiant 1984,Wolpert and McReady 1997, a.o.
FLT · SCiL | 2020/01/05 J. Heinz | 37
![Page 63: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/63.jpg)
The Perpetual Motion Machine
October 1920 issue of PopularScience magazine, on perpetualmotion.
“Although scientists have estab-lished them to be impossible un-der the laws of physics, perpet-ual motion continues to capturethe imagination of inventors.”
https://en.wikipedia.org/
wiki/Perpetual_motion
FLT · SCiL | 2020/01/05 J. Heinz | 38
![Page 64: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/64.jpg)
The Perpetual Misconception Machine
∃machine-learning algorithm A, ∀ patterns P with enough dataD from P : A(D) ≈ P .
1 It’s just not true.
2 What is true is this:∀ patterns P , ∃ data D and ML A : A(D) ≈ P .
3 In practice, the misconception means searching for A and Dso that your approximation is better than everyone else’s.
4 With next pattern P ′, we will have no guarantee A willwork, we will have to search again.
FLT · SCiL | 2020/01/05 J. Heinz | 39
![Page 65: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/65.jpg)
The Perpetual Misconception Machine
∃machine-learning algorithm A, ∀ patterns P with enough dataD from P : A(D) ≈ P .
1 It’s just not true.
2 What is true is this:∀ patterns P , ∃ data D and ML A : A(D) ≈ P .
3 In practice, the misconception means searching for A and Dso that your approximation is better than everyone else’s.
4 With next pattern P ′, we will have no guarantee A willwork, we will have to search again.
FLT · SCiL | 2020/01/05 J. Heinz | 39
![Page 66: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/66.jpg)
The Perpetual Misconception Machine
∃machine-learning algorithm A, ∀ patterns P with enough dataD from P : A(D) ≈ P .
1 It’s just not true.
2 What is true is this:∀ patterns P , ∃ data D and ML A : A(D) ≈ P .
3 In practice, the misconception means searching for A and Dso that your approximation is better than everyone else’s.
4 With next pattern P ′, we will have no guarantee A willwork, we will have to search again.
FLT · SCiL | 2020/01/05 J. Heinz | 39
![Page 67: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/67.jpg)
The Perpetual Misconception Machine
∃machine-learning algorithm A, ∀ patterns P with enough dataD from P : A(D) ≈ P .
1 It’s just not true.
2 What is true is this:∀ patterns P , ∃ data D and ML A : A(D) ≈ P .
3 In practice, the misconception means searching for A and Dso that your approximation is better than everyone else’s.
4 With next pattern P ′, we will have no guarantee A willwork, we will have to search again.
FLT · SCiL | 2020/01/05 J. Heinz | 39
![Page 68: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/68.jpg)
Computational Laws of Learning
Feasibly solving a learning problem requires defining a targetclass C of patterns.
1 The class C cannot be all patterns, or even all computablepatterns.
2 Class C must have more structure, and many logicallypossible patterns must be outside of C.
3 There is no avoiding prior knowledge.
4 Do not “confuse ignorance of biases with absence ofbiases.” (Rawski and Heinz 2019)
FLT · SCiL | 2020/01/05 J. Heinz | 40
![Page 69: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/69.jpg)
Computational Laws of Learning
Feasibly solving a learning problem requires defining a targetclass C of patterns.
1 The class C cannot be all patterns, or even all computablepatterns.
2 Class C must have more structure, and many logicallypossible patterns must be outside of C.
3 There is no avoiding prior knowledge.
4 Do not “confuse ignorance of biases with absence ofbiases.” (Rawski and Heinz 2019)
FLT · SCiL | 2020/01/05 J. Heinz | 40
![Page 70: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/70.jpg)
Computational Laws of Learning
Feasibly solving a learning problem requires defining a targetclass C of patterns.
1 The class C cannot be all patterns, or even all computablepatterns.
2 Class C must have more structure, and many logicallypossible patterns must be outside of C.
3 There is no avoiding prior knowledge.
4 Do not “confuse ignorance of biases with absence ofbiases.” (Rawski and Heinz 2019)
FLT · SCiL | 2020/01/05 J. Heinz | 40
![Page 71: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/71.jpg)
Computational Laws of Learning
Feasibly solving a learning problem requires defining a targetclass C of patterns.
1 The class C cannot be all patterns, or even all computablepatterns.
2 Class C must have more structure, and many logicallypossible patterns must be outside of C.
3 There is no avoiding prior knowledge.
4 Do not “confuse ignorance of biases with absence ofbiases.” (Rawski and Heinz 2019)
FLT · SCiL | 2020/01/05 J. Heinz | 40
![Page 72: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/72.jpg)
In Pictures: Given ML algorithm A
All patterns
p1 p2
C
p2
D1 from p1 D2 from p2
A(D1)A(D2)
FLT · SCiL | 2020/01/05 J. Heinz | 41
![Page 73: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/73.jpg)
In Pictures: Given ML algorithm A
All patterns
p1
p2
C
p2
D1 from p1 D2 from p2
A(D1)A(D2)
FLT · SCiL | 2020/01/05 J. Heinz | 41
![Page 74: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/74.jpg)
In Pictures: Given ML algorithm A
All patterns
p1 p2
C
p2
D1 from p1 D2 from p2
A(D1)A(D2)
FLT · SCiL | 2020/01/05 J. Heinz | 41
![Page 75: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/75.jpg)
In Pictures: Given ML algorithm A
All patterns
p1 p2
C
p2
D1 from p1 D2 from p2
A(D1)A(D2)
FLT · SCiL | 2020/01/05 J. Heinz | 41
![Page 76: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/76.jpg)
In Pictures: Given ML algorithm A
All patterns
p1 p2
C
p2
D1 from p1 D2 from p2
A(D1)A(D2)
FLT · SCiL | 2020/01/05 J. Heinz | 41
![Page 77: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/77.jpg)
In Pictures: Given ML algorithm A
All patterns
p1 p2
C
p2
D1 from p1 D2 from p2
A(D1)
A(D2)
FLT · SCiL | 2020/01/05 J. Heinz | 41
![Page 78: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/78.jpg)
In Pictures: Given ML algorithm A
All patterns
p1 p2
C
p2
D1 from p1 D2 from p2
A(D1)A(D2)
FLT · SCiL | 2020/01/05 J. Heinz | 41
![Page 79: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/79.jpg)
The Perpetual Misconception Machine
When you believe in things that you don’t understandthen you suffer.
– Stevie Wonder
FLT · SCiL | 2020/01/05 J. Heinz | 42
![Page 80: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/80.jpg)
Go smaller, not bigger!
All patterns
C
FLT · SCiL | 2020/01/05 J. Heinz | 43
![Page 81: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/81.jpg)
Go smaller, not bigger!
All patterns
C
FLT · SCiL | 2020/01/05 J. Heinz | 43
![Page 82: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/82.jpg)
Go smaller, not bigger!
All patterns
C
FLT · SCiL | 2020/01/05 J. Heinz | 43
![Page 83: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/83.jpg)
Dana Angluin
1 Characterized those classes identifiable in the Limit fromPositive Data (1980).
2 Introduced first non-trivial infinite class of languagesidentifiable in the limit from positive data with an efficientalgorithm (1982).
3 Introduced Query Learning; Problem and Solution(1987a,b).
4 Studied learning with noise, from stochastic examples(1988a,b).
FLT · SCiL | 2020/01/05 J. Heinz | 44
![Page 84: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/84.jpg)
Grammatical Inference
ICGI 2020 in NYC August 26-28!!https://grammarlearning.org/
FLT · SCiL | 2020/01/05 J. Heinz | 45
![Page 85: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/85.jpg)
From the 20th to the 21st Century
FLT · SCiL | 2020/01/05 J. Heinz | 46
![Page 86: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/86.jpg)
Example #1: SL, SP, TSL; ISL, OSL, I-TSL,O-TSL
1 These classes are parameterized by a window size k.
2 The k-classes are efficiently learnable from positiveexamples under multiple paradigms.
(Garcia et al. 1991, Heinz 2007 et seq., Chandlee et al. 2014,2015, Jardine and McMullin 2017, Burness and McMullin 2019a.o.)
FLT · SCiL | 2020/01/05 J. Heinz | 47
![Page 87: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/87.jpg)
Example #2: Other Applications
1 Using Grammatical Inference to understand NeuralNetworks.
1 Weiss et al. (2018, 2019) use Angluin’s L* (1987) algorithm(and more) to model behavior of trained NNs with FSMs.
2 Eyraud et al. (2018) use spectral learning to modelbehavior of trained NNs with FSMs.
2 Model checking, software verification, integration intorobotic planning and control, and so on.
FLT · SCiL | 2020/01/05 J. Heinz | 48
![Page 88: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/88.jpg)
Example #2: ISL Optionality
• For a given k, the k-ISL class of functions is identifiable inthe limit in linear time and data.
• Functions are single-valued, no?
• So what about optionality which is rife in naturallanguages?
(work in progress with Kiran Eiden and Eric Schieferstein)
FLT · SCiL | 2020/01/05 J. Heinz | 49
![Page 89: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/89.jpg)
Deterministic FSTs with Language Monoids
Optional Post-nasal Voicing(Non-deterministic)
1start 2
n:np:p
a:a
p:p
p:b
a:a
n:n
FLT · SCiL | 2020/01/05 J. Heinz | 50
![Page 90: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/90.jpg)
Deterministic FSTs with Language Monoids
Optional Post-nasal Voicing(Non-deterministic)
1start 2
n:np:p
a:a
p:p
p:b
a:a
n:n
/ a n p a /
1 1 21 1
1 1a n
p:pa
p:b a
FLT · SCiL | 2020/01/05 J. Heinz | 50
![Page 91: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/91.jpg)
Deterministic FSTs with Language Monoids
Optional Post-nasal Voicing(Deterministic)
1start 2
n:{n}p:{p}
a:{a}
p:{p,b}
a:{a}
n:{n}
Beros and de la Higuera (2016) call this ‘semi-determinism’.
FLT · SCiL | 2020/01/05 J. Heinz | 50
![Page 92: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/92.jpg)
Deterministic FSTs with Language Monoids
Optional Post-nasal Voicing(Deterministic)
1start 2
n:{n}p:{p}
a:{a}
p:{p,b}
a:{a}
n:{n}
/ a n p a /
1 1 2 1 1{a} {n} {p,b} {a}
FLT · SCiL | 2020/01/05 J. Heinz | 50
![Page 93: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/93.jpg)
Deterministic FSTs with Language Monoids
Optional Post-nasal Voicing(Deterministic)
1start 2
n:{n}p:{p}
a:{a}
p:{p,b}
a:{a}
n:{n}
/ a n p a / 7→ {a} · {n} · {p, b} · {a} = {anpa, anba}
FLT · SCiL | 2020/01/05 J. Heinz | 50
![Page 94: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/94.jpg)
Iterative Optionality is more challenging
Vaux 2008, p. 43
FLT · SCiL | 2020/01/05 J. Heinz | 51
![Page 95: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/95.jpg)
Abstract Example
V → ∅ / VC CV (applying left-to-right)
/cvcv/ /cvcvcv/ /cvcvcvcv/ /cvcvcvcvcv/
cvcv cvcvcv cvcvcvcv cvcvcvcvcv faithfulcvccv cvccvcv cvccvcvcv 2nd vowel deletes
cvcvccv cvcvccvcv 3rd vowel deletescvccvccv 2nd, 4th vowels delete
FLT · SCiL | 2020/01/05 J. Heinz | 52
![Page 96: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/96.jpg)
Problem: Output-oriented Optionality
1start
2
3 4
5
6
c:c
v:v
v:v
c:c
v:v
v:λc:c
v:v
• The output determines the state!
• 4 ??v:{v,λ}
• For deterministic transducers, the next state isnecessarily determined by the input symbol!
FLT · SCiL | 2020/01/05 J. Heinz | 53
![Page 97: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/97.jpg)
What would Kisseberth say?
Kisseberth 1970: 304-305
By making . . . rules meet two conditions (one relating to theform of the input string and the other relating to the formof the output string; one relating to a single rule, the otherrelating to all the rules in the grammar), we are able to writethe vowel deletion rules in the intuitively correct fashion. Wedo not have to mention in the rules themselves that they cannotyield unpermitted clusters. We state this fact once in the formof a derivational constraint.
FLT · SCiL | 2020/01/05 J. Heinz | 54
![Page 98: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/98.jpg)
Learn the ISL function and surfaceconstraints independently and simultaneously
Strategy
Learn an Input-based function (T1) and filter the outputs withphonotactic constraints (T2).
T1 ◦ T2 = target
FLT · SCiL | 2020/01/05 J. Heinz | 55
![Page 99: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/99.jpg)
Learning the ISL function
• Algorithm synthesizes aspects of Jardine et al. (2014) andBeros and de la Higuera (2016).
Before Learning
λstart c cv
cvc vcv
nc:� v:�
c:�
n:�
v:�
c:�
n : �
Input Strictly Local Transducer with 4-size window
FLT · SCiL | 2020/01/05 J. Heinz | 56
![Page 100: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/100.jpg)
Learning the ISL function
• Learns to optionally delete every vowel except the 1st!!
After Learning
λstart c cv
cvc vcv
nc:{cv} v:{λ}
c:{c}
n:{λ}
v:{λ}
c:{c,vc}
n : {v, λ}
FLT · SCiL | 2020/01/05 J. Heinz | 56
![Page 101: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/101.jpg)
Summary of this example
1 Optional processes can be deterministic.(Multi-valued 6→ non-deterministic.)Deterministic String Relations
2 Non-decomposed, output-oriented, optional processes arenon-deterministic.
3 But they can be factored into a deterministic process whichovergenerates and a constraint which filters out theunwanted overgenerates.
T = T1 ◦ T2
4 T2 can be learned with existing grammatical inferencemethods.
5 T1 appears to be learnable with a synthesis of recent resultsin grammatical inference.
FLT · SCiL | 2020/01/05 J. Heinz | 57
![Page 102: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/102.jpg)
Summary of this example
1 Optional processes can be deterministic.(Multi-valued 6→ non-deterministic.)Deterministic String Relations
2 Non-decomposed, output-oriented, optional processes arenon-deterministic.
3 But they can be factored into a deterministic process whichovergenerates and a constraint which filters out theunwanted overgenerates.
T = T1 ◦ T2
4 T2 can be learned with existing grammatical inferencemethods.
5 T1 appears to be learnable with a synthesis of recent resultsin grammatical inference.
FLT · SCiL | 2020/01/05 J. Heinz | 57
![Page 103: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/103.jpg)
Summary of this example
1 Optional processes can be deterministic.(Multi-valued 6→ non-deterministic.)Deterministic String Relations
2 Non-decomposed, output-oriented, optional processes arenon-deterministic.
3 But they can be factored into a deterministic process whichovergenerates and a constraint which filters out theunwanted overgenerates.
T = T1 ◦ T2
4 T2 can be learned with existing grammatical inferencemethods.
5 T1 appears to be learnable with a synthesis of recent resultsin grammatical inference.
FLT · SCiL | 2020/01/05 J. Heinz | 57
![Page 104: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/104.jpg)
Summary of this example
1 Optional processes can be deterministic.(Multi-valued 6→ non-deterministic.)Deterministic String Relations
2 Non-decomposed, output-oriented, optional processes arenon-deterministic.
3 But they can be factored into a deterministic process whichovergenerates and a constraint which filters out theunwanted overgenerates.
T = T1 ◦ T2
4 T2 can be learned with existing grammatical inferencemethods.
5 T1 appears to be learnable with a synthesis of recent resultsin grammatical inference.
FLT · SCiL | 2020/01/05 J. Heinz | 57
![Page 105: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/105.jpg)
Summary of this example
1 Optional processes can be deterministic.(Multi-valued 6→ non-deterministic.)Deterministic String Relations
2 Non-decomposed, output-oriented, optional processes arenon-deterministic.
3 But they can be factored into a deterministic process whichovergenerates and a constraint which filters out theunwanted overgenerates.
T = T1 ◦ T2
4 T2 can be learned with existing grammatical inferencemethods.
5 T1 appears to be learnable with a synthesis of recent resultsin grammatical inference.
FLT · SCiL | 2020/01/05 J. Heinz | 57
![Page 106: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/106.jpg)
Discussion
1 Formal proof of correctness of the algorithm for learningclasses of structured multi-valued functions is in progress.
2 Probabilities can be appended to the outputs for learningclasses of functions.
3 We hope to apply to other problems:1 learning URs and phonological grammars simultaneously2 sociolinguistic variation3 NLP problems such as G2P, P2G, and so on.
realizationabstraction
Problem Solution
Complicated messy system
FLT · SCiL | 2020/01/05 J. Heinz | 58
![Page 107: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/107.jpg)
Discussion
1 Formal proof of correctness of the algorithm for learningclasses of structured multi-valued functions is in progress.
2 Probabilities can be appended to the outputs for learningclasses of functions.
3 We hope to apply to other problems:1 learning URs and phonological grammars simultaneously2 sociolinguistic variation3 NLP problems such as G2P, P2G, and so on.
realizationabstraction
Problem Solution
Complicated messy system
FLT · SCiL | 2020/01/05 J. Heinz | 58
![Page 108: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/108.jpg)
Discussion
1 Formal proof of correctness of the algorithm for learningclasses of structured multi-valued functions is in progress.
2 Probabilities can be appended to the outputs for learningclasses of functions.
3 We hope to apply to other problems:1 learning URs and phonological grammars simultaneously2 sociolinguistic variation3 NLP problems such as G2P, P2G, and so on.
realizationabstraction
Problem Solution
Complicated messy system
FLT · SCiL | 2020/01/05 J. Heinz | 58
![Page 109: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/109.jpg)
Discussion
1 Formal proof of correctness of the algorithm for learningclasses of structured multi-valued functions is in progress.
2 Probabilities can be appended to the outputs for learningclasses of functions.
3 We hope to apply to other problems:1 learning URs and phonological grammars simultaneously2 sociolinguistic variation3 NLP problems such as G2P, P2G, and so on.
realizationabstraction
Problem Solution
Complicated messy system
FLT · SCiL | 2020/01/05 J. Heinz | 58
![Page 110: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/110.jpg)
Part V
A Summary of Sorts
FLT · SCiL | 2020/01/05 J. Heinz | 59
![Page 111: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/111.jpg)
Personal View
Two seeds in the 20th century
1 Mathematics can be developed to providestronger/tighter characterizations of natural languagepatterns.
2 Computational Learning Theory stresses the importanceand necessity of structured hypothesis spaces. Don’ttreat them cavalierly!
These compatible ideas are bearing fruit well into the 21stcentury.
FLT · SCiL | 2020/01/05 J. Heinz | 60
![Page 112: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/112.jpg)
More 21st century Highlights
1 Abstract Categorial Grammars (De Groote 2001)
2 The Syntactic Concept Lattice (Clark 2013)
3 The Tolerance Principle (Yang 2016)
4 . . .
FLT · SCiL | 2020/01/05 J. Heinz | 61
![Page 113: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/113.jpg)
Conclusion
1 Far from being a fossil from a former era, mathematicalthinking about language
1 continues to play an essential role in understanding naturallanguages and
2 continues to make critical contributions to ourunderstanding of how things which compute—both humansand machines—can learn.
realizationabstraction
Problem Solution
Complicated messy system
FLT · SCiL | 2020/01/05 J. Heinz | 62
![Page 114: Mathematical Linguistics in the 21st Centuryjeffreyheinz.net/talks/Heinz-2020-ML21C-slides.pdf · Mathematical Linguistics in the 21st Century FLT SCiL j2020/01/05 J. Heinz j8. Timeframe](https://reader030.vdocuments.mx/reader030/viewer/2022041014/5ec59dd868eec74f36141520/html5/thumbnails/114.jpg)
The End
Thanks for listening, and thanks to everyone I have ever spokenwith: students, mentors and peers.
Let’s discuss more on the Outdex!
https://outde.xyz/
FLT · SCiL | 2020/01/05 J. Heinz | 63