encodings and ambiguity communication across different priors “ implicature ” arises naturally
DESCRIPTION
Compression Without a Common Prior An information-theoretic justification for ambiguity in language. Brendan Juba (MIT CSAIL & Harvard) with Adam Kalai (MSR) Sanjeev Khanna (Penn) Madhu Sudan (MSR & MIT). Encodings and ambiguity Communication across different priors - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/1.jpg)
Compression Without a Common Prior
An information-theoretic justification for ambiguity in language
Brendan Juba (MIT CSAIL & Harvard)with Adam Kalai (MSR)Sanjeev Khanna (Penn)
Madhu Sudan (MSR & MIT)
![Page 2: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/2.jpg)
2
1.Encodings and ambiguity
2.Communication across different priors
3.“Implicature” arises naturally
![Page 3: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/3.jpg)
3
Encoding schemes
BirdChicken Cat Dinner Pet LambDuck Cow Dog
“MESSAGES”
“ENCODINGS”
![Page 4: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/4.jpg)
4
Communication model
CAT
RECALL: ( , CAT) E
![Page 5: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/5.jpg)
5
Ambiguity
BirdChicken Cat Dinner Pet LambDuck Cow Dog
![Page 6: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/6.jpg)
6
WHAT GOOD IS AN
AMBIGUOUS ENCODING??
![Page 7: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/7.jpg)
7
Prior distributions
BirdChicken Cat Dinner Pet LambDuck Cow Dog
Decode to a maximum likelihood message
![Page 8: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/8.jpg)
8
Source coding (compression)
• Assume encodings are binary strings• Given a prior distribution P, message m,
choose minimum length encoding that decodes to m.
FOR EXAMPLE, HUFFMAN CODES AND SHANNON-FANO (ARITHMETIC) CODES
NOTE: THE ABOVE SCHEMES DEPEND ON THE PRIOR.
![Page 9: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/9.jpg)
9
More generally…
Unambiguous encoding schemes cannot be too efficient. In a set of M distinct messages, some message must have an encoding of length lg M.
+If a prior places high weight on that message, we aren’t compressing well.
![Page 10: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/10.jpg)
≈
≈
SINCE WE ALL AGREE ON A PROB. DISTRIBUTION OVER WHAT I MIGHT SAY, I CAN COMPRESS IT TO: “THE
9,232,142,124,214,214,123,845TH MOST LIKELY MESSAGE.
THANK YOU!”
![Page 11: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/11.jpg)
12
1.Encodings and ambiguity
2.Communication across different priors
3.“Implicature” arises naturally
![Page 12: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/12.jpg)
13
SUPPOSE ALICE AND BOB SHARE THE SAME ENCODING SCHEME, BUT DON’T SHARE THE SAME PRIOR…
P Q
CAN THEY COMMUNICATE??HOW EFFICIENTLY??
![Page 13: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/13.jpg)
14
Disambiguation property
An encoding scheme has the disambiguation property (for prior P) if for every message m and integer Θ,there exists some encoding e=e(m,Θ) such thatfor every other message m’
P[m|e] > Θ P[m’|e]
WE’LL WANT A SCHEME THAT SATISFIES DISAMBIGUATION
FOR ALL PRIORS.
![Page 14: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/14.jpg)
15
THE CAT.THE ORANGE CAT.THE ORANGE CAT WITHOUT A HAT.
![Page 15: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/15.jpg)
16
Closeness and communication
• Priors P and Q are α-close (α ≥ 1) if for every message m,αP(m) ≥ Q(m) and αQ(m) ≥ P(m)
• The disambiguation property and closeness together suffice for communication
Pick Θ=α2—then, for every m’≠m,Q[m|e] ≥ 1/αP[m|e] > αP[m’|e] ≥ Q[m’|e]
SO, IF ALICE SENDS e THEN MAXIMUM LIKELIHOOD DECODING
GIVES BOB m AND NOT m’…
![Page 16: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/16.jpg)
17
Constructing an encoding scheme.
(Inspired by Braverman-Rao)
Pick an infinite random string Rm for each m,Put (m,e) E e is a prefix of R⇔ m.
Alice encodes m by sending prefix of Rm s.t.m is α2-disambiguated under P.
COLLISIONS IN A COUNTABLE SET OF MESSAGES HAVE MEASURE ZERO, SO CORRECTNESS IS IMMEDIATE.
CAN BE PARTIALLY DERANDOMIZED
BY UNIVERSAL HASH FAMILY. SEE
PAPER!
![Page 17: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/17.jpg)
18
AnalysisClaim. Expected encoding length is at most
H(P) + 2log α + 2Proof. There are at most α2/P[m] messages with P-probability at least P[m]/α2. By a union bound, the probability that any of these agree with Rm in the first log α2/P[m]+k bits is at most 2-k.
E[|e(m)|] ≤ log α2/P[m] +2
So: ΣkPr[|e(m)| ≥ log α2/P[m]+k] ≤ 2
![Page 18: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/18.jpg)
19
Remark
Mimicking the disambiguation property of natural language provided an efficient strategy for communication.
![Page 19: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/19.jpg)
20
1.Encodings and ambiguity
2.Communication across different priors
3.“Implicature” arises naturally
![Page 20: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/20.jpg)
21
Motivation
If one message dominates in the prior, we know it receives a short encoding. Do we really need to consider it for disambiguation at greater encoding lengths?
PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU, PIKACHU,
PIKACHU…
![Page 21: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/21.jpg)
22
Higher-order decoding
• Suppose Bob knows Alice has an α-close prior, and that she only sends α2-disambiguated encodings of her messages.
☞ If a message m is α4-disambiguated under Q,P[m|e] ≥ 1/αQ[m|e] > α3Q[m’|e] ≥ α2P[m’|e]So Alice won’t use an encoding longer than e!
☞Bob “filters” m from consideration elsewhere: constructs EB by deleting these edges.
![Page 22: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/22.jpg)
23
Higher-order encoding
• Suppose Alice knows Bob filters out the α4-disambiguated messages
☞If a message m is α6-disambiguated under P, Alice knows Bob won’t consider it.
☞So, Alice can filter out all α6-disambiguated messages: construct EA by deleting these edges
![Page 23: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/23.jpg)
Higher-order communication
• Sending. Alice sends an encoding e s.t. m is α2-disambiguated w.r.t. P and EA
• Receiving. Bob recovers m’ with maximum Q-probability s.t. (m’,e) EB
![Page 24: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/24.jpg)
25
Correctness
• Alice only filters edges she knows Bob has filtered, so EA E⊇ B. ⇒So m, if available, is maximum likelihood message
• Likewise, if m was not α2-disambiguated before e, at all shorter e’
⇒m is not filtered by Bob before e.∃m’≠m α2P[m’|e’] ≥ P[m|e’] α3Q[m’|e’] ≥ ≥ 1/αQ[m|e’]
![Page 25: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/25.jpg)
26
Conversational Implicature
• When speakers’ “meaning” is more than literally suggested by utterance
• Numerous (somewhat unsatisfactory) accounts given over the years– [Grice] Based on “cooperative principle” axioms– [Sperber-Wilson] Based on “relevance”
☞Our Higher-order scheme shows this effect!
![Page 26: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/26.jpg)
27
Recap. We saw an information-theoretic problem for which our best solutions resembled natural languages in interesting ways.
![Page 27: Encodings and ambiguity Communication across different priors “ Implicature ” arises naturally](https://reader036.vdocuments.mx/reader036/viewer/2022070405/56813d58550346895da71edf/html5/thumbnails/27.jpg)
28
The problem. Design an encoding scheme E so that for any sender and receiver with α-close prior distributions, the communication length is minimized.
(In expectation w.r.t. sender’s distribution)
Questions?