language as an algebra€¦ · quotient algebras finite-dimensional algebras algebras from monoids...

BackgroundContext Theories

Language as an Algebra

Daoud Clarke

Department of Computer ScienceUniversity of Hertfordshire

The Categorical Flow of Information in Quantum Physicsand Linguistics, Oxford, 2010

Daoud Clarke Language as an Algebra

Overview

1 BackgroundDistributional SemanticsContext-theoretic Semantics

2 Context TheoriesMotivating ExampleExample

3 ToolsQuotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids

Distributional SemanticsContext-theoretic Semantics

Beyond lexical distributional semantics

Distributional semantics:Hypothesis of Harris (1968)LSA, distributional similarity etc.Many applicationsGood for words/short phrases

How can we go beyond the lexical domain?Data sparseness

Beyond lexical distributional semantics

Distributional semantics:Hypothesis of Harris (1968)LSA, distributional similarity etc.Many applicationsGood for words/short phrases

How can we go beyond the lexical domain?Data sparseness

Physicists (xkcd.com)

Context-theoretic Semantics

Suggestion: explore the use of unital associative algebrasover the real numbersAn algebra is a vector space together with a bilinearmultiplication:

x · (y + z) = x · y + x · z (x + y) · z = x · z + y · z

Associative algebras form a monoidal category K -Alg

x · (y + z) = x · y + x · z (x + y) · z = x · z + y · z

Al-kitab al-mukhtas.ar fı h. isabi-l-jabr wa’l-muqabala

Algebra over a field

An algebra over a field:abstraction of the space of operators on a vector space

Hilbert space operators→ C∗-algebrasVector lattice operators→ lattice-ordered algebrasMatrices of order n form an algebra under normal matrixmultiplication

Context-theoretic Framework

A context theory is a tuple 〈A,A, , φ〉The meaning of a string x ∈ A∗ is a vector x ∈ A

A∗ is the free monoid on an alphabet AA is a real associative unital algebra

The mapping ˆ from A∗ to A is a monoid homomorphismthe cat sat = the · cat · sat

Multiplication · is distributive with respect to vector spaceaddition (bilinearity)φ is a linear functional on A indicating probability

Motivating ExampleExample

Motivating Example: Context Algebras

Meaning as contextFormal language→ syntactic monoidFuzzy language→ context algebraE.g. Latent Dirichlet Allocation→ commutative algebra

Context Algebras: How it works

Fuzzy language C : A∗ → [0,1]

For x ∈ A∗, define x : A∗ × A∗ → [0,1] by

x(y , z) = C(yxz)

Define A as vector space generated by {x : x ∈ A∗}Choose a basis and use multiplication on A∗ to definemultiplication on A

Multiplication is the same, no matter what basis we choose!

x(y , z) = C(yxz)

Context Algebras: Example

Corpus defined by

C(the cat sat) = 0.8C(the big cat sat) = 0.2

cat = 0.8e(the, sat) + 0.2e(the big, sat)

big · cat = big cat = 0.2e(the, sat)

where e(x , y) is the basis element corresponding to(x , y) ∈ A∗ × A∗.

Context Algebras: Example

Corpus defined by

C(the cat sat) = 0.8C(the big cat sat) = 0.2

cat = 0.8e(the, sat) + 0.2e(the big, sat)

big · cat = big cat = 0.2e(the, sat)

where e(x , y) is the basis element corresponding to(x , y) ∈ A∗ × A∗.

Context-theoretic Semantics: Summary

Map strings to elements of an algebraMotivating example:

Meaning as contextFuzzy language→ context algebra

Lots of other examples in Clarke (2007)

Quotient AlgebrasFinite-dimensional AlgebrasAlgebras from Monoids

Tools for Constructing Algebras

Quotient Algebras (Clarke, Lutz and Weir 2010)Finite dimensional algebras

with David Weir, Rudi Lutz and Ben Campion

Enveloping Algebras (You?)

Quotient Algebras

Construct a free algebra (tensor algebra)

T (V ) = R⊕ V ⊕ (V ⊗ V )⊕ (V ⊗ V ⊗ V )⊕ · · ·

Choose relations u1 = v1,u2 = v2, . . . we wish to hold

Λ = {u1 − v1,u2 − v2, . . .}

Construct ideal I generated by Λ

Take equivalence classes to get quotient algebra

T (V )/I

Quotient Algebras

T (V ) = R⊕ V ⊕ (V ⊗ V )⊕ (V ⊗ V ⊗ V )⊕ · · ·

Λ = {u1 − v1,u2 − v2, . . .}

T (V )/I

Quotient Algebras

T (V ) = R⊕ V ⊕ (V ⊗ V )⊕ (V ⊗ V ⊗ V )⊕ · · ·

Λ = {u1 − v1,u2 − v2, . . .}

T (V )/I

Quotient Algebras

T (V ) = R⊕ V ⊕ (V ⊗ V )⊕ (V ⊗ V ⊗ V )⊕ · · ·

Λ = {u1 − v1,u2 − v2, . . .}

T (V )/I

Quotient Algebra: Why?

Vectors that were orthogonal in T (V ) can benon-orthogonal in T (V )/IStrings of different lengths can be compared

Quotient Algebra: How?

An ideal I of an algebra A is a sub-vector space of A suchthat xa ∈ I and ax ∈ I for all a ∈ A and all x ∈ ICongruence: x ≡ y if x − y ∈ IQuotient algebra A/I formed from equivalence classes

Data-driven Composition

Use a treebankFor each rule π : X → X1 . . .Xr with head Xh we addvectors

λπ,i = ei − X1 ⊗ . . .⊗ Xh−1 ⊗ ei ⊗ Xh+1 ⊗ . . .⊗ Xr

for each basis element ei of VXh to the generating set.X is the sum over all individual vectors of subtrees rootedwith X

Preserve meaning of head

Example

Example Corpus

see big citymodernise citysee modern citysee red applebuy applevisit big appleread big bookthrow old small red bookbuy large new book

Grammar:

N′ → Adj N′

N′ → N

Generating set Λ:

λi = ei − Adj⊗ ei

Adj = 2eapple +6ebook +2ecity

where ei ranges over basisvectors for noun contexts.

Example: Cosine Similarities

apple 1.0 0.26 0.24 0.52 0.13 0.12 0.33 0.086 0.080big apple 1.0 0.33 0.13 0.52 0.17 0.086 0.33 0.11red apple 1.0 0.12 0.17 0.52 0.080 0.11 0.33city 1.0 0.26 0.24 0.0 0.0 0.0big city 1.0 0.33 0.0 0.0 0.0red city 1.0 0.0 0.0 0.0book 1.0 0.26 0.24big book 1.0 0.33red book 1.0

Finite-dimensional Algebras

Fix dimensionality n of the vector spaceLearn vectors for words and pairs of words using e.g. LSAFind the bilinear product on the vector space which bestfits these vectors

Least squaresLinear optimisation

Finite-dimensional Algebras: Results

Algebras from Monoids

Montague semantics with lambda calculus?Cartesian closed categories?Curry-Howard-Lambek correspondence?Enveloping C∗ algebras?

Algebras from Monoids: the Idea

Given any monoid S, we can construct an algebraPut complex syntactic and semantic information in SThen “vectorize” it using a standard constructionRepresent words as weighted sums of elements of S

From Syntax and Semantics to Monoids

Let S be the set of all pairs (s, σ)

s is a syntactic type (e.g. in Lambek calculus)σ is semantics (e.g. a combination of lambda calculus andfirst order logic)

Multiplication defined by reduction to normal formLambek calculus ∼ residuated latticeLambda calculus is a Cartesian closed category underβη-equivalence (Curry-Howard-Lambek correspondence)

From Syntax and Semantics to Monoids

Let S be the set of all pairs (s, σ)

s is a syntactic type (e.g. in Lambek calculus)σ is semantics (e.g. a combination of lambda calculus andfirst order logic)

Multiplication defined by reduction to normal formLambek calculus ∼ residuated latticeLambda calculus is a Cartesian closed category underβη-equivalence (Curry-Howard-Lambek correspondence)

From Monoids to Algebras

Define multiplication on L1(S) by convolution:

(u · v)(x) =∑

y ,z∈S:yz=x

u(y)v(z)

We want lattice properties of S to carry overC∗ enveloping algebra?Need an involution on SUse right adjoint of Cartesian closed category?

From Monoids to Algebras

Define multiplication on L1(S) by convolution:

(u · v)(x) =∑

y ,z∈S:yz=x

u(y)v(z)

We want lattice properties of S to carry overC∗ enveloping algebra?Need an involution on SUse right adjoint of Cartesian closed category?

Monoid to Algebra Example

Term Context vectorfish (0,0,1)big (1,2,0)

ni = (N, λx nouni(x))

ai = (N/N, λpλy adji(y) ∧ p.y)

Define big = a1 + 2a2 and fish = n3,

Then big · fish = a1n3 + 2a2n3, where

ainj = (N, λx(nounj(x) ∧ adji(x))).

Conclusion

Hypothesis: meanings live in a unital associative realalgebraThree ways to construct such algebras:

Quotient algebras — apply relations between vectorsSearch finite-dimensional algebrasWrap up a monoid in an algebra

Conclusion

Hypothesis: meanings live in a unital associative realalgebraThree ways to construct such algebras:

Quotient algebras — apply relations between vectorsSearch finite-dimensional algebrasWrap up a monoid in an algebra

language as an algebra€¦ · quotient algebras finite-dimensional algebras algebras from monoids...

Documents