towards logics for neural conceptors

Conceptors Japanese vowels A fuzzy logic for conceptors Conclusions

Towards logics for neural conceptors

Till Mossakowskijoint work with Razvan Diaconescu and Martin Glauer

Otto-von-Guericke-Universität Magdeburg

AITP 2018, Aussois, March 30, 2018


Overview

1 Conceptors

2 Conceptors at work: Japanese Vowels Pattern Recognition

3 A fuzzy logic for conceptors

4 Conclusions


Motivation

Conceptors [Jaeger14]Combination of neural networks and logicUsing a distributed representation like in deep learning andhuman brain

most neural-symbolic integration use localist represtatione.g. logic tensor networks (AITP17): one network for eachpredicate

Boolean operatorsprovide concept hierarchynew samples can be added without re-training

Our contributionConceptors obey the laws of fuzzy setsFuzzy logic is the natural logic for conceptors


Reservoir dynamics [Jaeger14]

reservoir = randomly created recurrent neural networkinput signal p drives this networkfor timesteps n = 0,1,2, . . .L,

x(n+1) = tanh(W x(n)+W inp(n+1)+b)

W : N×N matrix of reservoir-internal connection weightsW in: N×1 vector of input connection weightsb: biasp: input signal (pattern)

W , W in and b are randomly created


Conceptors [Jaeger14]

collect state vectors x0, . . .xL into N×L matrix X = cloud ofpoints in the N-dimensional reservoir state spacereservoir state correlation matrix: R = XX T/Lconceptor: normalised ellipsoid (inside the unit sphere)representing the cloud of points

C = R(R+α−2I)−1 ∈ [0,1]N×N

α: aperture (scaling parameter)We here use a simplified version where C = diag(c1 . . .cn). Theci are called conception weights.


Boolean operations on simplified conceptors

(¬c)i := 1−ci

(c∧b)i :=

{cibi/(ci +bi −cibi), if not ci = bi = 00, if ci = bi = 0

(c∨b)i :=

{(ci +bi −2cibi)/(1−cibi), if not ci = bi = 11, if ci = bi = 1

Aperture adaption

ϕ(c,γ)i := ci/(ci + γ−2(1−ci)) for 0 < γ < ∞

ϕ(c,0)i :=

{0, if ci < 11, if ci = 1

ϕ(c,∞)i :=

{1, if ci > 00, if ci = 0


Overview

1 Conceptors



4 Conclusions


"Japanese Vowels" Pattern Recognition[Jaeger14]

Data: 12-channel recordings of short utterance of 9 maleJapanese speakers270 training recordings, 370 test recordingsTask: train speaker recognizer on training data, test on testdata


Conceptors at work: Japanse vowels[Jaeger14]

for each speaker j , build a conceptor Cj

for a test pattern p, compute the reservoir response signal rpositive classification for speaker j : use 1

L(rT Cj r)

negative classification for speaker j , using Booleanconceptor logic

1L(rT¬(C1∨·· ·∨Cj−1∨Cj+1∨·· ·∨Cn)r)


Result of Japanese vowel classification[Jaeger14]

with 10-neuron reservoirs, mean (50 trials with freshreservoirs) test errors for 370 tests:8.4 (positive ev.) / 5.9 (neg-neg ev.) / 3.4 (combined)incremental model extension possible, again enabled byBoolean logic


Overview

1 Conceptors



4 Conclusions


Conceptors are fuzzy

Central thesis:Conceptors and conception vectors behave like fuzzysets, and their logic should be a fuzzy logic.

PropositionConceptors form a (generalised) de Morgan triplet, i.e. a t-norm,a t-conorm and a negation that interact usefully.


Laws for a deMorgan triplet

¬0 = 1,¬1 = 0x < y implies ¬x > ¬y (strict anti-monotonicity)¬¬x = x (involution)T1: x ∧1 = x (identity)T2: x ∧y = y ∧x (commutativity)T3: x ∧ (y ∧z) = (x ∧y)∧z (associativity)T4: If x ≤ u and y ≤ v then x ∧y ≤ u∧v (monotonicity)S1: x ∨0 = x (identity)T2: x ∨y = y ∨x (commutativity)T3: x ∨ (y ∨z) = (x ∨y)∨z (associativity)T4: If x ≤ u and y ≤ v then x ∨y ≤ u∨v (monotonicity)x ∨y = ¬(¬x ∧¬y) (de Morgan)


Further algebraic laws

∧ and ∨ do not form a lattice, so De Morgan algebras,residuated lattices, BL-agebras, MV-algebras,MTL-algebras etc. do not apply

[Jaeger14] lists:C∨C = ϕ(C,

√2)

C∧C = ϕ(C,√

12)

A≤ B iff ∃C.A∨C = BA≤ B iff ∃C.A = B∧C


Implication

In a De Morgan triplet, we can define implication as

x → y = ¬x ∨y

Alternative: residual implication

R(x ,y) = sup{t | x ∧ t ≤ y}

But: has the unpleasant property that

R(c,0)i ={

0, if ci > 01, if ci = 0

while our implication behaves more smoothly: (c→ 0)i = 1−ci


Fuzzy conceptor logic

parameterised over dimension N ∈ Ntwo sorts: individuals and conception vectorsSignatures: constants for individuals and for conceptionvectorsModels: interpret constants as N-dimensional vectors in[0,1]N

individuals are interpretated as feature vectors, conceptorterms as conception vectors

Conceptor terms:

C ::= c | x | 0 | 1 | ¬x | C1∨C2 | C1∧C2 | ϕ(C, r)

Atomic formulas:1 ordering relations between conceptor terms2 memberships of individual constants in conception vectors


Fuzzy conceptor logic: semantics

A formula yields a fuzzy truth value in [0,1]:

[[C1 ≤ C2]] = minj=1...N([[C1]]j → [[C2]]j)

[[i ∈ C]] = 1N [[i ]]T diag([[C]])[[i ]]

Complex formulas like in FOL:

F ::= i ∈C |C1≤C2 | ¬F |F1∨F2 |F1∧F2 |∀x i .F |∀xc.F |∃x i .F |∃xc.F

x i : variable ranging over individualsxc: variable ranging over conception vectors.Interpretation of formulas like in fuzzy FOL:

infimum for universal quantificationsupremum for existential quantification


Fuzzy conceptor logic: subset relationsConsider C ≤ D versus ∀x i .(x i ∈ C→ x i ∈ D):


Fuzzy conceptor logic in action

Suppose we have two sets of speakers, call them Dialect1 andDialect2. Using disjunction, we can build conceptors C1 and C2for these sets. Then we can ask:

how far is Dialect1 similar to Dialect2? (C1 ≤ C2∧C2 ≤ C1)how much is Dialect1 a sub-dialect of Dialect2? (C1 ≤ C2)

If we have an ontology of dialects, we cantest the ontology by checking how far it follows from speakerdatainfer new consequences by (fuzzy/crisp) reasoning in the(fuzzy/crisp) ontology


Overview

1 Conceptors



4 Conclusions


Conclusions

Defined a new fuzzy logic for conceptorsIn his conceptor report, Jaeger only defines two crisp logics

Can be basis for neural-symbolic integrationCrisp and fuzzy reasoning about ontologies of conceptsLearning and classification using conceptors


Future work

Fuzzy conceptor logicsuitable algebraisationproof calculusautomated theorem proving

Work out details of integrated reasoning

towards logics for neural conceptors

Documents