applying inductive program synthesis to induction of number series a case study ... · applying...

Applying Inductive Program Synthesis to Induction ofNumber Series – A Case Study with IGOR2

Jacqueline Hofmann1, Emanuel Kitzelmann2, Ute Schmid3

1Universitat Konstanz2Berufskolleg Ratingen3Cognitive Systems

Fakultat Wirtschaftsinformatik und Angewandte Informatik (WIAI)Otto-Friedrich Universitat Bamberg

KI’14U. Schmid (Uni BA) IGOR2-NumberSeries KI’14 1 / 31

Solving Number Series

2 7 15 31 63 ?

U. Schmid (Uni BA) IGOR2-NumberSeries KI’14 2 / 31


3 7 15 31 63 127



3 7 15 31 63 127

How did you generate the solution?

Presupposed: Knowledge about natural numbers and arithmeticoperations

Identify difference between the first two numbers

Generate arithmetic expressions, i.e. hypotheses, to capture how thesecond number can be generated using the firstf (n − 1) + 4, 2 × f (n − 1) + 1, f (n − 1)2 − 2, f (n − 1)2 − n, . . .

Bias: more simple expressions first

Working memory restriction: harder dealing with larger numbers

Check the first hypothesis by applying it to the next position, if itworks, proceed, otherwise, generate and try next hypothesis –constrained by the regularities identified so far



3 7 15 31 63 127

Is this a specific or a general cognitive skill?

Is solving number series problems based on a special purposealgorithm?Or is it based on a general purpose algorithm for detecting,extracting, and generalizing (possibly complex) regularities in streamsof observations?

In I.Q. tests, number series are a sub-test measuring the g-factor(general intelligence)

Is there an algorithm, which can be applied to different domainswhere humans learn by generalizing over detected regularities?e.g. natural language grammar learning, Tower of Hanoi, ancestorrelation, program construction?


Overview

Approaches to solving number series

The inductive programming system IGOR2

Applicability of IGOR2 to number series problems


Number series in I.Q. tests

E.g., included in:I American council of education (ACE) testsI I-S-T-2000 (and I-S-T-70), Amthauer

Measurement of g-factor, which is seen as fundamental for morespecialized abilities (Cattell)

Typically, 20 series in ascending complexity (for I-S-T-2000: mean ofsolved problems is 12, for Abiturienten 15)

Small numbers (to focus on reasoning, not on arithmetic skills)

Psychometrics: Item difficulty assessed as percentage of test personswho solved a problem

Cognitive approaches (in AI and psychology): explain difficulty bycognitive complexity based on a process model

E.g., Holzman, Pellegrino, Glaser (1982, 1983), Journal ofEducational Psychology: Influence of magnitude of starting value andof operator on human performance



(Early work on Thurstone letter series: Simon & Kotovsky, PsyRev,1963)

—– Big time gap —–

Sanghi & Dowe (ICCS’2003)

Burghardt (AI, 2005)

Ragni & Klein (KI’11)

Siebers & Schmid (KI’12)

Strannegard, Amirghasemi, Ulfsbacker (CSR, 2013)

Hernandez-Orallo, J., Martınez-Plumed, F., Schmid, U., Siebers, M. Dowe, D. (Revision

submitted), Computer models solving human intelligence test problems: progress and

implications, Artificial Intelligence.


Approaches to solving number series – Simon & Kotovsky,1963

ababababaaabbbcccddcadaeafarscdstdetuef

Thurstone (1947): 7 factor theory of primary mental abilities

Letter series: reasoning ability (part of some psychometric tests)

Simon & Kotovsky (PsyRev, 1963): procedure for pattern descriptions

Could score better than 10 of 12 humans on 15 problems


Approaches to solving number series – Sanghi & Dowe,2003

960 lines of Perl code, pre-defined patterns

Published at a cognitive science conference but no claim of anunderlying cognitive model

Philosophical perspective wrt relation between Turing test andintelligence


Approaches to solving number series – Burghardt, 2005

E-generalization using grammars

Focus: Lemma-generation in automated proofs

0,1,4,9 vp ∗ vp 0,1,2,1,4,1 if (ev(vp); vp; 1)0,2,4,6 s(s(vp)) 0,0,1,1,0,0,1,1 ev(v2)0,2,4,6 vp + vp 0,1,3,7 s(v1 + v1)1,1,2,3,5 v1 + v2 1,2,2,3,3,3,4,4,4,4 —


Approaches to solving number series – Ragni & Klein, 2011

Artificial neural network

50,000 number series from the Online Encyclopedia of IntegerSequences (OEIS)

Training artificial neural networks (one for each problem) wherelearning rate, number of input nodes, number of hidden nodes andnumber of training iterations are manually varied

Overall results comparable to humans, but very different errordistribution

7,10,9,12,11 f (n − 1) + 3, f (n − 1) − 1 solved3,7,15,31,63 2 ∗ f (n − 1) + 1 not solved6,9,18,21,42 f (n − 1) + 3, f (n − 1) ∗ 2 solved2,5,9,19,37 f (n − 1) ∗ 2 + 1, f (n − 1) ∗ 2 − 1 not solved


Approaches to solving number series – Siebers & Schmid,2012

Predefined basic numerical operators: addition, subtraction, division,multiplication and exponentiation

Construction of patterns guided by an analytical strategy (notgenerate-and-test)

Enumeration and search are strongly restricted by a plausible bias

Tested on 25.000 randomly created number series (using the basicnumerical operators), accuracy: 93.2%


Approaches to solving number series – Strannegard et al.,2013

Repertoire of pre-defined patterns

Working memory restrictions (anthropomorphic cognitive system) fordecoding patterns

arithmetic 2 3 8 11 14 f (n − 1) + 3geometric 3 6 12 24 f (n − 1) × 2Fibonacci 1 2 3 5 8 13 21 34 f (n − 1) + f (n − 2)



Special purpose, symbolic, rule-based approaches: Simon &Kotovsky, Sanghi & Dowe, Siebers & Schmid, Strannegard et al.

General approaches:I Burghardt: E-generalization (based on generation of all congruent

expression based on a pre-defined grammar)I Ragni & Klein ANN (trained separately for each problem)

Cognitively motivated approaches: Simon & Kotovsky, Strannegardet al.

Can the inductive programming system IGOR2 be applied tonumber series problems without any modification of the inductionalgorithm and the bias?


Igor2 is . . .

Inductive

Induces programs from I/O examples

Inspired by Summers’ Thesys system

Successor of Igor1 (Schmid)

Analytical

data-driven

finds recursive generalization by analyzing I/O examples

integrates best first search

Functional

learns functional programs

first prototype in Maude by Emanuel Kitzelmann

re-implemented in Haskell and extended (general fold) by MartinHofmann


Example

reverse

I/O Example

reverse [] = [] reverse [a,b] = [b,a]

reverse [a] = [a] reverse [a,b,c] = [c,b,a]

Generalized Program

reverse [] = []

reverse (x:xs) = last (x:xs) : reverse(init (x:xs))

Automatically induced functions (renamed from f 1, f 2)

last [x] = x init [a] = []

last (x:xs) = last xs init (x:xs) = x:(init xs)


Input

Data-type Definitions

data [a] = [] | a:[a]

Target Function

reverse :: [a]-> [a]

reverse [] = []

reverse [a] = [a]

reverse [a,b] = [b,a]

reverse [a,b,c] = [c,b,a]

Background Knowledge

snoc :: [a] -> a -> [a]

snoc [] x = [x]

snoc [x] y = [x,y]

snoc [x,y] z = [x,y,z]

Input must be the first k I/O examples (wrt to input data type)

Background knowledge is optional


Output

Set of (recursive) equations which cover the examples

reverse Solution

reverse [] = []

reverse (x:xs) = snoc (reverse xs) x

Restriction Bias

Subset of Haskell or Maude

Case distinction by pattern matching

Syntactical restriction: patterns are not allowed to unify

Preference Bias

Minimal number of case distinctions


Basic Idea

Search a rule which explains/covers a (sub-) set of examples

Initial hypothesis is a single rule which is the least generalgeneralization (anti-unification) over all examples

Example Equations

reverse [a] = [a]

reverse [a,b] = [b,a]

Initial Hypothesis

reverse (x:xs) = (y:ys)

Hypothesis contains unbound variables in the body!


Basic Idea cont.

Initial Hypothesis

reverse (x:xs) = (y:ys)

Unbound variables are cue for induction.

Three Induction Operators (to apply simultaneously)

1 Partitioning of examples Sets of equations divided by case distinction

2 Replace right-hand side by program call (recursive or background)

3 Replace sub-terms with unbound variables by to be inducedsub-functions

Kitzelmann & Schmid, JMLR, 7, 2006; Kitzelmann, LOPSTR, 2008; Kitzelmann

doctoral thesis 2010


Some Empirical Results (Hofmann et al. AGI’09)

isor

t

reve

rse

wea

ve

shif

tr

mul

t/ad

d

allo

dds

ADATE 70.0 78.0 80.0 18.81 — 214.87FLIP × — 134.24⊥ 448.55⊥ × ×FFOIL × — 0.4⊥ < 0.1⊥ 8.1⊥ 0.1⊥

GOLEM 0.714 — 0.66⊥ 0.298 — 0.016⊥

IGORII 0.105 0.103 0.200 0.127 ⊙ ⊙MAGH. 0.01 0.08 ⊙ 157.32 — ×

last

s

last

mem

ber

odd/

even

mul

tlas

t

ADATE 822.0 0.2 2.0 — 4.3FLIP × 0.020 17.868 0.130 448.90⊥

FFOIL 0.7⊥ 0.1 0.1⊥ < 0.1⊥ < 0.1GOLEM 1.062 < 0.001 0.033 — < 0.001IGORII 5.695 0.007 0.152 0.019 0.023MAGH. 19.43 0.01 ⊙ — 0.30

— not tested× stack overflow⊙ timeout ⊥ wrongall runtimes in seconds


Application of IGOR2 to Cognitive ProblemsSchmid & Kitzelmann, CSR, 2011

Problem Solving

Clearblock (4 examples,0.036 sec)

Rocket (3 examples, 0.012 sec)

Tower of Hanoi (3 examples, 0.076 sec)

Car Park (4 examples, 0.024 sec)

Blocks-world Tower (9 examples, 1.2 sec)

Recursive Concepts

Ancestor (9 examples, 10.1 sec)

Syntactic Rules

Phrase structure grammar (3 examples, 0.072 sec)S → NP VP

NP → d n

VP → v NP | v S


Learning Tower of HanoiInput to Igor2

eq Hanoi(0, Src, Aux, Dst, S) =

move(0, Src, Dst, S) .

eq Hanoi(s 0, Src, Aux, Dst, S) =

move(0, Aux, Dst,

move(s 0, Src, Dst,

move(0, Src, Aux, S))) .

eq Hanoi(s s 0, Src, Aux, Dst, S) =

move(0, Src, Dst,

move(s 0, Aux, Dst,

move(0, Aux, Src,

move(s s 0, Src, Dst,

move(0, Dst, Aux,

move(s 0, Src, Aux,

move(0, Src, Dst, S))))))) .

Induced Tower of Hanoi Rules (3 examples, 0.076 sec)

Hanoi(0, Src, Aux, Dst, S) = move(0, Src, Dst, S)

Hanoi(s D, Src, Aux, Dst, S) =

Hanoi(D, Aux, Src, Dst,

move(s D, Src, Dst,

Hanoi(D, Src, Dst, Aux, S)))



Crucial: How to represent number series problems as input for IGOR2

(1) Input List – Output Successor Value

eq Plustwo((s 0) nil) = s^3 0

eq Plustwo((s^3 0) (s 0) nil) = s^5 0

eq Plustwo((s^5 0) (s^3 0) (s 0) nil) = s^7 0

(2) Input Position – Output List

eq Plustwo(s 0) = (s 0) nil

eq Plustwo(s^2 0) = (s^3 0)(s 0) nil

eq Plustwo(s^3 0) = (s^5 0)(s^3 0)(s 0) nil

eq Plustwo(s^4 0) = (s^7 0)(s^5 0)(s^3 0)(s 0) nil

(3) Input Position – Output Value

eq Plustwo(s 0) = s 0

eq Plustwo(s s 0) = s s s 0

eq Plustwo(s s s 0) = s s s s s 0

eq Plustwo(s s s s 0) = s s s s s s s 0



Can IGOR2 solve number series problems?

Are finding a solution and solution time dependent on therepresentation?

How does size of numbers influence solution success and solutiontime? (exponential growth of possible representations of differencesbetween numbers)

How do further features of number series affect solution success andtime?

I Operator: +, ×, +/×, =I Number of operators: 1, 2I Size of starting value: small or largeI Size of argument: small or largeI Structure: linear, complexI Back reference: n − 1, n − 2, n − 3, n − 1/n − 2I Argument Type: constant/position

Generation of 100 problems, many of them also to be found in OEIS

Details see bachelor thesis of Jacqueline HofmannU. Schmid (Uni BA) IGOR2-NumberSeries KI’14 26 / 31

Series examples

1 2 3 12 13 14 23 f (n − 3) + 11 +, 1, small, large, linear, n − 3, const1 2 3 5 8 f (n − 1) + f (n − 2) +, 1, small, small, comp, n − 1/n − 2, const6 7 8 18 21 24 54 f (n − 3)× 3 ×, 1, small, small, linear, n − 3, const3 4 12 48 576 f (n − 1)× f (n − 2) ×, 1, small, sm/lrg, comp, n − 1/n − 2, const5 10 30 120 600 f (n − 1)× n ×, 1, large, large, linear, n − 1, pos15 15 16 15 15 16 15 f (n − 3) =, 1, large, large, linear, n − 3, const


Evaluation

IGOR2 could solve all number series

However, there were large run-time differences, e.g.,f (n − 1) + 2 in 17 ms (Rep1)f (n − 2) × 3 + 6 in 336449 ms (Rep2) but in only 14173 ms (Rep1)

Is solution time dependent on the representation?

Representation 1 and 3 perform similar, however, there is much morevariance for representation 1

Representation 2 is significantly slower

Explanation: calculation of an output list is computationally morecomplex than of an output element

mean sd

Rep1 (List - Succ) 35,87 198,49Rep2 (Pos - List) 87,78 217,60Rep3 (Pos - Element) 32,54 74,17


Evaluation

Does size of numbers influence solution time?

Higher size of starting value leads to a significant rise in run-time forall representations

Size of argument value leads to a rise in run-time (but not significant)for all representations

Explanation: Identifying a constant for a starting value is based onexploration of all possible expressions characterizing differencesbetween elements. For large numbers there is a combinatorialexplosion even if the set of operators is restricted (e.g., to additiononly).We assume that humans use a heuristics there, comparing severalelements in the series.


EvaluationDoes the operator influence solution time?

Representation 1 has significantly higher run-times for equality

Explanation: none

Does the back reference influence solution time?

(n − 3) is hardest for all representations

Comment: We expected that already a back-reference to thepredecessor would lead to an increase in run-time


Conclusions

The inductive programming system IGOR2 can be applied “from theshelf” to number series problems!

IGOR2 captures the general cognitive skill to identify regularities andgeneralize over them as already demonstrated for several cognitivedomains

Current restrictions (in the implemented version): negative numbers,alternating series

Currently: Comparison with human performanceEmpirical study with number series systematically varied overdifferent features: operator (+, ×, xn), size of numbers, complexity ofexpression (last element and constant vs. more than one previouselement and index positions)


applying inductive program synthesis to induction of number series a case study ... · applying...

Documents