Introducing Psychometric AI

Selmer Bringsjord & Bettina Schimanski & …?Department of Cognitive Science

Department of Computer Science

RPI

Troy NY 12180

As exploration of this avenue proceeds.

Roots of this R&D…

Seeking to Impact a # of Fields• This work weaves together

relevant parts of:– Artificial Intelligence: Build machine

agents to “crack” and create tests.– Psychology: Use experimental

methods to uncover nature of human reasoning used to solve test items.

– Philosophy: Address fundamental “big” questions, e.g., What is intelligence? Would a machine able to excel on certain tests be brilliant?…

– Education: Discover the nature of tests used to make decisions about how students are taught what, when.

– Linguistics: Reduce reasoning in natural language to computation.

Many applications!

The Primacy of Psychology of ReasoningThere is consensus among the relevant luminaries in AI and theorem provingand psychology of reasoning and cognitive modeling that: machinereasoning stands to the best of human reasoning as a rodent stands to thelikes of Kurt Godel. In the summer before Herb Simon died, in apresentation at CMU, he essentially acknowledged this fact -- and set outto change the situation by building a machine reasoner with the power offirst-rate human reasoners (e.g., professional logicians). Unfortunately,Simon passed away. Now, the only way to fight toward his dream (which ofcourse many others before him expressed) is to affirm the primacy ofpsychology of reasoning. Otherwise we will end up building systems thatare anemic. The fact is that first-rate human reasoners use techniquesthat haven't found their way into machine systems. E.g., humans useextremely complicated, temporally extended mental images and associatedemotions to reason. No machine, no theorem prover, no cognitivearchitecture, uses such a thing. The situation is different than chess --radically so. In chess, we knew that brute force could eventually beathumans. In reasoning, brute force shows no signs of exceeding humanreasoning. Therefore, unlike the case of chess, in reasoning we are goingto have to stay with the attempt to understand and replicate in machineterms what the best human reasoners do. We submit that a machine able toprove that the key in an LR/RC problem is the key, and that the otheroptions are incorrect, is an excellent point to aim for, perhapsthe best that there is. As a starting place, we can turn to simpler tests.

“Chess isTooEasy”

Multi-Agent Reasoning, modeled inMental Metalogic, is the keyto reaching Simon’s Dream!Pilot experiment shows that groupsof reasoners instantly surmountthe errors known to plague individualreasoners!Come next Wed 12n SA3205

What is Psychometric AI?

An Answer to: What is AI?• Assume the ‘A’ part isn’t the problem: we know

what an artifact is.• Psychometric AI offers a simple answer:

– Some artificial agent is intelligent if and only if it excels at all established, validated tests of intelligence.

• Don’t confuse this with: “Some human is intelligent…”

• Psychologists don’t agree on what human intelligence is.– Two notorious conferences. See The g Factor.

• But we can agree that one great success story of psychology is testing, and prediction on the basis of it. (The Big Test)

Some of the tests…

Intelligence Tests: Narrow vs. BroadSpearman’sview of intelligence

Thurstone’s view ofintelligence

Let’s look @ RPM(Sample 1)

RPM Sample 2

RPM Sample 3

Artificial Agent to Crack RPM

---------------- PROOF ----------------1 [] a33!=a31.3 [] -R3(x)| -T(x)|x=y| -R3(y)| -T(y).16 [] R3(a31).24 [] T(a31).30 [] R3(a33).31 [] T(a33).122 [hyper,31,3,16,24,30,flip.1] a33=a31.124 [binary,122.1,1.1] $F.------------ end of proof -------------

----------- times (seconds) -----------user CPU time 0.62 (0 hr, 0 min, 0 sec)

Artificial Agent to Crack RPM

---------------- PROOF ----------------1 [] a33!=a31.7 [] -R3(x)| -StripedBar(x)|x=y| -R3(y)| -StripedBar(y).16 [] R3(a31).25 [] StripedBar(a31).30 [] R3(a33).32 [] StripedBar(a33).128 [hyper,32,7,16,25,30,flip.1] a33=a31.130 [binary,128.1,1.1] $F.------------ end of proof -------------

----------- times (seconds) -----------user CPU time 0.17 (0 hr, 0 min, 0 sec)

Artificial Agent to Crack RPM

=========== start of search ===========given clause #1: (wt=2) 10 [] R1(a11).given clause #2: (wt=2) 11 [] R1(a12).given clause #3: (wt=2) 12 [] R1(a13)....given clause #4: (wt=2) 13 [] R2(a21).given clause #278: (wt=16) 287 [para_into,64.3.1,3.3.1] R2(x)| -R3(a23)| -EmptyBar(y)| -R3(x)| -EmptyBar(x)| -T(a23)| -R3(y)| -T(y).given clause #279: (wt=16) 288 [para_into,65.3.1,8.3.1] R2(x)| -R3(a23)| -StripedBar(y)| -R3(x)| -StripedBar(x)| -EmptyBar(a23)| -R3(y)|-EmptyBar(y).Search stopped by max_seconds option.============ end of search ============

Correct!

Possible Objection

“If one were offered a machine purported to be intelligent, what would be an appropriate method of evaluating this claim? The most obvious approach might be to give the machine an IQ test … However, [good performance on tasks seen in IQ tests would not] be completely satisfactory because the machine would have to be specially prepared for any specific task that it was asked to perform. The task could not be described to the machine in a normal conversation (verbal or written) if the specific nature of the task was not already programmed into the machine. Such considerations led many people to believe that the ability to communicate freely using some form of natural language is an essential attribute of an intelligent entity.” (Fischler & Firschein 1990, p. 12)

WAISA Broad Intelligence Test…

Cube Assembly

Basic Setup

Problem: Solution:

Harder Cube Assembly

Basic Setup

Problem: Solution:

Picture Completion

Currently untouchable AI -- but we shall see.

And ETS’ tests…

“Blind Babies”

A. For babies the survival advantage of smiling consists in bonding the care-giver to the infant.

B. Babies do not smile when no one is present.

C. The smiling response depends on an inborn trait determining a certain pattern of development.

D. Smiling between people basically signals a mutual lack of aggressive intent.

E. When a baby begins smiling, its care-givers begin responding to it as they would to a person in conversation.

Children born blind or deaf and blind begin social smiling onroughly the same schedule as most children, by about three monthsof age.

The information above provides evidence to support which of the following hypotheses:

correct

“Blind Babies” in Prop. Calc.

1 SSBB SS-SCHBBNB

2 SSBB (1; elim)

3 SSL SSI

4 (SSBB SSL) SEE-SOMEONEBB

5 SEEBB

5b SEEBB SEE-SOMEONEBB

6 SEE-SOMEONEBB (5, 5b; elim)

7 6b (SSBB SSL) (6, 4 modus tollens)

8 6c SSBB SSL (6b, demorgan’s)

7 SSL (6c, 2; disjunctive syllogism)

8 SSI (3, 7 disj. Syll.)

Pilot protocol analysisexperiment indicates thathigh-performers representthese items at the level ofthe propositional calculus.But that level not detailed enough for generating theItems. VPA experiment planned for this semester.

The Now Time-Honored “Lobster”Lobsters usually develop one smaller, cutter claw and one larger,crusher claw. To show that exercise determines which claw becomesthe crusher, researchers placed young lobsters in tanks and repeatedlyprompted them to grab a probe with one claw – in each case alwaysthe same, randomly selected claw. In most of the lobsters the grabbingclaw became the crusher. But in a second, similar experiment, whenlobsters were prompted to use both claws equally for grabbing, mostmatured with two cutter claws, even though each claw was exercisedas much as the grabbing claws had been in the first experiment.

Which of the following is best supported by the information above?

A Young lobsters usually exercise one claw more than the other.B Most lobsters raised in captivity will not develop a crusher clawC Exercise is not a determining factor in the development of crusher claws in lobsters.D Cutter claws are more effective for grabbing than are crusher claws.E Young lobsters that do not exercise either claw will nevertheless usually develop one crusher and one cutter claw.

SamplePartof

D(LRE)

sentences 2 & 3 in text not needed for proof of correct option (A)

But they are needed for proof that option C is inconsistent with text!!

A. For babies the survival advantage of smiling consists in bonding the care-giver to the infant.

B. Babies do not smile when no one is present.

C. The smiling response depends on an inborn trait determining a certain pattern of development.

D. Smiling between people basically signals a mutual lack of aggressive intent.

E. When a baby begins smiling, its care-givers begin responding to it as they would to a person in conversation.

Whereas in “Blind Babies”the foils all involve predicatespresumably outside of R(LRE)e.g.,

Same Approach Used

---------------- PROOF ----------------1 [] -Lobster(x)|Cutter(r(x)).3 [] -Lobster(x)| -Exercise(r(x))| -Exercise(l(x))|Cutter(l(x)).4 [] -Lobster(x)| -Cutter(r(x))| -Cutter(l(x)).5 [] Lobster($c1).6 [] Exercise(r($c1)).7 [] Exercise(l($c1)).9 [hyper,5,1] Cutter(r($c1)).10 [hyper,7,3,5,6] Cutter(l($c1)).11 [hyper,10,4,5,9] $F.------------ end of proof -------------

----------- times (seconds) -----------user CPU time 0.38 (0 hr, 0 min, 0 sec)

Therefore option AIs correct!

Underlying Math

…

Additional Objections…

Psychometric AIin Context …

A Classic “Cognitive System” Setup Under Development

Cognitive System

Test Item

Choice of correctoption, and rulingout of others, and…“percept”

“action”

actions that involve physical manipulation of objects and locomotion.

Fits forthcomingSuperminds

book by Bringsjord & Zenzen…

• “Weak” AI based on testing going back to Turing is implied for the practice of AI.

Fits “Complete” CogSci…

Cognitive System

Environm

ent

Perception

Action

Perception and Action

Low-levelHigh-level

subdeclarative com

putation

Cognitive System

Environm

ent

Perception

Action

Cognitive Modeling

Short Term Memory

Long Term Memory

Perception& Action

Low-levelHigh-level

subdeclarative com

putation

AC

T-

R

Cognitive System

Environm

ent

Perception

Action

Reasoning

Short Term Memory

Long Term Memory

Perception& Action

Low-levelHigh-level

subdeclarative com

putation

AC

T-

R

Mental

Metalogic

SyntacticReasoning

SemanticReasoning

Cognitive System

Environm

ent

Perception

Action

Cognitive Human Factors:Engineering the Interface b/t Cognitive Systems and their Environments

Short Term Memory

Long Term Memory

Perception& Action

Low-levelHigh-level

subdeclarative com

putation

AC

T-

R

Mental

Metalogic

SyntacticReasoning

SemanticReasoning

Should we consider IGERT?

The distinctive graduate education provided by RPI’s Department ofCognitive Science could be that we provide a truly integrated CogSci education:We produce students able to deal with cognitive systems top-to-bottom.A number of particular applications anchor this distinctive pedagogicalapproach, viz., Psychometric AI, Synthetic Characters, Cognitive Prostheses, etc.. These are applications which, by their very nature, call for top-to-bottom CogSci.

Large Variation in Difficulty

Evan’sANALOGY

Program