BUILDING A COGNITIVE SYSTEM BY GNOSYS

Co-ordinator: John Taylor (KCL) Asst Co-Ordinator: Stathis Kasderidis (FORTH)

EC PO: George StorkStart date: Oct 1; Kick-off Oct 20/21

gnosys@ics.forth.grWeb-site: http://www.ics.forth.gr/gnosys/

Department of Mathematics King’s College London, UK

emails: john.g.taylor@kcl.ac.uk

CONTENTS

1. Vision of GNOSYS

2. GNOSYS Partners

3. GNOSYS Prototypes

4. GNOSYS Tasks/Milestones

5. GNOSYS Summary

1. VISION OF GNOSYS

1) Embodied cognition (wheel-robot + gripper)2) Create concepts/rewarded-goals under

attention control3) Learns goal-directed tasks4) Learns novel environments5) Reasoning by forward models6) Guidance from brain (animal/infant/adult)7) Various memory types

(STM/LTM/associative/error-based)8) Interdisciplinary: Comp vision/ Cog NSci/

Neural Networks/ Robotics/ AI/ Maths

General Work Plan of GNOSYS

GNOSYS Cognitive Powers

• Feature-based perception (M1-16) WP2• Concepts/Goals/Attention (Sensory & Motor)

(M6-18, 12-24) WP2/WP3• Rewarded drive-based learning (M12-24) WP2• Goal-based Global Computation (M6-18) WP2• Abstraction Hierarchy (M12-24) WP3• Reasoning/Action Planning by motor

attention-base forward models(M18-33) WP3• Robot Platforms @ 2 levels (M18/M30)

HOW GNOSYS WORKS

v v

│ ↔ │

│ ↔ │

│ ↔ │

│ ↔ │

│ ↔ │

ANN Adaptive Stream(Concepts/Goals/Attention/Rewards/Values/ForwardModelslearnt as NNpredictors)

Symbolic Control Threads(5 components)

Linguistic Connections(Words/Fuzzy rules/ Symbolisation)

: Relate to COSPAR

Drives/Motivation/Rewards

• Assign values (in AMYG/OBFC) as direct input (learnt), or by ‘DA’ modulation from primary rewards (satisfying basic drives)

• Basic drives for GNOSYS:

Energy level/ Curiosity/ Stimulation/ Minimum pain (touch/pressure)/ Approbation/ Motor activity

• Use value maps --> assign value to stimuli

2. GNOSYS PARTNERS1 King’s College London (KCL) Comp Nsci Grp:

NNs, concepts, attn control 2 ZENON S.A., Greece (ZENON): robots3 Foundation of Research & Technology - Hellas

Greece (FORTH): global comput/robots4 Eberhard-Karls-Universität, Tübingen,

Germany (UTUB): perception/reward/robots5 Università di Genova, Dipartimento di

Informatica, Sistemistica, Telematica, Italy (UGDIST): motor control/robots

-> RobotCub

Attentional Agent Architect (EC FP5: DC, 2001-2003)

• Distributed entity with four layers (attentional multi-level agent):– L1: Sensors

– L2: Pre-processing

– L3: Local decision

– L4: Global decision

GLOBAL CONTROL ARCHITECTURE• EXTENDED ATTENTION V EMOTION

ARCHITECTURE (EC ERMIS, NF, 2002-4; BBSRC: 2004-7):

(extended Corbetta & Shulman, 2002)

InhibitoryInteraction through ACG:

Excitatoryinteraction

Excitatory

Excitatory/Inhibitory

Inhibition from DLPFCIn emotion recognition

Endogenousgoals

Exogenous goals

MOTOR CORTEX ACTION NETWORK (NT, MH, OM & JGT) (in NetSim for sequence learning; tested in PDs: J NSci24:702 )

MOTOR CORTEX

GLOBUS PALLIDUSEXTERNAL

GLOBUS PALLIDUSINTERNAL

NUCLEUS RETICULARISTHALMUS

SUBSTANTIA NIGRAPARS COMPACTA

CENTROMEDIANPARAFISCULAR NUCLEUS

SUBSTANTIA NIGRAPARS RETICULARIS

SUB-THALAMIC NUCLEUS

GLUTAMATERGIC INPUT

GABAERGIC INPUT

DOPAMINERGIC INPUT

SIMILAR STRUCTURES MODEL OBFC, DLPFC, ACG AND VLPFC

FROM OTHER CORTEX + OTHER

THALAMUS

FROM CEREBELLUM

TO OTHER CORTEX

THALAMUS

STRIATUM

Cerebellar Structure& Associated Regions: For Insertions,by error-based learning (with teacher)

BK

PONS

GrC

GoC

PK

IO

DCN+DCN-

GrC granule cellsGoC golgi cellsBK basket cellsPK purkinje cells

DCN deep cerebellar nuclei (excit. & inhib.)

IO inferior olivePONS pontine nucleiHIPP hippocampal regionsPFC pre-frontal cortex

inhibitory conns.excitatory conns.

HIPP

PFC

HIPPOCAMPUS & AMYGDALA (in NetSim for sequence learning, and x20 speed-up in SWS) (MH, NT & JGT): as teacher

EPSRC: Ventral & Dorsal Concept Learning (-> GNOSYS)

Ventral pathway

V1

V2

V4TEO

TE

LGN Input

Dorsal pathway

V1

V5

LIP

LGN Input

Learning

Hard-wired

Currently Hard-wired

Architecture Details: Percepts• V1: 4 excitatory & inhibitory layers for bar

orientations, hardwired (14*14)

• V2 (28*28) trained on reduced set of pairs of bars (6), # start positions in retina 121

• V4 (28*28)->TEO (28*28/14*14)->TE (7*7) trained on 2 different triangles (121 start positions)

• Now by cluster computing• Next step: to DL/VLPFC as goals-> attention

ERMIS/BBSRC: GLOBAL BRAIN CONTROL by ATTENTION:

Fusiform Gyrus

PFC

ACG/TPJ

PL

PL

VCX

PL-> Simulated Attentional Blink NF/JGT-> Consciousness by CODAM (Prog Neurobiology 03)

Model of Visuo-Motor Attention Control System

(JGT + NF, IJCNN’03)

-> MACS for Attention filtering

->MINDRACESfor anticipation

AB extended by AMYG as bias: ERPs for T2 in Lag3 when no

amygdala

ERPs for T2 in Lag3: amygdala input from T2’s

object rep, & fed back to same site

Tsuji T, Tanaka Y, Morasso P, Sanguineti V. Kaneko M (2002) IEEE Trans SMC-C, 32, 426-439.Morasso P, Sanguineti V, Spada G (1997) Neurocomputing, 15, 411-434

UGDIST: Biomimetic trajectory formation via artificial potential fields

… the importance of smoothness and continuity …

Khepera: the artificial bodyThe in-vitro brain

From the Neurobit project

Real-time control of robot motion by sub-symbolic neural activity

… the importance of bidirectional communication …

… the importance of softness and a soft touch …

Robotized haptic interface

Computational Vision and Robotics Lab (CVRL)

Institute of Computer Science

Foundation for Research andTechnology – Hellas

(FORTH)

CVRL - FORTH

• Mission: Study the mechanisms involved in the development of autonomous robotic systems

Cognition Action

LearningPerception

SystemArchitecture

Right sub-network

Left sub-network

S.O. n-1

A

B

M

A

B

C

M

C

Right sub-network

Left sub-network

A

B

M

A

B

C

M

C

Right sub-network

Left sub-network

A

B

M

A

B

C

M

C

BS-L

BS-R

S.O. n S.O. n+1

• Current R&D activities  – perceptual competences based on visual and range sensors and sensor

fusion techniques– coupling of perception and action – autonomous navigation and control of complex robotic systems– development of networked robotic systems– content-based retrieval of images and video

• Future activities– development of robotic behaviours that simulate corresponding behaviours

of living organisms– emergence of cognition in artificial systems– complex heterogeneous robotic systems involving multiple robots

CVRL - FORTH

UTUB Experienced in robot movement

and planning Involved in GNOSYS perceptions &

rewards

ZENONRobotics Company in Athens

Experienced in robot applicationsTo construct robot platforms (2)

3. GNOSYS PROTOTYPES

• PROTOTYPE I (M18): Attn control of sensory inputs & response

• Learn concepts of simple shapes [3] & rewarded actions, under attention

• Responses to commands/learn new goals as new actions on new objects

• PROTOTYPE 2 (M28): As above but more complex objects [3] + sequences of action/object pairs in real scenes + forward models for virtual goal seeking (reasoning)

4. GNOSYS TASKS, etc: Reasoning Domains/Environments

(WP2&3)• Three levels of environment• Level 1: Learn shapes/colours; move &

touch; move & pick up; [2] & [3]-D objects• Powers: Concept/Attn/Goals as actions on

objects/Valence of objects in environment• Level 2: Complex objects & actions• Powers: ibid/manipulate to achieve goals• Level 3: Hierarchy of objects; run virtual

object/action sequences to achieve goals• Powers: Reasoning/ novel objects/actions

Application to Patrolling, etc

• Construct loc/action and object/action map in patrol environment

• Reasoning tasks: to discover actions: (loc1, action)→loc2, (obj1,action)→obj2

• Meets barrier of boxes. Reasoning: move box to pass through, instead of moving round barrier

• Over pond: reasoning: find plank to put across pond

• Plus many psychological tasks (WCST/Tower of London, etc, etc)

MILESTONES

• Level 1: Simple actions & stimuli [2] (M6)

• Level 2: More complex actions & stimuli [3]/colour/motion/audition/touch (M16)

• Level3: Real-world stimuli (M24)

• Prototype 1 (M18)

• Prototype 2 (M28)

• Assessment (M34)

5. GNOSYS SUMMARY

• Create concepts/goals by learning

• Can handle novel environments

• Embodied cognitive system

• Learning by infant-style development (by hierarchy of modules sequentially coming on line)

• Reasoning by forward models created by reward-based learning