reverse engineering brain
TRANSCRIPT
-
7/29/2019 Reverse Engineering Brain
1/52
Krasnow Institute for Advanced Studies -- George Mason University
James Albus
Senior FellowKrasnow Institute for Advanced Studies
George Mason University
Founder & Chief (Retired) Intelligent Systems DivisionNational Institute of Standards and Technology
An Engineering Perspectiveon
Reverse Engineering the Brain
-
7/29/2019 Reverse Engineering Brain
2/52
Krasnow Institute for Advanced Studies -- George Mason University
Outline
An Engineering Viewpoint
The Neuroscience
Reverse Engineering the Brain
-
7/29/2019 Reverse Engineering Brain
3/52
Krasnow Institute for Advanced Studies -- George Mason University
65-75 NASA-NBS -- Cerebellum model for learning control (CMAC neural net)73-85 Navy/ NBS Robot control, Automated Manufacturing Research Facility86-87 DARPA -- Multiple Unmanned Undersea Vehicles (MAUV)88-89 DARPA -- Submarine Operational Automation System (SOAS)90-92 GD Electric Boat -- Next generation nuclear submarine control86-88 NASA -- Space Station Flight Telerobotic Servicer (NASREM)
87-89 Bureau of Mines -- Coal mine automation87-91 U.S. Postal Service -- Stamp distribution center, General mail facility86-08 Army -- TEAM, TMAP, MDARS, Picatinny Arsenal UGV, Demo I and III ARLCollaborative Technology Alliance, JAUGS, VTA, FCS-ANS96-97 Navy Double Hull Robot, Multiple UAV SWARM94-95 DARPA / General Motors Enhanced CNC & CMM Control99-01 Boeing Cell Control, Riveting, Hi Speed machine tool92-01 Commercial CNC - plasma & water jet cutting96-98 DARPA MARS, PerceptOR02-04 Boeing/SAIC FCS Autonomous Navigation System, Integrated Combat Demo02-07 AirForce RoboCrane Paint Stripping Robot for Large Aircraft08-09 DOT Intelligent vehicles, Foveal-Peripheral Vision for Driving06-07 DARPA Learning Applied to Ground Robotics (LAGR)
08-10 DARPA EATR Foraging Robot
Intelligent Systems EngineeringIntelligent Control Projects ~ $100M total over 43 years
-
7/29/2019 Reverse Engineering Brain
4/52
Krasnow Institute for Advanced Studies -- George Mason University
HWS workstation
Intelligent Machining Workstation
circa 1981
-
7/29/2019 Reverse Engineering Brain
5/52
Krasnow Institute for Advanced Studies -- George Mason University
CDBWS workstation
Intelligent Cleaning andDeburring Workstation
circa 1982
-
7/29/2019 Reverse Engineering Brain
6/52
Krasnow Institute for Advanced Studies -- George Mason University
Intelligent Coal Mining Machine
circa 1988
-
7/29/2019 Reverse Engineering Brain
7/52Krasnow Institute for Advanced Studies -- George Mason University
MAUV pics
Multiple Autonomous Undersea Vehicles
circa 1989
-
7/29/2019 Reverse Engineering Brain
8/52Krasnow Institute for Advanced Studies -- George Mason University
Intelligent Vehicle Control
circa 1993
-
7/29/2019 Reverse Engineering Brain
9/52Krasnow Institute for Advanced Studies -- George Mason University
NIST Autonomous Mobility Team
-
7/29/2019 Reverse Engineering Brain
10/52Krasnow Institute for Advanced Studies -- George Mason University
4D/RCS Reference Model Architecturefor Unmanned Vehicle Systems
Hierarchical structureof goals and commands
Representation of the worldat many levels
Planning, replanning,and reacting at many levels
Integration of many sensorsstereo CCD & FLIR, LADAR,radar, inertial, acoustic, GPS,internal
Adopted by GDRS for FCS Autonomous Navigation SystemAdopted by TARDEC for Vetronics Technology Integration
-
7/29/2019 Reverse Engineering Brain
11/52Krasnow Institute for Advanced Studies -- George Mason University
4D/RCS Reference Model
O P E R A T O R I N T E R F A C E
SPWM
BG
SP WM BG
SP WM BG
SP WM BG
Points
Lines
Surfaces
SP WM BG SP WM BG
SP WM BG
0.5 second plansSteering,velocity
5 second plansSubtask on object surfaceObstacle-free paths
SP WM BGSP WM BG
SP WM BGSP WM BG SP WM BG
SERVO
PRIMITIVE
SUBSYSTEM
SURROGATE SECTION
SURROGATE PLATOON
SENSORS AND ACTUATORS
Plans for next 2 hours
Plans for next 24 hours
0.05 second plansActuator output
SP WM BGSP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG SP WM BG
Objects of attention
LocomotionCommunication Mission Package
VEHICLE Plans for next 50 secondsTask to be done on objects of attention
Plans for next 10 minutesTasks relative to nearby objects
Section Formation
Platoon Formation
Attention
Battalion Formation SURROGATE BATTALION
6
Intelligent System Architecture
Spinal MotorCenters
MidbrainCerebellum
PrimarySensory-MotorCortex
CorticalSensory-MotorHierarchy
Small groups
SituationsEpisodes
-
7/29/2019 Reverse Engineering Brain
12/52Krasnow Institute for Advanced Studies -- George Mason University
A 4D/RCS Computational Node
4D/RCSnode
FrontalCortex
PosteriorCortex
Mapping to the Brain
-
7/29/2019 Reverse Engineering Brain
13/52Krasnow Institute for Advanced Studies -- George Mason University
What is the Goal?
The Engineering GoalTo build machines that DO what the brain does
A Scientific GoalTo understand HOW the brain does what it does.
A second Scientific Goa lTo understand how the brain LEARNS to do what it does
-
7/29/2019 Reverse Engineering Brain
14/52Krasnow Institute for Advanced Studies -- George Mason University
Front to back:Behavior generation in frontSensory processing in back
Overall Structure of Brain
Side to side:Representation of right egosphere on left sideRepresentation of left egosphere on right side
Top to bottom:Conscious self at topSensors and muscles at bottom
At the center:Emotions, Appetites, & Internal state
-
7/29/2019 Reverse Engineering Brain
15/52Krasnow Institute for Advanced Studies -- George Mason University
What is the brain for?
Fine manipulation, language, and reasoningare recent developments
Early evolution => control of locomotion
Swimming motion & gait generation coordination of actuators
Path planning how to get from A to B
Decision making where to go, when, why, how
Tactical behaviors hunting for food, evading predators, . . .
Strategic behaviors migrating, establishing territory, mating, . . .
E v ol u
t i on
The brain is first and foremost a control system
-
7/29/2019 Reverse Engineering Brain
16/52Krasnow Institute for Advanced Studies -- George Mason University
Gravity sensors establish the horizontal planefor an internal egosphere representation
Body kinematics measured by proprioception
Tactile input
-
7/29/2019 Reverse Engineering Brain
17/52Krasnow Institute for Advanced Studies -- George Mason University
What are the Outputs?
forces and velocities in the limbs and torso
Behavior consisting of:
goal-driven tasks and subtasks on objects in the world
control signals to muscles
Behavior that has many levels of resolution in: feedback error correction feed-forward control
planning and coordination
Behavior consistent with goals that aregenerated in the frontal cortex by processes that use:
a rich internal model of the external world an internal model of body kinematics and dynamics an internal representation of needs and desires
-
7/29/2019 Reverse Engineering Brain
18/52Krasnow Institute for Advanced Studies -- George Mason University
Hierarchical ArchitectureBrain is organized hierarchically
Frontal hierarchy: decision making, goal selection,priority setting, planning and execution of behavior
Posterior hierarchy: attention, segmentation, grouping,computing attributes, classification, establishing relationships
Unitary SELF at topMillions of sensors and actuators at bottom
Complex strategies at topSimple actions at bottom
-
7/29/2019 Reverse Engineering Brain
19/52Krasnow Institute for Advanced Studies -- George Mason University
Cortex and Mind
Joaquin Fuster
The brain is ahierarchical
signal processing
& control system
Cortical Architecture
-
7/29/2019 Reverse Engineering Brain
20/52Krasnow Institute for Advanced Studies -- George Mason University
Cortex and Mind
Joaquin Fuster
Moreneurons
at the top
Hierarchical Architecture
Brainhierarchy
is not apyramid
-
7/29/2019 Reverse Engineering Brain
21/52Krasnow Institute for Advanced Studies -- George Mason University
Computational Mechanisms
Synapse is an electronic gate-- complex biochemistry, site of long-term memory
Neuron is a computational element-- non-linear processes on many inputs, & decide
Neural Cluster is a functional unit
-- arithmetic or logical operations, correlation, convolution-- coordinate transformation-- finite-state automata-- rules, grammar, direct and indirect addressing
-
7/29/2019 Reverse Engineering Brain
22/52Krasnow Institute for Advanced Studies -- George Mason University
Neural Clusters in Spinal Cord
Anteriorhorn
Posteriorhorn
Posteriornucleus
-
7/29/2019 Reverse Engineering Brain
23/52
Krasnow Institute for Advanced Studies -- George Mason University
Input OutputCommand & feedback
AddressAddress
If (Situation)
ActionContentsPointerThen (Consequent)
Anatomical structure
Random access table-look-up computationwith generalization
Marr 1969, Albus 1971
Functional structure
Neural Clusters in Midbrain(e.g. Cerebellum)
-
7/29/2019 Reverse Engineering Brain
24/52
Krasnow Institute for Advanced Studies -- George Mason University
General Functional Model
S(t) P(t + D t) = H(S(t))
memory storage & recall, arithmetic or logical functions,IF/THEN rules, goal-seeking reactive control,
forward & inverse kinematics, direct & indirect addressing
Input vector
or array
Output vector
or array
Feedback forLearning
Neural Cluster
-
7/29/2019 Reverse Engineering Brain
25/52
Krasnow Institute for Advanced Studies -- George Mason University
differential and integral functions, dynamic models, phase-lock loops,time and frequency analysis, recursive estimation, Kalman filtering
S(t) P(t + D t) = H(S(t))
Functional Model + FeedbackFeedback for
Learning
Neural Cluster
-
7/29/2019 Reverse Engineering Brain
26/52
Krasnow Institute for Advanced Studies -- George Mason University
A Neural Finite State Automaton
Markov processes, scripts, plans, behaviors, grammars,Bayesian networks, semantic nets, narratives
S(t) P(t + D t) = H(S(t))
State Next state
Feedback for
Learning
Neural Cluster
-
7/29/2019 Reverse Engineering Brain
27/52
Krasnow Institute for Advanced Studies -- George Mason University
Cortical Structure
Cortex is a 2D sheet 2000 cm 2 area x 3 mm thick
Each region is segmented into arrays of columns
Regions are arranged in hierarchical layers
Cortical sheet is partitioned into functional regions
Each column has capabilities of a fsa + memory
-
7/29/2019 Reverse Engineering Brain
28/52
Krasnow Institute for Advanced Studies -- George Mason University
Circuit diagramof visual system
in brain12 layers32 areas
Each area is an array of
Cortical Columns
Felleman &van Essen 1991
Each area represents theVisual Field of Regard
-
7/29/2019 Reverse Engineering Brain
29/52
Krasnow Institute for Advanced Studies -- George Mason University
Cortical Column Structure
Microcolumns 100 250 neurons30 50 diameter, 3000 long
Posterior: detect patterns, compute attributes
Frontal: evaluate alternatives, recommend actions
Hypercolumns (a.k.a. columns)100+ microcolumns in a bundle
500 in diameter, 3000 long Posterior: segmentation, grouping, classification, relationships Frontal: set goals, make plans, control action
There are about 10 6 hypercolumns in human cortex
-
7/29/2019 Reverse Engineering Brain
30/52
Krasnow Institute for Advanced Studies -- George Mason University
Communication in the BrainAxon is an active fiber connecting one neuron to others
(transmits a scalar variable on a publish-subscribe network with bandwidth ~ 500 Hz)
Two kinds of axons:
Drivers Preserve topology and local sign Data vectors or arrays of attributes and state-variablesi.e., images, objects, events, attributes and state -- e.g., color, shape, size, position, orientation, motion
Modulators Dont preserve topology or local sign Context & broadcast variables, addresses, and pointerse.g., select & modify algorithms, set parameters, define relationships
Exploring the Thalamus
Sherman & Guillery 2006
-
7/29/2019 Reverse Engineering Brain
31/52
Krasnow Institute for Advanced Studies -- George Mason University
EarlyCorticalVision
ProcessesCorticalColumns
in V1
+LateralGeniculate
in Thalamus
Hypercolumn
Modulator Input
Modulator Output toother cortical regions
Input from lgn
Driver Outputto Higher Level & superior colliculus
Modulator OutputBack to lgn
Microcolumn
Receptivefield
DiffuseFibers(Modulators)
PixelAttributes(Drivers)
C i l H l Th l i l
-
7/29/2019 Reverse Engineering Brain
32/52
Krasnow Institute for Advanced Studies -- George Mason University
Cortical Hypercolumn + Thalamic loop
windowing, segmentation, grouping, computing group attributes &state, filtering, classification, setting and breaking relationships
CorticalComputational
Unit
(CCU)
drivers = attribute vectors& arrays
modulators = broadcast variables,& address pointers
Receptive field
-
7/29/2019 Reverse Engineering Brain
33/52
Krasnow Institute for Advanced Studies -- George Mason University
Drivers(data)
Modulators(addresses)
CCUOutputs
CCU Data Structure Hypothesis
-
7/29/2019 Reverse Engineering Brain
34/52
Krasnow Institute for Advanced Studies -- George Mason University
drivers = attribute vector array
Cortico-Thalamic Loop
12
3456
t
modulators = address pointers
windowingsegmentation & groupingcompute group attributes
recursive filteringclassification
corticalhypercolumnthalamic
nucleus
A Cortical
ComputationalUnit(CCU)
-
7/29/2019 Reverse Engineering Brain
35/52
Krasnow Institute for Advanced Studies -- George Mason University
Posterior
Cortico-ThalamicLoop Hierarchy
windowingsegmentation & groupingcompute group attributes
recursive filteringclassification
at each level
-
7/29/2019 Reverse Engineering Brain
36/52
Krasnow Institute for Advanced Studies -- George Mason University
1. Receptive field hierarchies
Two types of hierarchies
Receptive field hierarchies are defined by driveranatomical connectivity and are relatively fixed
2. Entity and Event hierarchies
Entity and event hierarchies are defined bymodulator activity that can establish or break
belongs-to and has-part pointers in ~ 10 ms
-
7/29/2019 Reverse Engineering Brain
37/52
Krasnow Institute for Advanced Studies -- George Mason University
CCU Receptive Field Hierarchy
Defined by driver neurons flowing up the processing hierarchy
-
7/29/2019 Reverse Engineering Brain
38/52
Krasnow Institute for Advanced Studies -- George Mason University
CCUEntity/Event
HierarchyDefined by pointers
result of segmentation& grouping processes
Pointers link pixelsto symbols & vice versa
Provides symbol grounding
Pointers resettop to bottom within
a saccade ~ 150 ms
-
7/29/2019 Reverse Engineering Brain
39/52
Krasnow Institute for Advanced Studies -- George Mason University
Segmentation& Grouping
ProcessEach level detects
patterns within itsreceptive field&
sets or breaks pointers
This produces anEntity/Event
Hierarchy
CCU C di t F
-
7/29/2019 Reverse Engineering Brain
40/52
Krasnow Institute for Advanced Studies -- George Mason University
CCU Coordinate Frames
Each level has multiplecoordinate frames
Oculomotor signals,vestibular inputs, &
range estimates providetransform parameters
Coordinate transformscomputed in parallel
at each level in ~ 10 ms
-
7/29/2019 Reverse Engineering Brain
41/52
Krasnow Institute for Advanced Studies -- George Mason University
World CenteredHierarchy of
Entity PointersCoordinate frame is fixed
in the world
Entities have continuityin space & time
Entities have state position, orientation, velocity
Entities have attributes-- size, shape, color, texture, behavior
Entities have relationships-- class, rank, spatial, temporal, causal
-
7/29/2019 Reverse Engineering Brain
42/52
Krasnow Institute for Advanced Studies -- George Mason University
NAMEattributesstate
pointers
9876543210123456789
belongs-tohas-part1has-part2class1class2
shape
sizecolor behavior
positionmotion
observed states expected states
EntityFrame
Temporal Continuity for an Entity
Each CCU containsan entity frame
Each entity frame
contains:a Trace of O bserved States&a Prediction of E xpected States
-
7/29/2019 Reverse Engineering Brain
43/52
Krasnow Institute for Advanced Studies -- George Mason University
Building computational machines that are
functionally equivalent to the brain in their ability to perceive, think, decide, and act in a purposeful
way to achieve goals in complex, uncertain, dynamic, and possiblyhostile environments, despite unexpected events and unanticipated
obstacles, while guided by internal values and rules of conduct.
Reverse Engineering the Brain
Functional equivalence ::= producing the sameinput/output behavior
What does that mean?
R E i i th B i
-
7/29/2019 Reverse Engineering Brain
44/52
Krasnow Institute for Advanced Studies -- George Mason University
Will require a deep understanding of how thebrain works and what the brain does
How does the brain generate the incredibly complexcolorful, dynamic internal representation that
we consciously perceive as external reality?
Reverse Engineering the Brain
How are signals transformed into into symbols?
How are images processed?How are messages encoded?
How are relationships established and broken?
What are the knowledge data structures?
What are the functional operations?
How is information represented in the brain?How is computation performed?
-
7/29/2019 Reverse Engineering Brain
45/52
Krasnow Institute for Advanced Studies -- George Mason University
Reverse Engineering the BrainCited as a Grand Challenge by
U.S. National Academy of Engineering
Decade of the Mind InitiativeA proposed 10 year $4B program
to understand the mechanisms of mind Krasnow Lead
Steering Committee of Top ScientistsRecent workshops at:
Sandia National Labs Jan 13 15, 2009Berlin Sept 10 12, 2009
Singapore October 18-20, 2010
Focus of DARPA SyNAPSE & NEOVISION2, IARPA ICArUS,& European Union FACETS & POETICON Programs
Wh N ?
-
7/29/2019 Reverse Engineering Brain
46/52
Krasnow Institute for Advanced Studies -- George Mason University
Why Now?
Neurosciences computation and representation in the brain- biochemistry, synaptic transmission, brain imaging, neuron modeling- neuroanatomy, neurophysiology, network & whole brain models
Cognitive Modeling representation and use of knowledge- mathematics, logic, language, learning, problem solving- psychophysics, cognitive psychology, functional brain modeling
Intelligent Control making machines behave appropriately- control theory, cybernetics, AI, knowledge representation, planning- manipulation, locomotion, manufacturing, vehicles, weapons
Computational Power speed and memory that rival the brain- supercomputer = 1015 ops today, laptop > 10 15 ops in 20 years
The science & technology is ready
Progress is rapid, an enormous literature
-
7/29/2019 Reverse Engineering Brain
47/52
Krasnow Institute for Advanced Studies -- George Mason University
Computational Power isApproaching a Critical Threshold
ComputingPower
(ops)
Estimatedcomputing power
of human brain:~10 13-10 16 ops
2005 2010 2015 2020 2025 2030 203510 1010
11
10 1210 1310 14
10 1510 16
Moores LawComputational power
x10 every 5 years
Single chip
MultiCore
Supercomputer
teraflop chip
Road Runner
-
7/29/2019 Reverse Engineering Brain
48/52
Krasnow Institute for Advanced Studies -- George Mason University
Military FCS, UGV, UAV, UUV, USV, UGSCommercial manufacturing, transportationEntertainment video games, cell phonesAcademic neuroscience, computer science
Intelligent Machines Will Be Critical forMilitary Security and Economic Prosperity
Money Is Flowing
Billions of $ will be invested over the next decade
h h h
-
7/29/2019 Reverse Engineering Brain
49/52
Krasnow Institute for Advanced Studies -- George Mason University
overall system level (central nervous system) arrays of macro-computational units (e.g., cortical regions) macro-computational units (e.g., cortical hypercolumns & loops) micro-computational units (e.g., cortical microcolumns & loops) neural clusters (e.g., spinal and midbrain sensory-motor nuclei) neurons (elemental computational units) input/output functions synapses (electronic gates, memory elements) synaptic phenomena membrane mechanics (ion channel activity) molecular phenomena
What is the path to successfor reverse engineering the brain?
Pick the right level of resolution
AI and Cognitive Psychology
Mainstream Neuroscience & Neural Nets
CCUs
-
7/29/2019 Reverse Engineering Brain
50/52
Krasnow Institute for Advanced Studies -- George Mason University
State of art supercomputer running at 10 15 fops
provides 10 million fops per CCU modeling cycle
106 CCUs running at 100 Hz requires108 CCU modeling cycles per second
Estimated communication load between CCUs106 bytes per second for each CCU, or
1012
bps for full brain modelThis appears to be within the state of the art
for current supercomputers
Computational Requirementsfor Engineering Human Brain at CCU level
S & C l i
-
7/29/2019 Reverse Engineering Brain
51/52
Krasnow Institute for Advanced Studies -- George Mason University
Summary & Conclusions
Near term success will require selecting the rightlevel of resolution, e.g. CCU level
The science and engineering can support it
Reverse Engineering the Human Brain
appears feasible in the near term
Real-time modeling at CCU level of resolution
appears achievable now with supercomputersand in ~ 20 years with laptop class computers
The impact will be revolutionary
The benefits will justify the investment
-
7/29/2019 Reverse Engineering Brain
52/52
Thank you
Questions?