have we missed half of what the neocortex does? by jeff hawkins (12/15/2017)
TRANSCRIPT
MIT
December 15, 2017
Jeff [email protected]
Have We Missed Half of What the Neocortex Does?
Allocentric Location as the Basis of Perception
1) Reverse engineer the neocortex- an ambitious but realizable goal- seek biologically accurate theories- test empirically and via simulation
2) Enable technology based on cortical theory- active open source community- basis for Machine Intelligence
L2
L3aL3b
L4
L6a
L6b
L6 ipL6 mpL6 bp
L5 ttL5 ccL5 cc-ns
L2/3
L4
L6
L5
Input
The Cortical Column
1) Cortical columns are complex- Twelve or more excitatory cellular layers- Two parallel FF pathways- Parallel FB pathways (not shown)- Numerous intra- and inter-column connections (not shown)- Inhibitory neurons/circuits are equally complex
2) The function of a cortical column must also be complex.
3) Whatever a column does applies to everything the cortex does.
L5: Calloway et. al, 2015L6: Zhang and Deschenes, 1997
Simple
Output, via thalamus
50%10%
CortexThalamus
Output, direct
L5 CTC: Guillery, 1995Constantinople and Bruno, 2013
A Couple of Thoughts
Output
Observation:
The neocortex is constantly predicting its inputs.
How do networks of neurons, as seen in the neocortex,learn predictive models of the world?
Research:
1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Grid cells in entorhinal cortex solve a similar problem- Big Idea: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Big Idea: Pyramidal neuron model for prediction- A single layer network model for sequence memory- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects
Proximal synapses: Cause somatic spikesDefine classic receptive field of neuron
Distal synapses: Cause dendritic spikesPut the cell into a depolarized, or “predictive” state
Depolarized neurons fire sooner, inhibiting nearby neurons.
A neuron can predict its activity in hundreds of unique contexts.
5K to 30K excitatory synapses- 10% proximal- 90% distal
Distal dendrites are pattern detectors- 8-15 co-active, co-located synapses
generate dendritic spike- sustained depolarization of soma
HTM Neuron Model
Prediction Starts in the Neuron
Pyramidal Neuron
Major, Larkum and Schiller 2013
Properties of Sparse Activations
L2
L3aL3b
L4
L6a
L6b
L6 ipL6 mpL6 bp
L5 ttL5 ccL5 cc-ns
Example: One layer of cells, 5,000 neurons, 2% (100) active
1) Representational capacity is virtually unlimited(5,000 choose 100) = 3x10211
2) Randomly chosen representations have minimal overlap
3) A neuron can robustly recognize an activation pattern by forming 10 to 20 synapses
4) Unions of patterns do not cause errors in recognition
Hypothesis: Cellular layers use unions to represent uncertainty
Hawkins, Ahmad, 2016Ahmad, Hawkins, 2015
Pattern 1 (100 active cells)
Cell robustly recognizes pattern1by forming synapses to small sub-sample of active cells
UnionPatterns 1-10 (1,000 active cells)
Cell still robustly recognizes pattern 1
A Single Layer Network Model for Sequence Memory
- Neurons in a mini-column learn same FF receptive field.- Neurons forms distal connections to nearby cells.
No prediction Predicted input
(Hawkins & Ahmad, 2016)
(Cui et al, 2016)
- High capacity (learns up to 1M transitions)
- Learns high-order sequences: “ABCD” vs “XBCY”
- Makes simultaneous predictions: “BC…” predicts “D” and “Y”
- Extremely robust (tolerant to 40% noise and faults)
- Learning is unsupervised, continuous, and local
- Satisfies many biological constraints
- Multiple open source implementations (some commercial)
t=0
t=1
Predicted cells fire first and inhibit neighbors
Next prediction t=2
t=0
t=1
1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Grid cells in entorhinal cortex solve a similar problem- Hypothesis: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Pyramidal neuron model- A single layer network model for sequence memory- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects
How Could a Layer of Neurons Learn a Predictive Model of Sensorimotor Sequences?
Sequence memory
Sensorimotor sequences
SensorMotor-related context
Hypothesis:By adding motor-related context, a cellular layer can predictits input as the sensor moves.
What is the correct motor-related context?
L2
L3aL3b
L4
L6a
L6b
L6 ipL6 mpL6 bp
L5 ttL5 ccL5 cc-ns
50%
Sensoryfeature
Two Layer Model of Sensorimotor Sequence Memory
Feature @ location
Object Stable over movement of sensor
With allocentric location input, a column can learn models of complete objects by sensing different locations on object over time.
SensorFeature
AllocentricLocation
Pooling
Seq Mem
Changes with each movement
Object
Feature @ LocationLocationon object
Column 1 Column 2 Column 3
Sensorfeature
Sensorimotor Inference With Multiple Columns
Each column has partial knowledge of object.
Long range connections in object layer allow columns to vote.
Inference is much faster with multiple columns.
FeatureFeatureFeatureLocationLocationLocation
Output
Input
Objects Recognized By Integrating Inputs Over Time
FeatureLocationFeatureLocationFeatureLocation
Column 1 Column 2 Column 3
Output
Input
Recognition is Faster with Multiple Columns
Yale-CMU-Berkeley (YCB) Object Benchmark (Calli et al, 2017)
- 80 objects designed for robotics grasping tasks
- Includes high-resolution 3D CAD files
YCB Object Benchmark
We created a virtual hand using the Unity game engineCurvature based sensor on each fingertip4096 neurons per layer per column
98.7% recall accuracy (77/78 uniquely classified)
Convergence time depends on object, sequence of sensations, number of fingers.
Simulation using YCB Object Benchmark
Pairwise confusion between objects after 1 touch
Convergence 1 finger 1 touch
Pairwise confusion between objects after 2 touches
Convergence 1 finger 2 touches
Pairwise confusion between objects after 6 touches
Convergence 1 finger 6 touches
Pairwise confusion between objects after 10
touches
Convergence 1 finger 10 touches
Convergence Time vs. Number of Columns
This is why we can infer complex objects in a single grasp or single visual fixation.
1) How does the cortex learn predictive models of extrinsic sequences?
2) How does the cortex learn predictive models of sensorimotor sequences?
Current research: How do columns compute allocentric location?
- Hypothesis: cortical columns contain analogs of grid cells and head direction cells- Starting to understand the function of numerous layers and connections
“Why Neurons Have Thousands of Synapses, a Theory of Sequence Memory in the Neocortex”Hawkins and Ahmad, Frontiers in Neural Circuits, 2016/03/30
- Pyramidal neuron model- A single layer network model for sequence memory- Properties of sparse activations
“A Theory of How Columns in the Neocortex Learn the Structure of the World”Hawkins, Ahmad, and Cui, Frontiers in Neural Circuits, 2017/10/25
- Extension of sequence memory model- Big Idea: Columns compute “allocentric” location of input- By moving sensor, columns learn models of complete objects
Entorhinal Cortexenvironments
A
B C
X
Y Z
R
S T
Room 3
Room 2Room 1
Location- Encoded by Grid Cells- Unique to location in room AND room- Location is updated by movement
Orientation (of head to room)- Encoded by Head Direction Cells- Anchored to room- Orientation is updated by movement
Location- Unique to location on object AND object- Location is updated by movement
Orientation (of sensor patch to object)- Anchored to object- Orientation is updated by movement
Cortical Columnobjects
Hypothesis:Cortical columns contain analogs of grid cells and head direction cells
A
C
B
X
Y
Z
Stensola, Solstad, Frøland, Moser, Moser: 2012
Location and Orientation are both necessary to learn the structure of rooms and predict sensory input.
Location and Orientation are both necessary to learn the structure of objects and predict sensory input.
L3
L4
L6a
L6b
L5a
L5b
Mapping Orientation and Location to a Cortical Column (most complex slide)
Sensation
Orientation
1) A column is a two-stage sensorimotor model for learning and inferring structure.
2) A column usually cannot infer a Feature or Object in one sensation.- Integrate over time (sense, move, sense, move, sense..)- Vote with neighboring columns
3) This system is most obvious for touch, but it applies to vision and other sensory modalities.
Because this architecture exists throughout the neocortex, it suggests we learn, infer,and manipulate abstract concepts the same way we manipulate objects in the world.
Location
Sensation @ Orientation
Feature
Feature @ Location
Object
Motor updated (HD cell-like)
Motor updated (grid cell-like)
Seq mem
Pooling
Seq mem
Pooling
Meaning Operation
Rethinking Hierarchy
Every column learns complete models of objects. They operate in parallel.
Inputs project to multiple levels at once. Columns operate at different scales of input.
Sense
Simple features
Complex features
Objects
Classic
Objects
Objects
Objects
Sensor array
Proposed
Region 3
Region 2
Region 1
Rethinking Hierarchy
Every column learns complete models of objects. They operate in parallel.
Inputs project to multiple levels at once. Columns operate at different scales of input.
Non-hierarchical connections allow columns to vote on shared elements such as “object” and “feature”.
Sense
Simple features
Complex features
Objects
Classic
Sensor array
Objects
Objects
Objects
Sensor array
vision touch
Proposed
Region 3
Region 2
Region 1
Summary
Goal: Understand the function and operation of the laminar circuits in the neocortex.
Method: Study how cortical columns make predictions of their inputs.
Proposals
1) Pyramidal neurons are the substrate of prediction.Each neuron predicts its activity in hundreds of contexts.
2) A single layer of neurons forms a predictive memory of high-order sequences.(sparse activations, mini-columns, fast inhibition, and lateral connections)
3) A two-layer network forms a predictive memory of sensorimotor sequences.(add motor-derived context and a pooling layer)
4) Columns need motor-derived representations of location and orientation, of the
sensor relative to the object. These are analogous to grid and head direction cells.
5) A framework for the cortical column.- Columns learn complete models of objects as “features at locations”, using twosensorimotor inference stages.
6) The neocortex contains thousands of parallel models, that resolve uncertainty byassociative linking and/or movement of the sensors.
Open Issues
Behaviors: how are they learned, encoded, and applied to objects?
Detailed model of hierarchy including thalamus
How can the model be applied to “Where” pathways, and how do “What” and “Where” pathways work together
Collaborations
There are many testable predictions in this model, a “green field”. We welcome collaborations and discussions.
We are always interested in hosting visiting scholars and interns.
Numenta Team
Subutai AhmadVP Research
Marcus Lewis
Thank You