Computational and Theoretical Neuroscience
Krishnamoorthy V. Iyer
Department of Electrical Engineering, IIT Bombay
August 22, 2011
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 1 / 169
Outline
1 IntroductionBrain as a Computer(Human) Brain: Neuroscience Basics 101Outline of Lectures 1 and 2Computational and Theoretical Neuroscience
2 Computer Vision and (Visual) Computational NeuroscienceComputer Vision and Visual Neuroscience BasicsPrimary Visual Cortex and Low-Level Psychophysical Visual PhenomenaOrganization of Part IIBasics of Retina and V1 Physiology and ArchitectureVisual Field Representation in V1 - the Complex Log MapComputation in Primary Visual Cortex
How V1 may do stereopsis
3 Single Neuron ComputationSingle Neuron BasicsSingle neuron as a ”computer“
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 2 / 169
Introduction
The (Human) Brain
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 3 / 169
Introduction
Remember this is a presentation, NOT a paper Keep the numberof references to a minimum Not more than ten references overe 2talks, average of 5 per talk Sejnowski and Churchland, Schwartz,Schwartz and Yeshurun, David Marr, Laurence Abbott and PeterDayan, William Bialek Spikes, one or two papers of Bialek andco-workers Maybe one or two websites This also reduces theamount of work
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 4 / 169
Introduction Brain as a Computer
Theme of this lecture: Brain as a “Computer”
Question: Is the brain a “computer“ in the sense that a Pentium or aMacintosh is? Obviously not!
Calling the brain a ”computer“ is a metaphor ....
Like describing an electron as ”both” a “particle“ and a “wave“ - themetaphor refers to the use of the mathematical techniques ofwave equations as well as properties associated with particles, such asposition, for instance
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 5 / 169
Introduction Brain as a Computer
What the brain as a ”computer“ metaphor refers to
The brain as an information processor,
How neural architecture and dynamics represents information (neuralcode) and processes information (neural computation)
Mathematical description i.e. building mathematical models ofneural architecture, dynamics, development, and neuralrepresentations and computation
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 6 / 169
Introduction Brain as a Computer
What is a computational explanation of a physical event?
Refers to the information content of the physical signals
How the information is used to accomplish a task
Not all physical events have information content: Theirdescription in terms of causation and the laws of physics suffice togive an ”understanding“ of the phenomenon
Some do: When we type numbers into a calculator and receive ananswer
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 7 / 169
Introduction Brain as a Computer
What is a computational explanation of a physical event?
Refers to the information content of the physical signals
How the information is used to accomplish a task
Not all physical events have information content: Theirdescription in terms of causation and the laws of physics suffice togive an ”understanding“ of the phenomenon
Some do: When we type numbers into a calculator and receive ananswer
These require an explanation that describes the computation, and notmerely one at the level of dynamics
Better ... an explanation that maps between the two levels [?],
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 7 / 169
Introduction (Human) Brain: Neuroscience Basics 101
The Human Brain: Major Regions
Before studying the brain as a ”computer“, we need to know somethingabout the brain
Much of the brain is divided into two hemispheres, a left and a right
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 8 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Connecting link is a bundle of nerves: corpus callosum
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 9 / 169
Introduction (Human) Brain: Neuroscience Basics 101
The (Human) Brain: Major Regions ContdCortex and Thalamus
(Cerebral) Cortex:
In popular parlance, the cerebrum “is” the brain.Cortex is Latin for “bark” or “outer rind”Seat of all higher mental processes:
Sensory Information ProcessingMovementLanguage (Speech and Comprehension)Reason and Empathy
Hugely developed in mammals.
Thalamus: Cortex’s
Mini-Me: 1-1 correspondence with cortical regionsGateway: 90 percent of information to cortex (except smell) routedthrough thalamus
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 10 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Major Regions ContdHippocampus
Hippocampus: (Episodic) Memory: Where were you and whatwere you doing when the World Trade Center was destroyed?
Long-Term Memory: Autobiographical details
Not crucial for short-term memory
Patient HM
Patient Clive Wearing
“Fifty First Dates” starring Drew Barrymore
“Ghajini” starring Aamir Khan
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 11 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Major Regions ContdHippocampus
Hippocampus: (Episodic) Memory: Where were you and whatwere you doing when the World Trade Center was destroyed?
Long-Term Memory: Autobiographical details
Not crucial for short-term memory
Patient HM
Patient Clive Wearing
“Fifty First Dates” starring Drew Barrymore
“Ghajini” starring Aamir Khan
Does not deal with muscle memory.For ex: Riding a bicyle, dancing, playing an instrument
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 11 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 12 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Other sub-cortical structures
1 Hypothalamus: Basic biological needs/drives:2 Basal Ganglia: Motivation aka “The Will”
affected in Parkinson’s disease
3 Cerebellum: Fine motor control: Playing the violin (but learninghow to play involves the motor cortex)
4 Amygdala: Experience of fear: Famous case of a woman patientwho literally did not experience fear as a consequence of damage tothe amygdala (Damasio)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 13 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Focus of this talk: (Primary Visual) Cortex
Historically, neuroscientists have considered two kinds of divisions ofcortical regions
1 Anatomical:
2 Functional:
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 14 / 169
Introduction (Human) Brain: Neuroscience Basics 101
CortexAnatomical Divisions
Note the colors are purely for visual appeal!
The cortical surface is a dull grey (hence, “grey matter”)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 15 / 169
Introduction (Human) Brain: Neuroscience Basics 101
CortexAnatomical Divisions
Anatomical divisions include:
1 Occipital: (almost exclusively) Vision
2 Temporal: Vision, (also Hearing i.e. Audition)
3 Parietal: Body Sense (somatosensory) and multisensory modalityintegration (including Vision)
4 Frontal: Reason and Empathy (Psychopaths are suspected to haveproblems with neural circuitry in this region).
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 16 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Functional Divisions
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 17 / 169
Introduction (Human) Brain: Neuroscience Basics 101
CortexFunctional Divisions
Functional divisions include:
1 Visual Cortex: Vision - more than 50% of the cortex in humans andprimates devoted exclusively to vision
2 Auditory Cortex: Hearing3 Somatosensory: Body sense and touch
Phantom Limb phenomenon
4 Motor Cortex: (Learning to) Dance, (Learning to) Play anInstrument
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 18 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Cortex and Thalamus Relation
Mini-Me
Gate-Keeper
area marked in red is the thalamus
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 19 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Recurring Theme
Do anatomical divisions correspond to functional divisions?
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 20 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Brain as a “Computer”Some Misconceptions
The architecture of the brain has very little in common with thevon Neumann architecture
The circuitry consists of multiple (and overlapping) spatial scales oforganization
Likewise, the dynamics consists of multiple (and overlapping)temporal scales of activity
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 21 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Brain as a “Computer”Some Misconceptions Contd
No CPU, and hence no homunculus
The frontal cortex may be said to come closest to the notion of a CPUNo “homunculus”:
Hence the naive theories of visual perception as a TV in the brain mustbe discardedRepresentation of information in the brain becomes a fundamentalproblemThe left hemifield of vision projects to the right cerebral cortex, andvice-versa the right hemifield of vision
The hippocampus comes closest to notion of a hard drive
So what does neural architecture look like?
What about neural dynamics?
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 22 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Levels of Organizationcorrespond closely to Spatial Scales
1 (Large) Molecules: Ion Channels (Proteins) 10−4 micrometer
2 Synapses: Scale: 1 micrometer
3 Single Neuron: Scale: 100 micrometers (Neuron is the primary braincell involved in signaling)
4 Networks (aka Microcircuitry) Scale: 1 millimeter
5 Maps: Scale: 1cm Histologically well-defined areas within a brainstructure: Ex: Striate Cortex
6 Systems: Scale: Major anatomical regions: Ex: Cortex, Hippocampus
7 Central Nervous System (CNS) 1 meter8 Focus on levels 3, 4 and 5:
Map level ((Primary) Visual) Cortex, level 5Single Neuron, level 3Networks of neurons, level 4
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 23 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Other kinds of spatial organization
1 Laminar i.e. layered structure:
All regions of cortex, except the region devoted to smell, have 6 layers.There is cross-connectivity b/w the layers as well as within the layers
2 Topographic organization and Self-similarity in the map fromretina to cortex: concatenated complex log map
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 24 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Other kinds of spatial organization
1 Laminar i.e. layered structure:
All regions of cortex, except the region devoted to smell, have 6 layers.There is cross-connectivity b/w the layers as well as within the layers
2 Topographic organization and Self-similarity in the map fromretina to cortex: concatenated complex log map
3 Columnar organization
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 24 / 169
Introduction (Human) Brain: Neuroscience Basics 101
Temporal scales
Events of interest have time scales ranging from:
1 10−3 seconds Ex: generation of an action potential
2 10 years Ex: developmental changes associated with the life of theorganism
3 and many time scales in-between
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 25 / 169
Introduction Outline of Lectures 1 and 2
Outline of this lecture
1 Part I: What is computational and theoretical neuroscience?
How computer scientists can contribute to computational andtheoretical neuroscience?
2 Part II: (Primary) Visual Cortex: Hero: Eric SchwartzVisual Cortex is the part of the brain devoted to visual informationprocessing Attention: CS475, CS675, CS663
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 26 / 169
Introduction Outline of Lectures 1 and 2
Outline of the next lecture
1 Part III: Single Neuron: Star: William Bialek
A principle in much of Bialek’s work is that neural systems (andbiological systems more generally) may have reached optimumperformance limits under the constraints of evolutionary history, andnoise and energy consumption limits Attention: CS435 and CS709
2 Part IV: Networks of Neurons: Laurence Abbott, Nancy Kopell
3 Part V: Other neuroscientists whose work I would encourage you toexplore
4 Part VI: Conclusion and Discussion: CS students interested inNeuroscience
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 27 / 169
Introduction Computational and Theoretical Neuroscience
Computational and Theoretical neuroscience: What it isNOT
1 “Branch” of Biology, but overlaps with it2 “Branch” of Neurology: Study of diseases of the brain: properly a
branch of medicine.
Sridevi Vedula Sarma, PhD, EECS, MIT, LIDS, now at Johns HopkinsNandini Chatterjee-Singh, PhD, Physics, University of Pune, now atNBRC, Gurgaon
3 “Branch” of Neurosurgery But has an important supporting role toplay
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 28 / 169
Introduction Computational and Theoretical Neuroscience
Computational and Theoretical neuroscience: What it isNOT Continued
1 Artificial Neural Networks Some examples: McCulloch-Pittsneuron, Rosenblatt’s Perceptron, Hopfield networks, Boltzmannmachines, backpropagation
2 Artificial Intelligence Goals somewhat similar: How to performcomputational tasks such as object recognition following a”rules-based” approach
3 But in CTNS the emphasis in is on figuring out how the braindoes it
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 29 / 169
Introduction Computational and Theoretical Neuroscience
Potential issue with the goals of computationalneuroscience
1 Maybe trying to figure out how the brain does a task is like trying tofigure out how birds fly w/o knowing the principles of aerodynamics?
2 Solution: C and T neuroscientists often maintain close connectionswith these fields
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 30 / 169
Introduction Computational and Theoretical Neuroscience
Computational and Theoretical Neuroscience: What it is
The brain as an information processor, taking into account
biological imperatives and history: purpose, development, evolutionaryhistoryphysical constraints: energy consumption, noise in sensors and networkelements, processing speed requirements
Interesting possibility: Evolutionary pressures may have caused neuralsystems to have achieved some kinds of optimizations. (WilliamBialek). (At least local, if not global, optima in the landscape ofevolutionary possibilities - my take).
As calculus is integral to Physics, and algorithms to ComputerScience, so the mathematical techniques of computational andtheoretical neuroscience are an integral part of the subject.
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 31 / 169
Introduction Computational and Theoretical Neuroscience
Scientists contributing to Computational and TheoreticalNeuroscience include
1 Physicists: Both theorists and experimentalists2 Engineers:
Electrical Engineers, especially signal processing, communication,control, computing, and VLSI, fields of EE that deal with informationprocessingNote that EE types in other areas and Chemical Engineers andMechanical Engineers also can play a role, with some retraining
3 Computer Scientists:
4 Mathematicians: Mathematicians interested in studying the brainare almost by definition applied mathematicians
5 A small but growing number of scientists from traditionallynon-mathematical scientific disciplines such as biology,biochemistry, psychology, and medicine who are willing and able tostudy mathematical techniques from Physics, Engineering, and CS
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 32 / 169
Introduction Computational and Theoretical Neuroscience
How CS types can contribute to computational andtheoretical neuroscience
Computer and Machine Vision meets Biological Vision: (ArtificialIntelligence meets (Visual) Neuroscience)
Ex: Eric Schwartz’s group
What are the algorithms the brain uses to do visual processing?
Machine Learning: Analysis of large data sets. Curse ofdimensionality
Theoretical Computer Science: Computational Learning Theory,Computational Complexity, Vapnik-Chervonenkis (VC) Dimension
Attention: CS435, CS475, CS675, CS663, CS709
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 33 / 169
Introduction Computational and Theoretical Neuroscience
Brief note about the term “Computational Neuroscience”
Eric Schwartz introduced the term ”ComputationalNeuroscience“
Eric Schwartz describes his areas of interest as ComputationalNeuroscience and Computer Vision
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 34 / 169
Introduction Computational and Theoretical Neuroscience
Brain as a “Computer”: Two themesRepresentation of information and Transformation of Information aka Computation
1 Theme I: Representation of Information:1 Map level: Mathematical techniques used: Complex analysis and more
specifically, numerical conformal mapping and computationalgeometry
2 Single neuron level: Mathematical techniques used: Information theory,machine learning, and possibly theory of computation
2 Theme II: Transformation of Information: Computation1 Single Neuron Level: Theory of computation,
Statistical/Computational Learning Theory2 Network Level: same as above, also graph theory and the theory of
network flows
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 35 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
(Computer/Machine) VisionBackground for Part II
The study of computer/machine vision has traditionally been divided into:
“Low-level“ Vision: Estimating the scene from the image
Image: Raw pixel values
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 36 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
(Computer/Machine) VisionBackground for Part II
The study of computer/machine vision has traditionally been divided into:
“Low-level“ Vision: Estimating the scene from the image
Image: Raw pixel valuesScene: Interpreting the image to obtain some (comparatively basic)information Example: edge detection
”High-level“ Vision:
Object recognition and classification
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 36 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
(Computer/Machine) VisionBackground for Part II
The study of computer/machine vision has traditionally been divided into:
“Low-level“ Vision: Estimating the scene from the image
Image: Raw pixel valuesScene: Interpreting the image to obtain some (comparatively basic)information Example: edge detection
”High-level“ Vision:
Object recognition and classificationVisual cognition and volition: extraction of shape properties andspatial relations while performing tasks such as manipulating objectsand planning movements
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 36 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
Lecture Theme: How the Brain does VisionVisual Cortex and Visual Field Representation
Thalamus (marked LGN) plays gatekeeper’s role
Information to the cortex (marked occipital poles) is routed viathalamus (LGN)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 37 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
Visual Field Representation
Each visual hemifield projects onto nasal (’nose’) hemiretina of thesame side eye and to the temporal (’temples’) hemiretina on theopposite side
Left visual hemifield projects onto
nasal hemiretina of the left eyetemporal hemiretina of the right eye
Take home message: Each visual hemifield is represented byneurons on opposite hemisphere visual cortex
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 38 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
Cortical Hemisphere Input
Each cortical hemisphere gets input from:
one hemifieldboth eyes
Important to the functional (computational) role of ocular dominancecolumns (see below)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 39 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
Visual Field Representation
We will come back to the important issue of representation of (visual)information at a later stage
Prerequisites:
Receptive FieldsRetinotopyCortical magnification
But before we study these matters, we try to obtain a broad overviewof vision as performed by the brain
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 40 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
Vision and Visual Cortical Processing
”Low-level“ vision - ”Early“ stages of visual cortex
”High-level“ vision - ”Later“ stages of visual cortex
Distinction not sharp (large number of ”top-down“ and”bottom-up“ connections in the brain).
”top-down“ refers to connections from ”higher“ to ”lower“ corticalareas of from cortex to subcortical structures. Ex: V2 to V1, cortex tothalamus”bottom-up” refers to connections from “lower“ to ”higher“ areas Ex:thalamus to V1, V1 to V2
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 41 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
Lecture Theme: How the Brain does VisionVisual Cortex and Visual Information Processing
Low-level Vision: Primary Visual Cortex
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 42 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
Lecture Theme: How the Brain does VisionVisual Cortex and Visual Information Processing
Low-level Vision: Primary Visual Cortex
Edge detection: What is an edge in an image?Contour detectionIllusory contours
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 42 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
Lecture Theme: How the Brain does VisionVisual Cortex and Visual Information Processing
Low-level Vision: Primary Visual Cortex
Edge detection: What is an edge in an image?Contour detectionIllusory contoursMotion detection/estimation
High-level Vision:
Object recognition and classification: Inferotemporal (IT) Cortex(”what“ aka ”perception“ stream of vision)Spatial perception, navigation and attention: (Posterior) ParietalCortex (”where“ or ”how“ aka ”action“ stream of vision)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 42 / 169
Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics
”What“ and ”Where“or ”How” Streamsaka “Perception” and “Action” streams
Purple: “What” stream, processed in inferotemporal (IT) cortex.Object recognition
Green: “Where” or “How” stream, processed in posterior parietalcortex. Spatial Perception and navigation
The classification is controversial and has been called into question
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 43 / 169
Computer Vision and (Visual) Computational NeurosciencePrimary Visual Cortex and Low-Level Psychophysical Visual
Phenomena
Primary Visual Cortex (V1) and Low-Level Vision
Image Scene
edge detection pixel values edgecontour detection pixel values contourillusory contours pixel values illusory contourshape from shading pixel val-
ues(luminanceinformation)
shape
figure-ground segmentation pixel values assigning contour to one of twoabutting regions
stereopsis one imageframe fromeach of tworetinae
est. rel. dist. to objects basedon lateral displacements b/wsuperimposed images
motion estimation two succ.image frames
extract motion info
super-resolution low-freq.image
estimated is a high-freq. image
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 44 / 169
Computer Vision and (Visual) Computational NeurosciencePrimary Visual Cortex and Low-Level Psychophysical Visual
Phenomena
Task VisualCortex Area
Cell Type
edge detection V1 Simplecontour detection V1 Complexillusory contours V2shape from shading pixel values
(luminanceinformation)
shape
figure-ground segmentation pixel values assigning contour to one of twoabutting regions
stereopsis V1 Monocular and Binocular /Hypercolumn
motion estimation V1 Complex
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 45 / 169
Computer Vision and (Visual) Computational NeurosciencePrimary Visual Cortex and Low-Level Psychophysical Visual
Phenomena
Kanizsa TriangleIllusory Contours
Believed to take place due to neurons in V2
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 46 / 169
Computer Vision and (Visual) Computational NeurosciencePrimary Visual Cortex and Low-Level Psychophysical Visual
Phenomena
Task Visual CortexArea
Structures
stereopsis V1 Monocular+Binocular Cells /Hypercolumn
Image representation V1 complex log map
Evidence from psychophysics, imaging studies and neurophysiology
Theoretical and modeling studies indicate that these processes take place inthe primary visual cortex aka V1 and V2
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 47 / 169
Computer Vision and (Visual) Computational Neuroscience Organization of Part II
Organization of Part IIArchitecture of V1
Neuronal Tuning and Receptive Fields
Orientation Columns
Ocular Dominance Columns
Hypercolumn
Ice-Cube Model
”Pinwheels“
Topographic map (Complex log map) from retina to V1
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 48 / 169
Computer Vision and (Visual) Computational Neuroscience Organization of Part II
Organization of Part II Contd
Visual information representation:
Retinotopy and Topographic OrganizationComplex Log Mapping between retina and visual cortexApplication to Machine Vision: Space-Variant Vision
Psychophysics: How does the brain perceive depth?
StereopsisPanum’s AreaHypercolumn’s role in Stereopsis
Psychophysics: How does the brain perceive edges?
What is an edge in an image?Different kinds of edges in an imageEdge-detection algorithmsHow the brain processes edges
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 49 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Neuronal Tuning I
Neuronal tuning curve: Graph of the average firing rate of theneuron as a function of stimulus parameters relevant to that classof neurons
A typical tuning curve (top) looks like a half-wave rectified cosinefunction
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 50 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Neuronal Tuning II
Interpretation: Stimuli that cause the neuron to fire at the highestpossible rate are considered to be most important
If neurons encode information in their average firing rate, then this islikely to be true
However, this is open to question (as we will see in the next lecture)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 51 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Receptive Fields (RFs) in Visual Processing
Visual receptive fields refer to the region of (visual) space in which anappropriate stimulus will cause the neuron to respond
Appropriate varies b/w regions: from retina to V1 to V2 to V4 and IT
We consider RFs in retina and V1
In ”higher“ visual cortical regions,
RF sizes increaseappropriate stimuli become more complexIn IT cortex, ∃ cells that respond when a person recognizes faceCalled face-recognition cells
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 52 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Receptive Fields in Retina
RFs in retina come in two types
On−SurroundOff−Center On−Center
Off−Surround
Retinal Cells Receptive Fields
The spatial structure of retinal (ganglion) cell RFs is well captured bya difference-of-Gaussians model
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 53 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Receptive Fields in Retina Contd
Why two different kinds? Won’t a single kind suffice?
The answer lies in energy efficiency
The brain is an extremely energy-hungry organ
brains typically weigh about 2% of body weightedconsume about 20% total body oxygenutlise about 25% body glucose
To signal swings in both directions about zero contrast, thespontaneous firing rate must be high.
This would cause energy consumption to rise
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 54 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Receptive Fields in V1Orientation Selectivity
Hubel and Wiesel discovered nature of RFs in V1
1981 Nobel Prize in Physiology or Medicine
Retinal cells respond to light or dark ”spots”
Individual V1 Cells are tuned to “edges” of a particular orientation.
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 55 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Simple Cells RFs
0 deg
90 deg
180 deg
360 degSimple cell RF orientations othersnot shown
0, 90, 180, 360 deg
Only cells with orientation preferences of 0, 90, 180, 360 degrees havebeen shown for the sake of clarity
Cells with intermediate orientation preferences have not been shown
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 56 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Simple Cells Contd
Every individual cell has an orientation preference (neuronaltuning)
Population as a whole has cells whose preference ranges from 0 to360 degrees
distinct excitatory and inhibitory regions
linear summation (superposition) property
regions are anatgonistic - if the light is diffuse, then the cell does notrespond
Different simple cells respond to different orientations
Receptive field sizes are much smaller than Complex Cells (see below)
May act as edge detectors at different scales
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 57 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Complex Cells
Similar to simple cells: also have orientation preferences
Complex cells have larger receptive fields, so some spatialinvariance
May act as movement detectors
Also complex cells receive inputs from many simple cells, may act todetect contours in an image, enabling figure-groundsegmentation at early stages of visual processing
I encourage you to explore the work of:
Stephen Grossberg and collaboratorsZhaoping Li
Hypercomplex Cells
Using Gabor type filters to model the RF of simple cells by
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 58 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Take Home Message: (Visual) Neuroscience inspiringComputer Vision
∃ cells in V1 that are tuned to distinct orientations
The population as a whole has cells responsive to every orientationfrom 0 to 360 degree
For each orientation, ∃ cells with different RF sizes
These may detect edges of the same orienation but at different scales
This inspired Jones and Malik multi-orientation multi-scale(MOMS) filters used to do stereo (more on this later)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 59 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Other computational theories that have been proposed forthe role of simple and complex cells
(Visual) Texture analysis: Process by which visual system definesregions that differ in the statistical properties of spatial structure
Matt or gloss finishWavy or straight or curly hair
Structure from Shading: how information about the curvature ofsurfaces can be extracted from changes in luminance due to depthstructure in the image
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 60 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Orientation Columns
Cells tuned to a particular orientation are arranged to form ofcolumns, called orientation columns
Cells tuned to nearby orientations are arranged next to each other
∃ cells responsive to all orientations
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 61 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Ocular Dominance ColumnsRole in Stereopsis
Some cells in V1 respond mainly to i/p from left eye,
Others to i/p from the right eye
A few driven by both: so-called binocular cells
These cells did not seem to be tuned for binocular disparityMany of these studies use somewhat ad hoc methods of classification,so terms such as monocular and binocular must be treated with caution
Monocular cells arranged to form columns called ocular dominancecolumns
Known to play a role in binocular vision and stereopsis.
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 62 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Relationship b/w Ocular Dominance and OrientationColumnsHypercolumn
Ocular dominance and orientation columns run almost orthogonal toeach other
Two adjacent ocular dominance columns (each containing cellsresponsive to all orientations) form a hypercolumn
Hypercolumn def as “a unit containing full set of values for any givenset of RF parameters”.
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 63 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
HypercolumnTheme: Relationship b/w Anatomical and Functional Modules
In human V1, a hypercolumn is about 2mm, forms a basic anatomicalmodule
Yeshurun and Schwartz show that it corresponds to a functional i.e.psychophysically measurable module
a biologically plausible algorithm instantiated on a hypercolumn thatsolves stereopsisconsistent with psychophysical data
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 64 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Ice-Cube Model of V1
L
R
ICE−CUBE MODEL OF V1
1 hypercolumn
repeated many times
L and R are the ocular dominance columns due to the left and righteyes resp.
the orientation columns are seen to run perpendicular to the oculardominance columns
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 65 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Short digression: Pinwheels in V1
Applying topological reasoning, Schwartz and co-workers showed thatthe ice-cube model must be modified
∃ singularities in the map of orientations: orientation vortices aka“pinwheels”
Introduce a figure shoring pinwheels of orientation selectivity
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 66 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Lecture 2 begins here
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 67 / 169
Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture
Story so far ...
Ice-Cube model
Qualitatively, each cortical hemisphere receives input from
the opposite hemifield onlyboth eyes
Next Step: Quantifying and mathematizing this description of visualfield representation in the visual cortex
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 68 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Visual field representation in V1Retinotopy and topographic organization
Retinotopy: Spatial organization of neural responses to visual stimuli
Retinotopic maps: Special case of topographic organization
Topographic organization: Projection of a sensory surface ontostructures in the central nervous system (CNS).Examples:
Somatosensory Map: Body surface (skin) to the somatosensory cortexRetinotopic Map: Retina to V1
Neighboring points in the sensor (skin or eye) activate (usually)neighboring regions in the (somatosensory or visual) cortex
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 69 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Digression: Topographic organization of touch:Somatosensory map
Cortical hemisphere surfaceBody regional surfaces mapped as aboveNote the large amount of space devoted to hands, face, lips, andtongueSkilled pianist/violinist - increased region to fingersSkilled dancer - increased region to legsPhantom limb and/or Neural plasticity
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 70 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
non-Uniform sampling and non-uniform representation
Intuitively obvious: Lips (as all of us are presumably aware) and fingertips (as Braille readers know) are more sensitive surfaces than (say)the back.
Demonstrated by experimental studies.
But why?
Reason: More sensors per unit area of the fingers than the back
Consequently, non-Uniform:
Sampling at the sensor surfaceRepresentation in the cortex
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 71 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Eric Schwartz
Eric Schwartz introduced the term ”Computational Neuroscience“
Eric Schwartz and co-workers’ mathematical models of corticalneuroanatomy one of the most successful quantitative models inNeuroscience
His ouevre demonstrates an intense preoccupation withstructure-function (computation) relationships in Neuroscience
In other words, ’’how the brain does a task“
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 72 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Topographic organization in visionComplex log map from retina to V1
Retinotopic map: Mathematical transformation is a complexlogarithmic map
Mathematical description of retinotopic map was discovered by EricL. Schwartz
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 73 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Complex log map from retina to V1 Part II
w = log(z + a), where
w represents position on cortical surfacez is a complex variable representing azimuthal angle, θ andeccentricity, ǫ on retinaz = ǫe iθ
Fε
F fixation point
θ
a is an experimentally obtained parameter
measure of foveal sizediffers from species to species
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 74 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Cortical magnification factor
(Magnitude of) derivative of w w.r.t z called (cortical)magnification factor
|dwdz
| = | 1z+a
|
Mathematically:
Centre (fovea): z << a, so | dwdz| ≈ constant, (map approx linear)
Periphery: z >> a, so δwδz
≈ 1ǫ, (map approx logarithmic)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 75 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Physical meaning and biological significance of the corticalmagnification factor
Physical meaning:
As eccentricity ǫ increases, magnification decreasesFoveal magnification (and thus sampling) greater than peripheryConsequently, foveal representation (much) greater than peripheryrepresentation in cortex
Biological significance:No. of neurons ’responsible’ for processing a stimulus of a given size,as a function of location in visual field decreases in the peripheralregions
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 76 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Complex log mapLimitations and Extensions
Excellent fit for foveal region across many species
Not a good fit for peripheral retina
More sophisticated models (numerical conformal mapping, dipolemap, wedge-dipole map), have been developed by Schwartz andco-workers for
peripheral retinal regionsother visual cortical regions (V2, V3)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 77 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Visual field representation in cortex’: Practical implicationsTheme: Neuroscience’s impact on Computer Vision and Robotics I
non-Uniform sampling (Somatosensory map also shows non-uniformsampling and representation)
Small portion of visual field sampled at high resolutionCoarser sampling at periphery
non-Uniform representation
Consequence: Far more processing resources i.e. neurons in cortex aredevoted to objects at the center of gaze
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 78 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Theme: Neuroscience’s impact on Computer Vision andRobotics II
Eric Schwartz pointed out that a correspondence could be made b/w:Cortical RF size variation Multiple Resolution
Surface of cortex ”Horizontal“Single Point in visual field ”Vertical“
∃ cells in hypercolumns that respond to the same RF location, but havedifferent RF sizes
These have inspired ”multiscale” or ”pyramid“ approaches to computervision
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 79 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Theme: Vision Neuroscience’s impact on Computer Visionand Robotics
Summary:
non-Uniform sampling and representationand multi-scale resolution aka ”pyramids“ have
inspired space-variant computer vision architectures, savings incomputational requirements, and thence to economy
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 80 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Information representation and Information Transformationaka Computation
Complex log map: Information representation (at the level of maps)
Now: Information transformation aka computation in the visual cortex
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 81 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Short digression: Nature versus Nurture debate
Nature versus nurture debate:
Respective roles played by the environment versus innate developmentrules (genetics and the womb/egg)
Many animals, especially reptiles, are born with astonishing survivalskills
Newborn mammals are largely helpless and usually spend theirchildhood learning
Humans have a very long period of childhood even compared withother mammals
Learning period typically lasts until mid-adolescence at least
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 82 / 169
Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map
Nature versus Nurture IIRepresentational Learning
Retinotopic mapping: Information representation in V1
Question: Are these representations
learned (due to nurture i.e. environmentally determined) orinnate (due to nature i.e. developmentally determined)?
Topographic mappings ”largely“ ”innate“.
Learning does play a role here, ∵ if one eye is damaged around birth,the OcuDom columns due to that eye will not form
For learning about (non-innate) neural representations, a goodstarting point is the classic textbook:Title: ”Theoretical Neuroscience‘‘Subtitle: Computational and Mathematical Modeling of NeuralSystemsAuthors: Peter Dayan and Laurence F. Abbott
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 83 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
CS213 in your Brain!!!
Attention: CS213 Data Structures and Algorithms
CS students: Symbiotic relationship b/w data structures andalgorithms
Good DSs enable elegant algorithmic solutions to a desiredcomputation
Historically significant example follows
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 84 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
A historically significant Data Structure
Arithmetic w/ Hindu vs. Roman numerals
Data Structure used by Hindus involved
place-value notationbase-10 representationthe zero as placeholder
DS enabled simplified algorithms to do arithmetic operations such asaddition (+), subtraction (-), multiplication (*) and division (/)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 85 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
What function/computation could the ocular dominancecolumns serve?
Slide title an example of a broader question regarding the visualcortex as a whole
Comp.Sc
Brain
info. representation DS OcuDom columns inV1
info. transformation (computation) algorithm windowed cepstralfilter (suggested)
cepstral filter:
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 86 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Why is the world perceived as 3-D?Why do we not perceive the world as 2 2-D images?
Images projected onto the retinas are 2-D (i.e. flat, lacking”thickness”)
But we do not (visually) perceive the world as two nearly identical(but slightly laterally displaced) and flat images
So where does the perception of ”depth“ come from?
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 87 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
3-D movies and 3-D glasses
3-D movies
Chota ChetanAvatar
3-D glasses fuse the two images into one to give depth perception
In day-to-day life, the brain automatically fuses the 2 2-D imageson the retina, so we perceive the world as one 3-D
How does the brain do it?
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 88 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Depth Perception
1 Depth perception refers to the perception of “solidity“
2 Brain uses many different cues to perceive depth
3 Cues may be monocular (single eye) or binocular (both eyes)4 Monocular Cue: Geometric information such as occlusion used to
compute depth
Occlusion refers to the hiding of one surface by another due to opacityof the intervening surface.Provides info re relative depth
5 Here we consider only one depth clue, a binocular clue calledstereopsis
6 Stereopsis is closely connected with binocular disparity
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 89 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Stereopsis
1 “Stereo“ ”solid“ or ”three-dimensional“
2 ”Opsis“ View or Sight
3 Stereopsis means ”three-dimensional vision”
4 Slightly misnamed phenomenon
5 ∵ Stereopsis refers to one of many ways humans perceive depth
6 Stereopsis: Perception of depth arising from the slightly differentprojections of the world to form two slightly different retinalimages
7 Only images themselves are used to perceive depth (see Random dotstereograms)
8 Other cues - such as occlusion - not used
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 90 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Binocular Disparity
Difference in image location of an object as seen by left and right eyes
Arises due to the separation b/w the two eyes
Left Eye Right Eye
CP F
F
P
C
C
PF
C closer points, crossed disparity
P point of fixation, no disparity
F further points, uncrossed disparity
Binoc. disp. used to extract depth info from the two two-dimensionalretinal images in stereopsis.
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 91 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Computing disparity
Retinal pixels do not come with “labels“ indicating correspondingpixels in the images
∴ binocular disparity b/w images must be computed from the imagesthemselves - first step in any stereo algorithm
Algorithm will have to figure out corresponding points b/w theimages.
Then disparity can be easily computed
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 92 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Binocular disparity, ”filling-in“, and stereopsis
Input Data (aka “Image“); Pair of “almost” “identical” images, onelaterally displaced from the other.
Intermediate step: Compute disparity i.e. lateral displacementb/w the images
Output: Stereopsis or a perception of depth
Final Step: following computation of disparity, ”fill in“ smoothsurfaces.
We will consider an algorithm for signaling disparity that the brainmay perform.
We do not discuss the step of ”filling in“
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 93 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Why do we not perceive the world as 2 2-D images?First (incomplete) solution
Because we focus our eyes on a point.
Shortcoming of this answer: circle (sphere) of fixation (horopter)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 94 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Horopter
The two eyes, and the point of fixation determine a circle called thehoropter
(Recall from school geometry that 3 points determine a circle)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 95 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Horopter: Significance
Points on horopter cast images at exactly corresponding pts inthe two retinae
Can be fused into one image w/o difficulty when the optical pathsfrom the eyes cross in V1
Points to the front and back cause images on the retinae that arelaterally displaced
Consequence: Except for a sphere of infinitesimal thickness, (almost)all points in visual field should give rise to double images.
Yet we do not perceive the world that way! Why?
Answer: Cortex performs a computation that
fuses the two images into one perceptin doing so, creates an illusion of depth
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 96 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Where does fusion take place?
Fusion can occur only if the optical info from two eyes’ imagesconverge.
In the retina and (visual) thalamus, the paths are separate
Convergence happens in area V1
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 97 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
What comes next .....
Human Stereo: Perceptual Aspects
Stereopsis: Formal statement of the computational problem(w/o ref to how the brain does stereo vision)
Random dot stereograms and their significance
Overview of suggested algorithms (w/o ref to how the brain doesstereo vision)
How the brain (V1) does stereopsis
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 98 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Perceptual Aspects of Human Stereo Vison IPanum’s Fusion Area
Region around horopter where disparate images can be perceptuallyfused into a single objectPsychophysical data s.t. area increases (linearly) with eccentricity, ǫ
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 99 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Perceptual Aspects of Human Stereo Vision IIRobustness of Stereo Perception
Human stereo vision is robust to many kinds of imagemanipulations:
image degradations (including Gaussian blur, random noise, changesin image intensity),rotations (upto 6 degrees),and differential expansions (upto 15%)
A complete psychophysical theory of stereoscopic depth perceptionshould explain these perceptual data re Panum’s area and robustness
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 100 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Stereopsis: The Computational Problem
Stereopsis includes
Matching corresponding image points in the two eyes (so-calledCorrespondence Problem)measuring their disparity,from this info. recovering the 3-D structure of the objects seen by theviewer (stage of “filling-in”)
Human stereopsis is also robust to minor image manipulatons of oneor the other image, as seen above
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 101 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
StereopsisThe Correspondence Problem
Matching corresponding points could be done either
after recognizing objects - easy and unambiguous computationbefore recognizing objects - using only disparity information
Physiological evidence indicates that binocular cells lie in V1 (muchbefore object recognition takes place in IT cortex)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 102 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Random dot stereograms aka Magic-Eye stereograms
Magic Eye stereogram: p. 215 Vision: From Photons toPerception
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 103 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Random dot stereogramsSignificance
Showed brain can compute stereoscopic depth w/o:
Objects
Perspective
Cues available to either eye alone i.e. monocular cues such asocclusion
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 104 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Radha Krishna image: Not a pure random dot stereogram
Same Principle
We can only form the percept after fusing the two images
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 105 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Stereopsis: The Computational ProblemSuggested Algorithms
Before studying how the brain does stereopsis (we don’t really know)...
We overview various proposed computational/algorithmic solutionsand their limitations from a:
a computational perspectivein terms of biological implementability i.e. whether the brain couldactually use the proposed solution
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 106 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Stereo algorithmsCategories
1 Pixel-based schemes
2 Edge-based schemes
3 Area-based schemes
4 Filtering algorithms
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 107 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Stereo algorithmsCategories II
Szeliski and Zabih have proposed another classification:
1 Correlation-style stereo algorithms:Example of an (unconventional) correlation-based algorithm that isbiologically plausible
2 Global methods
Use well-chosen energy minimization to obtain a depth map thatminimizes some energy functionMinimization can be done by means of simulated annealing, graphcuts, or mean field methodsOne global method that does not use energy minimization is due toZitnick and Kanade
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 108 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Stereo AlgorithmsCategories III
Stereo algorithms can also be classified based on whether algorithmused is:
1 sequential (uses iterations or relaxation, such as simulated annealing):Difficulty: Speed w/ which humans do stereo too fast for iterationschemes for computing stereo
2 parallel
We will consider a windowed (noncanonical) correlation-basedone-shot parallel algorithm called the windowed cepstrum due toYeshurun and Schwartz
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 109 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Algorithms for Stereo Vision IPixel-Based Schemes
Algorithms for computing stereo can be categorised into:
Pixel-based algorithms:Example: 1st Marr-Poggio algorithm (1979)
Match individual pixels in left and right images.Computational difficulty:
Correspondence problem: which features in one (retinal) imagecorrespond to which features in the otherNOT robust to minor changes other than simple shifting (unlikehuman stereo vision)
Biologically implausible -
Only in retina are RFs “pixel-like‘‘.But stereopsis requires convergence of info. from both eyes, cannottake place in the retina, ∵ images separateOTOH, in V1, edge detection would have already taken place.∴ unclear why the brain would use pixels to do stereo
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 110 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Algorithms for StereoEdge-Based Schemes
Edge-Based Algorithms: Marr and Poggio also presented anedge-based algorithm for stereo.
Algo attempts to match edges in the two images rather than pixelsComputational Issues:
Correspondence problem not as bad, ∵ (usually) far fewer edgesthan pixels in an image.OTOH, algo requires pre-processing to detect edges (itself anon-trivial problem)Limitation: Resulting depth information sparse, ∴ available at only afew locations.Further step needed to interpolate depth across surfaces in a scene
Biologically Plausibility:
More biologically plausible ∵ it’s known that binocular processingbegins in V1 after optical info. flow via simple cells (which may act asedge detectors)But not robust to the kinds of image manipulations discussed above
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 111 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Algorithms for Stereo VisionArea and Filtering-Based Schemes
Area-based algorithms:
Correlation b/w brightness patterns in local neighborhood of a pixel w/brightness patterns in the local nhd. of the other image.Simplest is correlation
Filtering Algorithms: Jones and Malik (1992)
Algo matches local regions around the point.Uses biologically inspired linear spatial filters that differ in theirorientation and size tuning, called multi-oriented multi-scale(MOMS) filters.Inspiration from biology: ∃ cells in a hypercolumn whose RFs arecentered on the same retinal position, each cell tuned to differentorientation and scale
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 112 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Jones and Malik’s motivation:
Some good algorithm for computing stereo matchingsBiological inspiration was the starting point, not the criterion forjudging soundness
Lecture theme: Algorithmic strategy that takes biological constraintsinto account - ”how does the brain do it‘‘
A windowed (unconventional) correlation-based scheme developed byYeshurun and Schwartz
Cross-Correlation (not the usual) used is called the cepstrum
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 113 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
How V1 does stereopsis
Yeshurun and Schwartz, in a series of papers, indicated:
Possible functional (computational) role for the ocular dominancecolummns by showing how:
OcuDom columns enable a data structure allowing a
One-shot (i.e. no iterations required) algorithm called cepstral filter to
Provide a robust solution to stereo matching that is
Consistent with psychophysical data on stereopsis
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 114 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Data StructureOcular Dominance Columns
Visual (hemi)-field representation:Right cortical hemisphere gets input from left visual hemifield fromboth eyes (and vice-versa)
Interlaced image:
A B
A C B D
C D
interlaced image
Cut the two images into stripsPlace corresponding strips in alternate positions
In cortex, OcuDom columns w/ left and right eye image strips fromleft hemifield alternating
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 115 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Power Spectrum or Power Spectral Density (PSD)
Power spectrum or Power Spectral Density (PSD) of a signal: Thesignal power as a function of frequency
Measures power content of component frequencies of the signal
Fourier transform of the autocorrelation function
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 116 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Algorithm supported by the data structureCepstrum
Cepstrum is the power spectrum of the log of the power spectrum
Windowed cepstrum can be applied to the binocular interlaced image
Computational Issues:
Easy to compute windowed spectrum given above DSNeed to estimate power spectral density and apply a log
Biological plausibility: Cortical neurons can:
Act as medium-bandwidth power spectral filtersPerform logarithmic computation and multiplication
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 117 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Cepstrum calculation I
Suppose the left and right eye images are identical i.e. no disparity
Suppose the width of a column is D
Image s(x , y) and the identical image repeated and abutted, viz.,s(x − D, y)
Interlaced image, composed of a single column pair is given byf (x , y) = s(x , y) ⊗ {δ(x , y) + δ(x − D, y)}
where ⊗ represents convolution
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 118 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Cepstrum calculation II
Fourier transform of interlaced image is:F (u, v) = S(u, v) · {1 + e−iπDu}
Log is: logF (u, v) = logS(u, v) + log(1 + e−iπDu)
Fourier transform of the 2nd term on RHS has a prominent peak atdisparity D (can be shown)
Peak position can be recovered by a peak detection algorithm
Spatial position in the cepstrum is a direct measure of”disparity“ of left and right images
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 119 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Requirements for a good theory of human stereopsis
Cepstrum robust to image manipulations that human stereo is robustto
We now turn to how Panum’s area change with eccentricity accordingto psychophysical data is explained by this algorithm
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 120 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Perceptual data re Panum’s area explained by the abovealgorithm
DS used for computation of stereopsis: OcuDom columns
Combined width of a pair of OcuDom columns (i.e. hypercolumn) is2 mm, almost constant through V1
Recall: ∆w∆z
≈ 1z+a
,
In words, ratio of change in cortical position to visual angle subtendedis prop. to magnification factor
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 121 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Perceptual data re Panum’s area explained by the abovealgorithm II
∴ ∆z ≈ ∆w1
z+a
If ∆w hypercolumn width, then ∆z is visual angle subtended
∴ angle subtended linearly proportional to inverse magnification factor
Magnification factor 1z+a
inversely proportional to eccentricity ǫ
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 122 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Perceptual data re Panum’s area explained by the abovealgorithm III
∴ angle subtended by hypercolumn scales linearly w/ eccentricity ǫ
Known from psychophysical experiments: Panum’s area (range offusion) also scales linearly w/ eccentricity
Data consistent w/ the hypercolumn as a functional(computational) module to compute stereo fusion
We conclude that range of fusion extends over a single hypercolumn,regardless of position in the visual field
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 123 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Edges
What are edges? Answer: Discontinuous changes in image luminance
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 124 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Lecture Theme: Algorithms the brain employs to do edgedetectionRF scatter enhances boundary contour representation in V1
Give a brief summary of Bomberger and Schwartz’s paper
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 125 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Eric L. Schwartz’s question
What are the computational functions of the visual cortex?
To Computer Science students of Computer Vision interested in howthe brain does vision, Eric Schwartz’s web-site:
http://cns.bu.edu/∼eric/
Plus a beautiful display of the complex log map!
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 126 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Potpourri: Suggested theoretical principles for thecomputational functions of cortexPoints to ponder, and names to Google
Bayesian Inference: David Mumford
Liquid State Machines: Wolfgang Maass, in collaboration with HenryMarkram
No. of ”top-down“ connections approx. ten times number of”bottom-up“ connections. What is the computational role played bythese? After all, facial recognition takes place in time too short fortop-down connections to play a role.
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 127 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Potpourri II: Suggested theoretical principles for thecomputational functions of cortexPoints to ponder, and names to Google
Schwartz and collaborators have taken one approach to therelationship between architecture and function, a continuum approach
Laurence Abbott and Peter Dayan have attempted to describenetwork connectivity and dynamics, which treats the neurons as”discrete“ units
“What and ”where“ pathways
Laminar Computation: Stephen Grossberg
Terrence Sejnowski
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 128 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Eric SchwartzNeuroscience’s Enrico Fermi
Recurring theme in his work: Biological plausibility of suggestedalgorithms for a computational task(s)
IOW, how the brain does it
Eric Schwartz: Department of Cognitive and Neural Systems, BostonUniversity
Professor of Cognitive and Neural SystemsProfessor of Electrical and Computer EngineeringProfessor of Anatomy and Neurobiology,PhD Experimental High Energy Physics, Columbia
His work demonstrates how a top-quality experimental science anddata analysis can suggest mathematically formulated theoretical work
How an all-round scientist (he has theoretical insights andexperimental and modeling work of the highest quality to his credit)can be at home with all of them
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 129 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Eric Schwartz
Experimental (wet-biology) work in Neurophysiology
Mathematical techniques from following subjects used in his work:
Computer Vision (Algorithms for space-variant vision)Image ProcessingSignal Processing (Application of cepstrum)Complex Analysis (Numerical conformal mapping)Graph Theory (Graph partitioning)
Non-trivial contributions to machine vision and robotics
Built a self-navigated robot
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 130 / 169
Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex
Eric Lee Schwartz’s webpage
Home Page: http://cns.bu.edu/∼eric/
Wikipedia entry: Eric L Schwartz
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 131 / 169
Single Neuron Computation Single Neuron Basics
Part III: Single Neuron Computation
How a single neuron looks to a:
Biologist - Basic terminology
Electrical Engineer or a Computer Scientist -
McCulloch-Pitts neuronRosenblatt’s PerceptronSpiking Neural Networks
Biophysicist: Hodgkin-Huxley model - describes biophysicalmechanisms of action potential generation
Theoretical Bio(logical) Physicist: William Bialek and co-workers
One of the world’s top experts in single neuron computation isBartlett Mel - I encourage you to explore his work
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 132 / 169
Single Neuron Computation Single Neuron Basics
Single Neuron: Biology basics I
Cell body (blue) - contains most of the cell’s volume, mass andnucleusAxon hillock: o/p ”terminal“ of the neuronAxon (or o/p ”wire“ of the neuron) marked RedDendrites: neuron’s i/p terminals (blue tendril-like structures on cellbody)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 133 / 169
Single Neuron Computation Single Neuron Basics
Single NeuronBasic Terminology
Soma aka Cell body: ”The” neuron.
Membrane Potential: Difference in voltage between the neuron’sinterior and exterior
Dendrites: Input terminals: How a neuron receives information fromother neurons
Axon Hillock: Output terminal: Where the output is generated
Axon: Neuron messages other neurons via this “wire”
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 134 / 169
Single Neuron Computation Single Neuron Basics
Single NeuronBasic Terminology
Action Potential aka Spike:Role: A neuron’s way of signaling other neuronsMechanism: Biophysical mechanisms cause a positive feedback processto cause the membrane potential to rise dramatically at the axonhillock, and this voltage change is transmitted down the axon.
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 135 / 169
Single Neuron Computation Single Neuron Basics
Single Neuron: Biology basics IIPyramidal Cell
Primary excitatory neurons in the cortex, hippocampus, amygdalaSo-called because soma or cell body is ”triangular“ or ”pyramid“shape
Stylised figure aboveKrishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 136 / 169
Single Neuron Computation Single Neuron Basics
Single Neuron: Modeled as an Artificial Neuron (circa1950s and 1960s)
Threhold Logic Unit, developed by McCulloch and Pitts, and calledthe McCulloch-Pitts neuron
Based on neurophysiology of the 1950so/p = φ(Σm
j=0wjxj)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 137 / 169
Single Neuron Computation Single Neuron Basics
Single Neuron: Modeled as an Artificial Neuron (circa1950s and 1960s) Contd
Perceptron, developed by Rosenblatt
Linear Binary classifierFeedforward neural network.Linear weighted summation (i.e. projection) followed by anonlinearity such as thresholding.Similar to McCulloch-PittsRosenblatt also developed a learning algorithmLearning algo does not terminate if i/p linearly inseparable
Perceptron cannot learn XOR (Not linearly separable)
Multilayer perceptrons overcome many of these obstacles
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 138 / 169
Single Neuron Computation Single Neuron Basics
Single Neuron: Modeled as an Artificial Neuron: circa2010sSpiking Neural Networks aka SNNs
Current generation of artificial neural network models attempt to moreclosely model the behaviour of a single neuron or a small microcircuit
Biological neurons may encode information in the timing of theiroutput (pulse code), and not just their average rate (as we will seebelow)
Motivation: To study information encoded in spike timing andspatiotemporal codes
See Wikipedia article on Spiking neural network
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 139 / 169
Single Neuron Computation Single Neuron Basics
Single Neuron: Modeled as an Artificial Neuron II: circa2010sSpiking Neural Networks
Textbook: Title: Spiking Neuron ModelsSubtitle: Single Neurons, Networks, PlasticityAuthors: Wulfram Gerstner, Werner KistlerTextbook available at his website, and also on the Wikipedia site
Liquid State Machines have the universal function approximationproperty
LSMs developed by :
Wolfgang Maass (theoretical computer scientist to Neuroscience)collaborator Henry Markram (biochemistry (?) switched toNeuroscience, director of the Blue Brain project)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 140 / 169
Single Neuron Computation Single Neuron Basics
Single Neuron Spiking: The first quantitatively successfulmodel in Neuroscience
developed by physiologists-cum-biophysicists Hodgkin and Huxley
Nobel Prize for Physiology or Medicine 1963
nonlinear ordinary differential equations describing
ionic mechanisms underlying the generation of an action potential
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 141 / 169
Single Neuron Computation Single Neuron Basics
Hodgkin-Huxley model of the “firing“ of a Squid Axon
Squids: Marine life-form, largest known invertebrates.
Axon is huge 1 millimeter in diameter - ∴ good to experimental work
nonlinear ODEs analytically intractable - ∴ numerical methods used
Model has been successfully extended to describe biophysics ofspiking behaviour in other animals and other kinds of neurons
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 142 / 169
Single Neuron Computation Single Neuron Basics
Why study spiking neur(on)al networks?Ans: Behaviors associated w/ SNNs
Single neuron:
Adaptation to constant stimuliRefractory period after a spike generation’Bursting’ behaviour
Neuronal populations:
ChaosSynchronization
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 143 / 169
Single Neuron Computation Single Neuron Basics
Spiking neuronal networks underlie physiologicalphenomena
Circadian rhythm, of which the sleep-wake cycle is one example
Epilepsy
Many kinds of learning. Ex: Hippocampal theta rhythm
Central Pattern Generator (CPG): a neuronal n/w producing rhythmicpatterned output without sensory feedbackInvolved in locomotion and respiration, among others
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 144 / 169
Single Neuron Computation Single neuron as a ”computer“
Beyond Hodgkin-Huxley: The single neuron as a”computer“
William Bialek’s question: What is the computation performedby a single neuron?
Can we find a general mapping between the description of a neuron:
as a dynamical system (Hodgkin-Huxley model mathematicaldescription in terms of nonlinear ODEs)and in terms of the computational functions of the neuron(mathematical description in terms of information theory, theory ofcomputation, computational learning theory)?
Coding: Representation of Information
Computation: Transformation b/w different representations
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 145 / 169
Single Neuron Computation Single neuron as a ”computer“
Holy Grail of Neuroscience: The Neural Code
Schwartz and collaborators work discussed above refers to therepresentation of information at the level of maps
In the case of coding by a single neuron, or by a small population ofneurons, both
spatial scaletemporal scale
are much smaller than the case with cortical mapping (i.e.topographic organization)
The Neural Code refers to these dynamically changing firingpatterns
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 146 / 169
Single Neuron Computation Single neuron as a ”computer“
Neural CodingEncoding and Decoding
Neural coding can be studied from two points of view:
Encoding: Map from stimulus to response
Decoding: Reverse map from response to stimulus
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 147 / 169
Single Neuron Computation Single neuron as a ”computer“
Neural CodingEncoding
1 Encoding: Map from stimulus to response
Neuronal tuning curve and the receptive fields were two of the earliestmodels of neural encodingNatural viewpoint for the earliest experimenters, who tried to figure outwhat stimuli would maximize neuronal firingBuild models that predict the neural response as a function of stimulusBut ...
Neural responses are stochastic and show high variabilitySame stimulus may elicit different responses on different trials
∴ encoding models try to capture average properties of neuralresponses, such as the average firing rate - the tuning curve being anexample
2 High variability of responses to stimuli raises an interesting question:
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 148 / 169
Single Neuron Computation Single neuron as a ”computer“
Neural CodingEncoding Cont’d
How is the neuronal response to be mathematically characterised?
First-the stimulus mathematical characterisation
The best discussion (by far) of these matters is in the textbook:Title: Spikes: Exploring the Neural CodeAuthors: Fred Rieke, David Warland, Rob de Ruyter van Steveninck,William Bialek
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 149 / 169
Single Neuron Computation Single neuron as a ”computer“
Characterizing the stimulus
1 The stimulus can be considered to be a random variable drawn from acollection of possible stimuli
2 The particular stimulus is a realization
3 Throwing a die:
4 Random Variable :outcome ofexperiment,
5 Ensemble is the collection of possible outcomes 1, 2, 3, 4, 5, 6,
6 Realization is the face that turns up after the throwing
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 150 / 169
Single Neuron Computation Single neuron as a ”computer“
1 Since both stimulus and response are random variables, with a(probabilistic) causation linking them, the joint distributionP(Stimulus, Response) is well-defined
2 Also, due to neuronal variability, the response is a probabilisticfunction of the stimulus
3 This can be modeled by a conditional probability distribution. Thus:P(Response|Stimulus)
4 By Bayes’s rule, we haveP(Stimulus|Response) = P(Stimulus)∗P(Response|Stimulus)
P(Response)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 151 / 169
Single Neuron Computation Single neuron as a ”computer“
Neural CodingDecoding
Decoding: Reverse map, from response to stimulus
Goal try to reconstruct some aspects of the stimulus
The “internal“ ”observer’s“ point of view
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 152 / 169
Single Neuron Computation Single neuron as a ”computer“
Need for mathematizing the questions of neural coding
Using Bayes’s Rule, we can characeterize the neural code by:
either listing the rules for translating stimuli into spikes -encoding
or by listing the rules for translating spikes into stimuli -decoding
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 153 / 169
Single Neuron Computation Single neuron as a ”computer“
Rate Codes, Temporal Codes, Population Codes
Early workers posited that neurons signaled/messaged/encoded infoby avg. firing rates
Rationale: When a neuron receives a stimulus it’s tuned to, it sendsout a stream of spikes
Otherwise it stays (almost) silent
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 154 / 169
Single Neuron Computation Single neuron as a ”computer“
Problem w/ Rate Code IdeaSimplest Experimental Objection
Known that facial recognition takes place within 150 milliseconds
Face recognition takes place in inferotemporal cortex, and requiresthat information be routed through at least 6 stages:
Retina(Visual) ThalamusArea V1Area V2Area V3Infero-Temporal Cortex
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 155 / 169
Single Neuron Computation Single neuron as a ”computer“
Problem w/ Rate Code Idea IISimplest Experimental Objection
∃ a time lag of 10 milliseconds between any two stages. Leaves about150 - 50 = 100 milliseconds for within stage processing over 5 stages
With 5 stages, this means an average of 20 milliseconds per stage
To compute an average firing rate in 20 milliseconds, assuming atleast 2 spikes are needed, would require that neurons in visual cortexspike at rates of 100 per second.
No neurons are known that fire at this rate
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 156 / 169
Single Neuron Computation Single neuron as a ”computer“
The Neural CodeTemporal Codes aka How neurons represent information in their firing pattern(s)
Individual neurons represent info in timing of o/p pulses, and not justtime-averaged rate (spike-count code)
Familiar to communication engineers in the form of pulse positionmodulation, where pulse time encodes info
Introduce a figure here showing PPM
A neural population can encode info. in relative timing of spikes
See Wikipedia article on phase-of-firing code: a temporal codecombining spike-count code w/ a time reference based on slowoscillations
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 157 / 169
Single Neuron Computation Single neuron as a ”computer“
The Neural CodeTemporal Codes aka How neurons represent information in their firing patterns
Spatiotemporal population activity patterns can support codes ofimmense complexity
Complex codes necessitate high computational power
Some questions arise: Can we quantify the computational power of aneuron/neuronal network?
Computational/Statistical Learning TheoryVapnik-Chervonenkis dimension
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 158 / 169
Single Neuron Computation Single neuron as a ”computer“
Computation: Neurons as feature detectors
Notion of neuron as a feature selector goes back to the receptive field
Images: Vectors in a high dimensional space,
Pixel intensities: coordinates
Receptive fields: Neuron as sensitive to one direction or projection inthis high dimensional space, and performs a correlation
Linear summation followed by thresholding to determine the firingprobability recalls the perceptron
Complex cells receive i/p from many simple cells,
Complex cells perform projections followed by nonlinear summation
and thresholding
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 159 / 169
Single Neuron Computation Single neuron as a ”computer“
Reverse correlation aka Spike-Triggered Average (STA)
Average stimulus preceding a spike
Provides an estimate of the neuron’s linear receptive field
Limitation of technique:i/p-o/p function of a neuron is inherently nonlinear:
adaptation of neural firing rates to a constant inputrefractory periods during which a neuron will not fire, no matter whatthe input
Assumption: No. of stimulus dim to which the neuron responds is 1, orin the worst case, a very small number
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 160 / 169
Single Neuron Computation Single neuron as a ”computer“
Estimating subspace dimensionality from data
Experimental support for the idea of characterizing the neuronalresponse as a reduction of dimensionality:
Studies of motion-sensitive neuron in the fly visual system
I encourage you to visit William Bialek’s web-page at Princeton
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 161 / 169
Single Neuron Computation Single neuron as a ”computer“
Motion estimation... in the fly visual system
Bialek and co-workers have shown that the fly’s estimate is the bestpossible given noise in its motion sensitive cell
”Best possible” as computed by applying optimal estimation theoryprinciples
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 162 / 169
Single Neuron Computation Single neuron as a ”computer“
Charaterizing the computation performed by a neuron
Three steps:
Estimate the number of relevant stimulus dimensions, and hope it issmall
The “curse of dimensionality” should be familiar to students ofmachine learning (CS 419, CS625, CS725)
Identify a set of filters that project onto the relevant subspace
Characterize the nonlinear function (Perceptron: Nonlinear function isthresholding)
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 163 / 169
Single Neuron Computation Single neuron as a ”computer“
Optimization as an overarching theoretical principle
Bialek and co-workers have discovered for optimal performance in:
photon counting in visionoptimal coding in spike trains
Neural systems seem to have reached the performance limits imposedby contraints on neural circuitry such as noise and energyconsumption
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 164 / 169
Single Neuron Computation Single neuron as a ”computer“
Part V: Networks of Neurons
I have chosen to tread lightly over the level of modeling inneuroscience that is most familiar to engineers and CS types, namely,the network level
One can study the connectivity of neurons in networks, whilede-emphasizing the role of biophyiscal mechanisms at the singleneuron scale, and also ignoring the map level scale of 1 cm
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 165 / 169
Single Neuron Computation Single neuron as a ”computer“
Neuronal Network Modeling as a tool to study networks ofneurons
NEURON
Liquid State Machines: A concept pioneered by Wolfgang Maass andHenry Markram
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 166 / 169
Single Neuron Computation Single neuron as a ”computer“
Pawan Sinha, CS switched to Neuroscience, currently at MIT,Cambridge, USA
In India, Arun Sripati at the Indian Institute for Science, Center forNeuroscience, alumnus of IIT Bombay, dept. of Electrical Engineering,PhD, Johns Hopkins, Electrical Engineering and Neuroscience
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 167 / 169
Single Neuron Computation Single neuron as a ”computer“
Textbooks and Websites
This list reflects my own tastes and interests
1 Theoretical Neuroscience by Peter Dayan and Laurence Abbott
2 Spikes: Reading the Neural Code by Fred Rieke, Rob de Ruyter vanSteveninck, David Warland and William Bialek
3 The Computational Brain by Terrence Sejnowksi and PatriciaChurchland
4 Vision: From Photons to Phenomenology by Stephen Palmer
5 Computational Neuroscience by Eric Schwartz (Not a textbook.Collection of original research papers)
6 Eye, Brain and Cortex by David Hubelhttp://hubel.med.harvard.edu/book/bcontex.htm
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 168 / 169
Single Neuron Computation Single neuron as a ”computer“
Acknowledgements
Thanks are due to:
Prof. Pushpak Bhattacharyya, for giving me this opportunity toaddress CS students of IIT Bombay
Teaching Assistants: Janardhan and Munish Minia and Arindam
Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 169 / 169