a bird's-eye view of petavision, the world's first petaflop/s neural...
TRANSCRIPT
![Page 1: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/1.jpg)
A Bird's-Eye View of PetaVision, theWorld's First Petaflop/s Neural Simulation*
Dan CoatesPortland State University,
Maseeh College of Engineering and Computer Science,
Portland OR
Parallel Implementations of Learning Algorithms:“What Have You Done For Me Lately?”
NIPS08Whistler, BC
December 13, 2008
Dan Coates Garrett Kenyon,Craig Rasmussen
Los Alamos National Laboratory, Los Alamos, NM
* The authors acknowledge the support of the National Science Foundation, under a grant administered by the New Mexico Consortium
![Page 2: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/2.jpg)
12/16/08 Coates NIPS08: PetaVision 2
PetaVision Project at LANL
Goal: Achieve human-level performancein a “synthetic visual cognition” system
On: IBM/DOE Roadrunner petascalesupercomputer (or a multicore PC)
Running: A spiking LIF neural networkinspired by visual cortex.
![Page 3: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/3.jpg)
12/16/08 Coates NIPS08: PetaVision 3
What level of abstraction?
Emulate the cortical circuits formid/low-level visual processing.
We model the gross architecture ofvisual cortex, trying not to violateproven neural science.
Binzegger, et. al.
![Page 4: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/4.jpg)
12/16/08 Coates NIPS08: PetaVision 4
What are the crucial features of V1?
Retinotopic mapping.
Edge detectorsof Hubel & Wiesel
Distinct laminar neuralpopulations.
![Page 5: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/5.jpg)
12/16/08 Coates NIPS08: PetaVision 5
What are the crucial features of V1?
Retinotopic mapping.
Edge detectorsof Hubel & Wiesel
Distinct laminar neuralpopulations.
![Page 6: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/6.jpg)
12/16/08 Coates NIPS08: PetaVision 6
What are the crucial features of V1?
Bannister. Laminar circuit.
Retinotopic mapping.
Edge detectorsof Hubel & Wiesel
Distinct laminar neuralpopulations.
![Page 7: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/7.jpg)
12/16/08 Coates NIPS08: PetaVision 7
Spiking neurons and specific connectivity
- efficient, possibly asynchronous operation- sparse inter-node communication
What are the elements, and how does that help us?
![Page 8: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/8.jpg)
12/16/08 Coates NIPS08: PetaVision 8
Spiking neurons and specific connectivity
- connections are primarily local
- function inherent in wiring
What are the elements, and how does that help us?
Bosking, et al. “Patchy” connectivity expresses orientation preference ofhorizontal connections.
![Page 9: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/9.jpg)
12/16/08 Coates NIPS08: PetaVision 9
Example: edge detection
V1 simple cells have been shown to respondlike a Gabor functions. We use 8 orientations.
![Page 10: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/10.jpg)
12/16/08 Coates NIPS08: PetaVision 10
Beyond edges: long-range association field
Ben-Shahar and Zucker have proposedadditional connectivity patterns formalizedusing differential geometry. [Neural Computation, 2004]
Besides curve integration, such a schemecould also be used for shape-from-shadingand natural texture identification.
![Page 11: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/11.jpg)
12/16/08 Coates NIPS08: PetaVision 11
Ben-Shahar and Zucker have proposedadditional connectivity patterns formalizedusing differential geometry. [Neural Computation, 2004]
Besides curve integration, such a schemecould also be used for shape-from-shadingand natural texture identification.
Beyond edges: long-range association field
![Page 12: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/12.jpg)
12/16/08 Coates NIPS08: PetaVision 12
Beyond edges: long-range association field
![Page 13: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/13.jpg)
12/16/08 Coates NIPS08: PetaVision 13
Summary of Biological Inspiration
Network structure for a computer visionsystem can be modeled after architectureof mammalian visual cortex.
There are analytic correlates of thesetechniques, although closed-formderivations are difficult.
Note: these connections have beenshown to be learnable, although wehard-code as mathematical functions.
![Page 14: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/14.jpg)
12/16/08 Coates NIPS08: PetaVision 14
Implementation: software abstractions
PVLayer: Population of neurons. Retina, LIF.
PVConnection: Connectivity pattern,represented by a mathematical weight function.Anything-to-anything routing possible.
![Page 15: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/15.jpg)
12/16/08 Coates NIPS08: PetaVision 15
Implementation: LIF
![Page 16: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/16.jpg)
12/16/08 Coates NIPS08: PetaVision 16
Implementation: PVConnection
Connection kernels are translation-invariant.
![Page 17: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/17.jpg)
12/16/08 Coates NIPS08: PetaVision 17
Implementation: Parallel Algorithm
Process each presynaptic event
Process each PVConnection:
Update effected postsynaptic neurons
Update each layer
Perform I/O
![Page 18: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/18.jpg)
12/16/08 Coates NIPS08: PetaVision 18
How to interpret results?
Readout: Spike trains are post-processed forfiring rate. Temporal correlations such assynchrony and oscillatory power are alsomeasured.
![Page 19: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/19.jpg)
12/16/08 Coates NIPS08: PetaVision 19
Roadrunner
![Page 20: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/20.jpg)
12/16/08 Coates NIPS08: PetaVision 20
Roadrunner “core”
1 Opteron Core●1.8 Ghz● 4 GB DDR2
IBM PowerXCell 8i● 3 Ghz clock speed● 200 Gflops w/singleprecision & pipelining● 4 GB DDR2● SPEs = 256k
![Page 21: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/21.jpg)
12/16/08 Coates NIPS08: PetaVision 21
Roadrunner node: triblade
![Page 22: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/22.jpg)
12/16/08 Coates NIPS08: PetaVision 22
Roadrunner
3,240 nodes:- 2 Opteron dual-cores1.8Ghz,16 GB memory- 4 PowerXCell 8i
Infiniband connections
Peak system performance: ~1.7 Petaflop/s.
![Page 23: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/23.jpg)
12/16/08 Coates NIPS08: PetaVision 23
Each node handles an image patch.
PetaVision SPMD on Roadrunner
![Page 24: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/24.jpg)
12/16/08 Coates NIPS08: PetaVision 24
PetaVision SPMD on Roadrunner
Process local activity
Process remote activity
Update layer
Send output spikes
![Page 25: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/25.jpg)
12/16/08 Coates NIPS08: PetaVision 25
Roadrunner SPMD Components
MPI
Coordination
Math: Euler IF & Connections
![Page 26: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/26.jpg)
12/16/08 Coates NIPS08: PetaVision 26
Visual task
“Closed contour present?”
- No need for higher-levelknowledge
- Nontrivial
- Humans can solveeffortlessly. (psychophysics)
![Page 27: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/27.jpg)
12/16/08 Coates NIPS08: PetaVision 27
Visual task - results
Prototypical network response.Color represents average firing rate.
![Page 28: A Bird's-Eye View of PetaVision, the World's First Petaflop/s Neural ...dst/NIPS/nips08-workshop/Dan_Coates_slides.pdf · A Bird's-Eye View of PetaVision, the World's First Petaflop/s](https://reader033.vdocuments.mx/reader033/viewer/2022041916/5e69d40d169121176c5837ae/html5/thumbnails/28.jpg)
12/16/08 Coates NIPS08: PetaVision 28
Thank you!
Questions?