a novel multi-gpu neural simulator
DESCRIPTION
A Novel Multi-GPU Neural Simulator. Corey Thibeault , Roger Hoang Frederick C. Harris, Jr. Brain Computation Laboratory University of Nevada, Reno. Reno, Nevada. University of Nevada, Reno. Brain Computation Laboratory University of Nevada, Reno in funded collaboration with - PowerPoint PPT PresentationTRANSCRIPT
A Novel Multi-GPU A Novel Multi-GPU Neural SimulatorNeural Simulator
Corey Thibeault, Roger HoangCorey Thibeault, Roger Hoang
Frederick C. Harris, Jr.Frederick C. Harris, Jr.
Brain Computation LaboratoryBrain Computation LaboratoryUniversity of Nevada, RenoUniversity of Nevada, Reno
Reno, NevadaReno, Nevada
University of Nevada, RenoUniversity of Nevada, Reno
Brain Computation LaboratoryBrain Computation LaboratoryUniversity of Nevada, RenoUniversity of Nevada, Reno
in funded collaboration with in funded collaboration with U de Cergy-Pontoise and CNRS, Paris, FranceU de Cergy-Pontoise and CNRS, Paris, France
Brain Mind Institute (Blue Brain Project), EPFL, Lausanne, SwitzerlandBrain Mind Institute (Blue Brain Project), EPFL, Lausanne, Switzerland
Mathias Quoy René Doursat
Henry MarkramJim King
Laurence Jayet Fred Harris, Jr Sergiu Dascalu Director
UNR Brain Computation Lab
Neuroscience
Scope of Work
Modeling & Neural Engineering
Robotic/Human Loops
ONR N000140710018: “Large-Scale Biologically Realistic Models of Cortical & Subcortical Dynamics with Social Robotic Applications”DURIP N000140510525: “Robotic Platform for Security and Service Applications”DURIP N000140710704: “Parallel Robotic Brains” HRL0011-09-C-001: “Project Synapse”
A Little NeuroscienceA Little Neuroscience
– NeuronsNeuronsExcitatoryExcitatory
Interneurons Interneurons (inhibitory)(inhibitory)
– ColumnsColumnsHigh connectivity High connectivity within columns.within columns.
Less connectivity Less connectivity across columnsacross columns
– NeuronsNeuronsExcitatoryExcitatory
Interneurons Interneurons (inhibitory)(inhibitory)
– NeuronsNeuronsExcitatoryExcitatory
Interneurons Interneurons (inhibitory)(inhibitory)
A Little NeuroscienceA Little Neuroscience
– ChannelsChannelsPotassium FamilyPotassium Family
– M, A, AHP ChannelsM, A, AHP Channels
– Suppressing behavior Suppressing behavior on parent cellon parent cell
– SynapsesSynapsesAnalog converter of Analog converter of binary spike event.binary spike event.
Contextual filters.Contextual filters.
A Little NeuroscienceA Little Neuroscience
Brain Slice technology from EPFLBrain Slice technology from EPFL
A Little NeuroscienceA Little Neuroscience
Voltage Injection & MeasurementsVoltage Injection & Measurements
A Little NeuroscienceA Little Neuroscience
But we need more than 1But we need more than 1– Currently a 12 clampCurrently a 12 clamp
A Little NeuroscienceA Little Neuroscience
This technology This technology is used to is used to measure the measure the neurons for neurons for reverse reverse engineeringengineering– ConnectivityConnectivity– Voltage Voltage
ResponseResponseYun Wang, Henry Markram, Philip H. Goodman, Junying Ma, Patricia S. Goldma-Rakic “Novel Microcircuit Specializations in the Prefrontal Cortex” Nature Neuroscience 2006 Apr;9(4):534-42.
Early Brain SimulationEarly Brain Simulation
Artificial Neural Networks (ANNs)Artificial Neural Networks (ANNs)– based on the nonlinear propagation of
average activity (analogous to ring rates)
Some Biologically Realistic Simulators– Neuron & Genesis– Very accurate, – But small models (<10 cells)
NCS HistoryNCS History
Version 1:1999 Version 1:1999 – MatlabMatlab– 160-cell, 2-column architecture
Each cell was modeled as a single integrative compartment (point neuron) with a spike mechanism,
– calcium-dependent (AHP) channels, and – voltage-sensitive A and M (muscarinic) potassium
channels.
M.M. Kellog, H.R. Wills, and P.H. Goodman. “A biologically realistic computer model of neocortical associative learning for the study of aging and dementia.” J. Investig. Med., 47(2), February 1999
NCS HistoryNCS History
Version 1b: 1999Version 1b: 1999– Direct translation to C from MatlabDirect translation to C from Matlab
Ali Etazadi-Amoli and Keith WeslowskiAli Etazadi-Amoli and Keith Weslowski
– 24 times faster.24 times faster.– tested on mixed excitatory-inhibitory networks
of up to 1,000 cells.
NCS HistoryNCS History
Version 2: 1999Version 2: 1999– code was then redesigned and rewritten for
distributed processing on an existing 20-cpu cluster (Pentium II).
– Initial trials of this code were performed on cortical networks of 2 to 1,000 cells.
NCS HistoryNCS History
Version 3: 2001Version 3: 2001– completely redesigned using object-oriented design
principles and recoded in C++– objects, such as cells, compartments, channels, and
the like, model the corresponding cortical entities. – The cells, in turn, communicate via messages passed
through synapse objects. – Input parameters allow the user to create many
variations of the basic objects, in order to model measured or hypothesized biological properties.
E. Courtenay Wilson, Phillip H. Goodman, and Frederick C. Harris, Jr. “Implementation of a biologically realistic parallel neocortical-neural network simulator” in Proceedings of the 10th SIAM Conf. on Parallel Process. for Sci. Computing, Portsmouth, Virginia, March 2001.
NCS HistoryNCS History
E. Courtenay Wilson, Frederick C. Harris, Jr., and Phillip H. Goodman. “A large-scale biologically realistic cortical simulator” in Proceedings of SC 2001, Denver, Colorado, November 2001
HardwareHardware
Several grants from DoD for Several grants from DoD for hardwarehardware– 2001 – 30 dual PIII (2GB/core)2001 – 30 dual PIII (2GB/core)– 2002 – 34 dual PIV (2GB/core), 2002 – 34 dual PIV (2GB/core),
Myrinet connectionMyrinet connection– 2004 – 40 dual opterons (2 2004 – 40 dual opterons (2
GB/core)GB/core)– 2007 – 4 16core opteron boxes 2007 – 4 16core opteron boxes
(128GB/box) and 3 24TB disk (128GB/box) and 3 24TB disk arrays, Infiniband connectionarrays, Infiniband connection
Web InterfaceWeb Interface
Goals:Goals:– Allow rapid creation Allow rapid creation
of brain modelsof brain models– Allow remote Allow remote
collaborationcollaboration
Kishor K. Waikul, Lianjun Jiang, E. Courtenay Wilson, Frederick C. Harris, Jr., and Philip H. Goodman, “Design and Implementation of a Web Portal for a NeoCortical Simulator,” in Proceedings of the 17th International Conference on Computers and Their Applications (CATA 2002) pp. 349-353, April 4-6, 2002, San Francisco, CA
Code Optimization & RevisionsCode Optimization & Revisions
Rewrote the input parser Rewrote the input parser Worked on code base Worked on code base – sevenfold sequential speedup over the
version 3 code – added new features while shrinking our code
base by more than 25%.
Added More Biological Parameters.35,000 cells and approximately 6.1 million synapses using 72% of the available 4GB of memory per node.
Code OptimizationCode Optimization
James Frye, James G. King, Christine J. Wilson, and Frederick C. Harris, Jr. “QQ: Nanoscale timing and profiling” In Proceedings of PMEO-PDS, Denver, CO, April 3-8 2005.
Modeling & Neural Engineering
Ripplinger MC, Wilson CJ, King JG, Frye J, Drewes R, Harris FC, and Goodman PH, “Computational Model of Interacting Brain Networks,” Journal of Investigative Medicine, Vol. 52, No. 1, Jan 2004, pp. S155.
Modeling & Neural Engineering
Modeling & Neural Engineering
(bAC)(bAC)
KKAHPAHP
Modeling & Neural Engineering
Romain Brette, Michelle Rudolph, Ted Carnevale, Michael Hines, David Beeman, James M. Bower, Markus Diesmann, Abigail Morrison, Philip H. Goodman, Frederick C. Harris, Jr., Milind Zirpe, Thomas Natschlager, Dejan Pecevski, Bard Ermentrout, Mikael Djurfeldt, Anders Lansner, Olivier Rochel, Thierry Vieville, Eilif Muller, Andrew P. Davison, Sami El Boustani and Alain Destexhe, “Simulation of networks of spiking neurons: A review of tools and strategies,” Journal of Computational Neuroscience Vol. 23, December, 2007, pgs 349-398.
Modeling & Neural Engineering
extensive domain of self-sustained
asynchronous irregular firing
R N
Modeling & Neural Engineering
RAIN Firing CharacteristicsRAIN Firing Characteristics
Robotic/Human Loops
Juan C. Macera, Philip H Goodman, Frederick C. Harris, Jr., Rich Drews, and James B. Maciokas“Remote-Neocortex Control of Robotic Search and Threat Identification,” Robotics and Autonomous Systems, Vol. 46, No. 2, February 2004, pp 97-110.`
Robotic/Human Loops
Qunming Peng “Brainstem: A NeoCortical Simulator Interface for Robotic Studies”, MS Thesis December 2006, University of Nevada,Reno
30
VC
Robotic/Human Loops
Coarse Gabor Filters Coarse Gabor Filters transduced into a transduced into a raster of spikesraster of spikes
31
Engaging & Rewarding
Evocative
Robotic/Human Loops
Robotic/Human Loops
Virtual RobotsVirtual Robots
Philip H. Goodman, Sermsak Buntha, Quan Zou, Sergiu M. Dascalu, :Virtual neurorobotics (VNR) to accelerate the development of plausible neuromorphic brain architectures” Frontiers in Neurorobotics 1(1) pp. 1-7 November 2007.
Other Preliminary WorkOther Preliminary Work
General Movement DisordersGeneral Movement Disorders– ParkinsonsParkinsons– AutismAutism
Goal: model properly Goal: model properly – That way we can study changesThat way we can study changes
Blue BrainBlue Brain
EPFL: IBM BlueGene, 8096-CPU cluster, 22 Trillion Flops
Objectives:Objectives:
– Understand scientific basis for superiority of human intelligence over current machine learning and AI
– Create neurally-based cognitively intelligent systems
– Develop neuromorphic robots which interact with humans
– Complement Neuroscience wet lab and cognitive research
Impacts:Impacts:
– Multi-compartment and million-cell brain models
– New theory of “mesocircuit” to link biology and behavior
– Robotic prototype advanced from CANINE to HUMANOID robot
– Enhanced new field of “virtual neurorobotics” (VNR)
Neuroscience
Present Scope of Work
Modeling & Neural Engineering
Robotic/Human Loops
Goals
A Novel Multi-GPU A Novel Multi-GPU Neural SimulatorNeural Simulator
The proof-of-concept simulation code The proof-of-concept simulation code described here is presented as an described here is presented as an illustration of both the design's scalability illustration of both the design's scalability and performance potential once integrated and performance potential once integrated to the existing environment.to the existing environment.
Written in CUDAWritten in CUDA– With hooks for MPIWith hooks for MPI
A Novel Multi-GPU A Novel Multi-GPU Neural SimulatorNeural Simulator
Supports the same input file as NCSSupports the same input file as NCS– We are adding support for NeuroMLWe are adding support for NeuroML
Once input has been read, Initialization Once input has been read, Initialization begins.begins.
Initialization OverviewInitialization Overview
The neurons are sorted based on the The neurons are sorted based on the number of synaptic connections. number of synaptic connections.
These are then distributed to the These are then distributed to the respective GPUs in a round-robin fashion.respective GPUs in a round-robin fashion.
Then simulation beginsThen simulation begins
Initialization OverviewInitialization Overview
Local Data structures are then created.Local Data structures are then created.
SimulationSimulation
Now that we are finished with setup we go Now that we are finished with setup we go on to the simulation:on to the simulation:
SimulationSimulation
The simulation begins by updating The simulation begins by updating (numerically integrating) the neurons. (numerically integrating) the neurons.
SimulationSimulation
The cell firings are exchanged and we The cell firings are exchanged and we update the cell computationsupdate the cell computations
TestingTesting
Some basic benchmarks were run to Some basic benchmarks were run to illustrate the scalability and functionality of illustrate the scalability and functionality of the design. the design.
The test network was based on the The test network was based on the polychronization models from polychronization models from Izhikevich et Izhikevich et al.al. [10] and Szat [10] and Szat et alet al. [11]. . [11].
TestingTesting
This network utilizes a ratio of four excitatory This network utilizes a ratio of four excitatory neurons to 1 inhibitory neuron.neurons to 1 inhibitory neuron.
– M/N is the probability of connectionsM/N is the probability of connections– M is the number of connections per Neuron. N is M is the number of connections per Neuron. N is
the number of Neurons.the number of Neurons.
ResultsResults
ResultsResults
Small amount of data to transferSmall amount of data to transfer
100,000 neurons with 50 connections per 100,000 neurons with 50 connections per neuron can run at 1.2 times faster than neuron can run at 1.2 times faster than real time.real time.
Conclusions/Future WorkConclusions/Future Work
This was a worst case scenario.This was a worst case scenario.– And it is better than real time.And it is better than real time.
We have already moved to a cluster of We have already moved to a cluster of boxes of GPUs [and have a proposal out boxes of GPUs [and have a proposal out for money to purchase a better/larger GPU for money to purchase a better/larger GPU cluster]cluster]
A Novel Multi-GPU A Novel Multi-GPU Neural SimulatorNeural Simulator
Corey Thibeault, Roger HoangCorey Thibeault, Roger Hoang
Frederick C. Harris, Jr.Frederick C. Harris, Jr.
Brain Computation LaboratoryBrain Computation LaboratoryUniversity of Nevada, RenoUniversity of Nevada, Reno