a novel multi-gpu neural simulator

A Novel Multi-GPU A Novel Multi-GPU Neural SimulatorNeural Simulator

Corey Thibeault, Roger HoangCorey Thibeault, Roger Hoang

Frederick C. Harris, Jr.Frederick C. Harris, Jr.

Brain Computation LaboratoryBrain Computation LaboratoryUniversity of Nevada, RenoUniversity of Nevada, Reno

Reno, NevadaReno, Nevada

University of Nevada, RenoUniversity of Nevada, Reno

Brain Computation LaboratoryBrain Computation LaboratoryUniversity of Nevada, RenoUniversity of Nevada, Reno

in funded collaboration with in funded collaboration with U de Cergy-Pontoise and CNRS, Paris, FranceU de Cergy-Pontoise and CNRS, Paris, France

Brain Mind Institute (Blue Brain Project), EPFL, Lausanne, SwitzerlandBrain Mind Institute (Blue Brain Project), EPFL, Lausanne, Switzerland

Mathias Quoy René Doursat

Henry MarkramJim King

Laurence Jayet Fred Harris, Jr Sergiu Dascalu Director

UNR Brain Computation Lab

Neuroscience

Scope of Work

Modeling & Neural Engineering

Robotic/Human Loops

ONR N000140710018: “Large-Scale Biologically Realistic Models of Cortical & Subcortical Dynamics with Social Robotic Applications”DURIP N000140510525: “Robotic Platform for Security and Service Applications”DURIP N000140710704: “Parallel Robotic Brains” HRL0011-09-C-001: “Project Synapse”

A Little NeuroscienceA Little Neuroscience

– NeuronsNeuronsExcitatoryExcitatory

Interneurons Interneurons (inhibitory)(inhibitory)

– ColumnsColumnsHigh connectivity High connectivity within columns.within columns.

Less connectivity Less connectivity across columnsacross columns

– NeuronsNeuronsExcitatoryExcitatory

Interneurons Interneurons (inhibitory)(inhibitory)

– NeuronsNeuronsExcitatoryExcitatory

Interneurons Interneurons (inhibitory)(inhibitory)

A Little NeuroscienceA Little Neuroscience

– ChannelsChannelsPotassium FamilyPotassium Family

– M, A, AHP ChannelsM, A, AHP Channels

– Suppressing behavior Suppressing behavior on parent cellon parent cell

– SynapsesSynapsesAnalog converter of Analog converter of binary spike event.binary spike event.

Contextual filters.Contextual filters.

A Little NeuroscienceA Little Neuroscience

Brain Slice technology from EPFLBrain Slice technology from EPFL

A Little NeuroscienceA Little Neuroscience

Voltage Injection & MeasurementsVoltage Injection & Measurements

A Little NeuroscienceA Little Neuroscience

But we need more than 1But we need more than 1– Currently a 12 clampCurrently a 12 clamp

A Little NeuroscienceA Little Neuroscience

This technology This technology is used to is used to measure the measure the neurons for neurons for reverse reverse engineeringengineering– ConnectivityConnectivity– Voltage Voltage

ResponseResponseYun Wang, Henry Markram, Philip H. Goodman, Junying Ma, Patricia S. Goldma-Rakic “Novel Microcircuit Specializations in the Prefrontal Cortex” Nature Neuroscience 2006 Apr;9(4):534-42.

Early Brain SimulationEarly Brain Simulation

Artificial Neural Networks (ANNs)Artificial Neural Networks (ANNs)– based on the nonlinear propagation of

average activity (analogous to ring rates)

Some Biologically Realistic Simulators– Neuron & Genesis– Very accurate, – But small models (<10 cells)

NCS HistoryNCS History

Version 1:1999 Version 1:1999 – MatlabMatlab– 160-cell, 2-column architecture

Each cell was modeled as a single integrative compartment (point neuron) with a spike mechanism,

– calcium-dependent (AHP) channels, and – voltage-sensitive A and M (muscarinic) potassium

channels.

M.M. Kellog, H.R. Wills, and P.H. Goodman. “A biologically realistic computer model of neocortical associative learning for the study of aging and dementia.” J. Investig. Med., 47(2), February 1999

NCS HistoryNCS History

Version 1b: 1999Version 1b: 1999– Direct translation to C from MatlabDirect translation to C from Matlab

Ali Etazadi-Amoli and Keith WeslowskiAli Etazadi-Amoli and Keith Weslowski

– 24 times faster.24 times faster.– tested on mixed excitatory-inhibitory networks

of up to 1,000 cells.

NCS HistoryNCS History

Version 2: 1999Version 2: 1999– code was then redesigned and rewritten for

distributed processing on an existing 20-cpu cluster (Pentium II).

– Initial trials of this code were performed on cortical networks of 2 to 1,000 cells.

NCS HistoryNCS History

Version 3: 2001Version 3: 2001– completely redesigned using object-oriented design

principles and recoded in C++– objects, such as cells, compartments, channels, and

the like, model the corresponding cortical entities. – The cells, in turn, communicate via messages passed

through synapse objects. – Input parameters allow the user to create many

variations of the basic objects, in order to model measured or hypothesized biological properties.

E. Courtenay Wilson, Phillip H. Goodman, and Frederick C. Harris, Jr. “Implementation of a biologically realistic parallel neocortical-neural network simulator” in Proceedings of the 10th SIAM Conf. on Parallel Process. for Sci. Computing, Portsmouth, Virginia, March 2001.

NCS HistoryNCS History

E. Courtenay Wilson, Frederick C. Harris, Jr., and Phillip H. Goodman. “A large-scale biologically realistic cortical simulator” in Proceedings of SC 2001, Denver, Colorado, November 2001

HardwareHardware

Several grants from DoD for Several grants from DoD for hardwarehardware– 2001 – 30 dual PIII (2GB/core)2001 – 30 dual PIII (2GB/core)– 2002 – 34 dual PIV (2GB/core), 2002 – 34 dual PIV (2GB/core),

Myrinet connectionMyrinet connection– 2004 – 40 dual opterons (2 2004 – 40 dual opterons (2

GB/core)GB/core)– 2007 – 4 16core opteron boxes 2007 – 4 16core opteron boxes

(128GB/box) and 3 24TB disk (128GB/box) and 3 24TB disk arrays, Infiniband connectionarrays, Infiniband connection

Web InterfaceWeb Interface

Goals:Goals:– Allow rapid creation Allow rapid creation

of brain modelsof brain models– Allow remote Allow remote

collaborationcollaboration

Kishor K. Waikul, Lianjun Jiang, E. Courtenay Wilson, Frederick C. Harris, Jr., and Philip H. Goodman, “Design and Implementation of a Web Portal for a NeoCortical Simulator,” in Proceedings of the 17th International Conference on Computers and Their Applications (CATA 2002) pp. 349-353, April 4-6, 2002, San Francisco, CA

Code Optimization & RevisionsCode Optimization & Revisions

Rewrote the input parser Rewrote the input parser Worked on code base Worked on code base – sevenfold sequential speedup over the

version 3 code – added new features while shrinking our code

base by more than 25%.

Added More Biological Parameters.35,000 cells and approximately 6.1 million synapses using 72% of the available 4GB of memory per node.

Code OptimizationCode Optimization

James Frye, James G. King, Christine J. Wilson, and Frederick C. Harris, Jr. “QQ: Nanoscale timing and profiling” In Proceedings of PMEO-PDS, Denver, CO, April 3-8 2005.

Modeling & Neural Engineering

Ripplinger MC, Wilson CJ, King JG, Frye J, Drewes R, Harris FC, and Goodman PH, “Computational Model of Interacting Brain Networks,” Journal of Investigative Medicine, Vol. 52, No. 1, Jan 2004, pp. S155.

Modeling & Neural Engineering

Modeling & Neural Engineering

(bAC)(bAC)

KKAHPAHP

Modeling & Neural Engineering

Romain Brette, Michelle Rudolph, Ted Carnevale, Michael Hines, David Beeman, James M. Bower, Markus Diesmann, Abigail Morrison, Philip H. Goodman, Frederick C. Harris, Jr., Milind Zirpe, Thomas Natschlager, Dejan Pecevski, Bard Ermentrout, Mikael Djurfeldt, Anders Lansner, Olivier Rochel, Thierry Vieville, Eilif Muller, Andrew P. Davison, Sami El Boustani and Alain Destexhe, “Simulation of networks of spiking neurons: A review of tools and strategies,” Journal of Computational Neuroscience Vol. 23, December, 2007, pgs 349-398.

Modeling & Neural Engineering

extensive domain of self-sustained

asynchronous irregular firing

R N

Modeling & Neural Engineering

RAIN Firing CharacteristicsRAIN Firing Characteristics

Robotic/Human Loops

Juan C. Macera, Philip H Goodman, Frederick C. Harris, Jr., Rich Drews, and James B. Maciokas“Remote-Neocortex Control of Robotic Search and Threat Identification,” Robotics and Autonomous Systems,  Vol. 46, No. 2, February 2004, pp 97-110.`

Robotic/Human Loops

Qunming Peng “Brainstem: A NeoCortical Simulator Interface for Robotic Studies”, MS Thesis December 2006, University of Nevada,Reno

30

VC

Robotic/Human Loops

Coarse Gabor Filters Coarse Gabor Filters transduced into a transduced into a raster of spikesraster of spikes

31

Engaging & Rewarding

Evocative

Robotic/Human Loops

Robotic/Human Loops

Virtual RobotsVirtual Robots

Philip H. Goodman, Sermsak Buntha, Quan Zou, Sergiu M. Dascalu, :Virtual neurorobotics (VNR) to accelerate the development of plausible neuromorphic brain architectures” Frontiers in Neurorobotics 1(1) pp. 1-7 November 2007.

Other Preliminary WorkOther Preliminary Work

General Movement DisordersGeneral Movement Disorders– ParkinsonsParkinsons– AutismAutism

Goal: model properly Goal: model properly – That way we can study changesThat way we can study changes

Blue BrainBlue Brain

EPFL: IBM BlueGene, 8096-CPU cluster, 22 Trillion Flops

Objectives:Objectives:

– Understand scientific basis for superiority of human intelligence over current machine learning and AI

– Create neurally-based cognitively intelligent systems

– Develop neuromorphic robots which interact with humans

– Complement Neuroscience wet lab and cognitive research

Impacts:Impacts:

– Multi-compartment and million-cell brain models

– New theory of “mesocircuit” to link biology and behavior

– Robotic prototype advanced from CANINE to HUMANOID robot

– Enhanced new field of “virtual neurorobotics” (VNR)

Neuroscience

Present Scope of Work

Modeling & Neural Engineering

Robotic/Human Loops

Goals

A Novel Multi-GPU A Novel Multi-GPU Neural SimulatorNeural Simulator

The proof-of-concept simulation code The proof-of-concept simulation code described here is presented as an described here is presented as an illustration of both the design's scalability illustration of both the design's scalability and performance potential once integrated and performance potential once integrated to the existing environment.to the existing environment.

Written in CUDAWritten in CUDA– With hooks for MPIWith hooks for MPI

A Novel Multi-GPU A Novel Multi-GPU Neural SimulatorNeural Simulator

Supports the same input file as NCSSupports the same input file as NCS– We are adding support for NeuroMLWe are adding support for NeuroML

Once input has been read, Initialization Once input has been read, Initialization begins.begins.

Initialization OverviewInitialization Overview

The neurons are sorted based on the The neurons are sorted based on the number of synaptic connections. number of synaptic connections.

These are then distributed to the These are then distributed to the respective GPUs in a round-robin fashion.respective GPUs in a round-robin fashion.

Then simulation beginsThen simulation begins

Initialization OverviewInitialization Overview

Local Data structures are then created.Local Data structures are then created.

SimulationSimulation

Now that we are finished with setup we go Now that we are finished with setup we go on to the simulation:on to the simulation:

SimulationSimulation

The simulation begins by updating The simulation begins by updating (numerically integrating) the neurons. (numerically integrating) the neurons.

SimulationSimulation

The cell firings are exchanged and we The cell firings are exchanged and we update the cell computationsupdate the cell computations

TestingTesting

Some basic benchmarks were run to Some basic benchmarks were run to illustrate the scalability and functionality of illustrate the scalability and functionality of the design. the design.

The test network was based on the The test network was based on the polychronization models from polychronization models from Izhikevich et Izhikevich et al.al. [10] and Szat [10] and Szat et alet al. [11]. . [11].

TestingTesting

This network utilizes a ratio of four excitatory This network utilizes a ratio of four excitatory neurons to 1 inhibitory neuron.neurons to 1 inhibitory neuron.

– M/N is the probability of connectionsM/N is the probability of connections– M is the number of connections per Neuron. N is M is the number of connections per Neuron. N is

the number of Neurons.the number of Neurons.

ResultsResults

ResultsResults

Small amount of data to transferSmall amount of data to transfer

100,000 neurons with 50 connections per 100,000 neurons with 50 connections per neuron can run at 1.2 times faster than neuron can run at 1.2 times faster than real time.real time.

Conclusions/Future WorkConclusions/Future Work

This was a worst case scenario.This was a worst case scenario.– And it is better than real time.And it is better than real time.

We have already moved to a cluster of We have already moved to a cluster of boxes of GPUs [and have a proposal out boxes of GPUs [and have a proposal out for money to purchase a better/larger GPU for money to purchase a better/larger GPU cluster]cluster]

A Novel Multi-GPU A Novel Multi-GPU Neural SimulatorNeural Simulator

Corey Thibeault, Roger HoangCorey Thibeault, Roger Hoang

Frederick C. Harris, Jr.Frederick C. Harris, Jr.

Brain Computation LaboratoryBrain Computation LaboratoryUniversity of Nevada, RenoUniversity of Nevada, Reno