Brain-Related Computing beyond Moore’s Law

Thomas Sterling

Professor of Informatics and Computing

Chief Scientist and Associate DirectorCenter for Research in Extreme Scale Technologies (CREST)

School of Informatics and Computing

Indiana University

February 24, 2014

The Human Brain• 100 billion neurons (Felix says “89”)• 10,000 synaptic junctions per neuron• < 1 KHz pulse repetition rate• 1400 cc volume• 2-4 Exa-Ops• 20 Watts power consumption• Hierarchical architecture• Neuro-function

– Analog summation in time and space– Time varying comparison threshold– Pulsed signal propagation – Modulated synaptic connections

Brain-inspired• Consciousness

– Identity– Self-aware– perception

• Thinking– Recognition of patterns, objects, people, concepts– Planning, decision making, learning, action, inference– Intelligence

• Extreme complexity– 70 neurons/cubic mm– 20 Atto Joules per operation– Localized asynchronous functions– Emergent operational behavior

Neuron Function

Neuro-Inspired• Neurons embody state and function– State both internal and at interfaces– Function both internal and at interface

• Neurons perform localized operations– Ops to first order are internal and isolated– complex

• independent in time– Asynchronous and slow, event driven

• Enormous connectivity– On the order of 10,000 input junctions– On the order of 10,000 output contacts

• Limited connectivity– Only 0.00001% of system– Nonlocal

• Operates in a context for emergent behavior– Execution model, graph structured subsystems

• Ultra low power

Analog of Neuro-Inspired Computational Element

• Vannevar Bush Machine– aka Analog Computer

• Solves sets of first-order differential equations• Core device satisfies Neuro-Inspired definition

– Internal state and complex functionality– Emergent behavior in context of total system of like devices

• Many technologies– Mechanical, vacuum tubes, transistors, op amps

• One execution model– Governing abstraction

Artificial Neural Networks (ANN)• An interconnected network of functional elements that reflect

some properties of neurons and their interconnections• Derived functions through learning from observations

– Training set– Adaptive weights

• Applications– Identification and pattern recognition

• Image, signature, character recognition, ATR

– Data mining, clustering, classification– Control of real time mechanical systems– Diagnosis– Robotics

Expert Systems• Also “production systems” or “rule-based systems”• Makes decisions in the presence of a set of external and

internal state• Comprises a set of rules or “productions” of two parts:

– Condition: “if” – determines satisfiability– Action: “then” – updates system state

• Productions are executed– When conditions are satisfied– Conflict resolution and prioritization– RETE algorithm optimizes performance

• Languages: OPS-5, ART, Keys, …

Two Brain-Inspired Projects

• CRIS– Cognitive Real-time

Interactive System– High-level abstraction– Brain Inspired:

• Intelligence• Self-aware

– Emergent behavior• Concurrent agents• Objective function-driven• Engages neighborhood of

interrelated objects• Responsive in real-time

• CCA– Continuum Computer

Architecture– Low-level abstraction– Brain Inspired:

• Fine grain functionality• Massive co-existing elements• Distributed control• Quasi independent operation

– Emergent behavior• Localized function & state• Event-driven• Logically highly interconnected

Relationship within System Structure

CRIS

CCA

Graph API

Runtime System

Operating System

Users

Logic/Store/Comm Devices

CCA Fundamental Concepts

• Peak Resources– X100 arithmetic/logic units– massive bisection and memory bandwidth

• Merge functionality into a single simple block (fonton)– communications– memory– logic

• Global cellular structure, 2.5 or 3-D mesh• Data/instruction structures distributed across fontons• Local rules determine fonton operations – cellular automata• Synergism among fontons yields emergent global behavior of general

parallel computing model• Natural graph structure store

CCA Structure: Fonton• Small block of fully associative tagged memory• Basic logical and arithmetic unit• Instruction register directs control to set data paths• Nearest neighbor communications with switching

Inst. Reg.

Control

ALU

Assoc. Memory

PIM

PIM

PIM

PIM

PIM

PIM

PIM

PIM

PIM

PIM

PIM

PIM

PIM PIM PIM

PIM

PIM

PIM

PIM

PIM

cell

cell

cell

cell

cell

cell

cell

cell

cell

cell

cell

cell

cell cell cell

cell

cell

cell

cell

cell

Emulates Neuron Structures in 3-D

• Emulates Neuron Structures with Hardware in Software– Localized functionality achieved by fontons or groups of fontons– Packet switching through fonton pathways achieves synaptic broadcast– In aggregate builds up dynamic irregular time-varying graph structures

• Data migration– objects are copied to adjacent fontons– copying exploits fine grain data parallelism, even for irregular data

structures– objects may transit by means of wormhole routing

• Data objects are virtual named– Address translation an intrinsic function

CRIS Motivation• Inspired to mechanize key properties of mental condition: thinking• Machine Intelligence• Question: How big a machine?

– Derive a lower bound of required resources• OPS, • storage, • communication, • energy

• Premise: “Intelligence”– Independent of the mammalian mental condition– An algorithm– Does not embody many properties of mental condition

• Determine balance of resource utilization to exhibit intelligence

Abstract Architecture: CRIS

17

Abstract Architecture: Knowledge State

18

Abstract Architecture: Objective Function

19

Execution Model Phase Change• Guiding principles for system design and operation

– Semantics, Mechanisms, Policies, Parameters, Metrics– Driven by technology opportunities and challenges– Historically, catalyzed by paradigm shift

• Decision chain across system layers– For reasoning towards optimization of design and

operation• Essential for co-design of all system layers

– Architecture, runtime and OS, programming models– Reduces design complexity from O(N2) to O(N)– Enables holistic reasoning about concepts and tradeoffs

• Empowers discrimination, commonality, portability– Establishes a phylum of HPC class systems

Vector Model1975

SIMD-array Model1983

CSP Model1991

SIF-MOE Model1968

Von Neumann Model1949

? Model2020

ParalleX Execution Model- A Virtual Brain?• Lightweight multi-threading

– Divides work into smaller tasks– Increases concurrency

• Message-driven computation– Move work to data– Keeps work local, stops blocking

• Constraint-based synchronization– Declarative criteria for work– Event driven– Eliminates global barriers

• Data-directed execution– Merger of flow control and data

structure• Shared name space

– Global address space– Simplifies random gathers

Isomorphism between ParalleX and Neuron Precepts• Compute Complexes (instantiated threads)

– Neurons – Performs local operations

• Parcels– Connectivity – Event driven

• Local Control Objects– Defines satisfiability constraints for firing – Manages asynchrony

• Global Address Space & hierarchical name space (processes)– Brain– Provides single global context

• Can be used directly for parallel emulation in near-term

Summary Conclusions

• Dennard scaling and Moore’s Law are at an end• Neuro-inspired computing and computational elements

– Power perspective in expanding beyond conventional practices– A long history of prior art

• New technologies advancing new opportunities• Execution models are fundamental abstraction required for

success of any general or special purpose computing strategy• Neo-digital era or non-von Neumann architectures• Revolutionary

– Isomorphic computing– Neuro-Inspired computing