brain-like software architecture confessions of an ex-neuroscientist bill softky
TRANSCRIPT
![Page 1: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/1.jpg)
Brain-like software architecture
Confessions of an ex-neuroscientist
Bill Softky
![Page 2: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/2.jpg)
Which comes first: the problem or the solution?
• Reverse engineering starts with hardware, works backward
• Usually only succeeds if problem is understood
• “Forward” software engineering starts with the problem, and saves hardware for last
![Page 3: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/3.jpg)
“Forward” software engineering
Question UPS example e*Trade example
1 What is the “truth” out there?
Package en route John Doe, investor
2 How does input data approximate the truth?
Tracking #, destination Customer #portfolio ownedtrades made
3 What do we want to do with the data?
Deliver to next station; display web tracking page
Log in, display net worth, trade stocks
4 What architecture? Client-server, relational DB
Distributed serversseparate web front-end
5 What implementation? Oracle, C++, cgi-script web, Solaris machines
MySQL, Java server pages, Linux/Intel clusters
![Page 4: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/4.jpg)
“Reverse” engineeringQuestion Neocortex vision Neocortex audio
? What is the “truth” out there?
Moving objects Speech
? How does input data approximate the truth?
Retinal “pixels”: contours, color, correlation, disparity
Sound pressure waveforms: frequencies, stereo, echoes
? What do we want to do with the data?
Find what and where an object is
Figure who/where is talking, what they're saying, what they mean
1-2 What architecture? Cortical columns, attractors, spikes, associative
1-2 What implementation? Hebbian synapses, integrate-and-fire, shunting inhibition
From an engineering perspective, this is nuts!
![Page 5: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/5.jpg)
Initial goals here
• Input: we need a generic description of sensory input (at least audio & visual)
• Processing: speculate on generic, modular processing “API” which can untangle those correlations
• No neurons, synapses, spikes…yet.
![Page 6: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/6.jpg)
Simple “truth” tangled inputs
Hypothesis: each entangling transformation is fairly simple
![Page 7: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/7.jpg)
Stepwise decorr untangled truth
Hypothesis: a sequence of similar compressions will yield useful representation
![Page 8: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/8.jpg)
First toy problem: cocktail party with echoes
• Multiple independent speakers• Multiple “ears” (mics)• Multiple echoes/amplitudes for each speaker/mic
combo• Echo patterns constrained (3-D) and unchanging
Try to remove echoes and separate speakers (our brains can do this...)
![Page 9: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/9.jpg)
Echo kernels = location info
M-a
S 1
S 2
M-c
M-b=
M-a
M-b
M-c
“pure signals” Echo kernels, transfer
functions, “maps”
“entangled signals”
3 x 10kHz(x,y,z) static2 x 10kHz+
+
+
![Page 10: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/10.jpg)
Second toy problem: video
•Moving “objects” (simple shapes)
•Constant velocity
•Spatiotemporal pixel pattern is just echoes from t=0 at center
![Page 11: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/11.jpg)
Echo kernels = location/shape/velocity
(0,0)
(4,4)
(0,1)=
“Time at center”
Spatio-temporal Pixel responses
“entangled signals”
100 x 1 kHz{v, } semi-static
1 kHz
+
.
.
.
(0,0)
(0,1)
(4,4)
+
+
![Page 12: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/12.jpg)
Generic entanglement
=
Very few independent pure signals
to track
Echo kernels in low-dim
subspace give persistent structure
many entangled, correlated,
high-bw signals as
inputs
.
.
. .
.
.
. .
![Page 13: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/13.jpg)
Recap: echo-entanglement as a generic perceptual problem
• Very similar to early vision
• Just like audio echo-removal
• Structured “echoes” carry near-static info
• Associative memory and vector quantization are special cases
![Page 14: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/14.jpg)
How to dis-entangle?
• Want to reveal original signals and structures
• Problem is hard (unsolved!)So…– Skip the mere algorithms– Skip the neurons and biology– Focus on a module’s inputs & outputs– Try to make modules work together
![Page 15: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/15.jpg)
What would one disentangling module do?
• Note separate timescales:– Many channels of high-BW input– 1-3 indep channels med-BW output (time blurring)– Many channels near-static output & input
• Learn correlations (echoes) in input• Find low-dim subspace for echos (e.g. {x,y,z}, or
{v, })• Reconstruct inputs all at once (batch)• Minimize reconstruction error
(Assume typically 1 pure signal max during learning)
![Page 16: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/16.jpg)
Basic disentangling module
T=-500 +100, coarseFloat outputs
Float inputs
Decorrelation &vector quantization
Reconstruction &prediction
x,y,z Pure signal
T=-500 +100, fine
e.g. for cocktail-party decorrelation
“mics”
“now”
“now”
![Page 17: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/17.jpg)
Add multiple, independent outputs
• Multiple speakers/objects multiple outputs
• Each output represents one object (max 3)
• Output streams and mappings are independent
• An even harder disentangling task
• (complications too!....)
![Page 18: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/18.jpg)
Module with multiple outputs
X1,y1,z1
X2,y2,z2
X3,y3,z3
Speaker 1
Speaker 2
Speaker 3
![Page 19: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/19.jpg)
Add confidence estimates (sigmas)
• Disentangling is already a statistical-estimation task
• Confidence estimates come for free during reconstruction
• Propagate inputs’ sigmas forward
• Create output sigmas based on input sigmas and reconstruction error
![Page 20: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/20.jpg)
Module with sigmas
![Page 21: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/21.jpg)
Add layers
• Pure signal outputs become inputs to next layer• Many modules below feed each module above• Maybe, modules below can feed more than one
above• Whole upper layer uses longer and coarser
timescale• Stackable indefinitely• Top layers have huge input range, long memory,
broad abstractions
![Page 22: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/22.jpg)
Modules in layers
T=-1000200
-500100
![Page 23: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/23.jpg)
Add feedback
• Upper layer reconstructions provide estimates to lower modules (might help, can’t hurt)
• Near-static channels provide cheap “prediction” of input interrelations
• Update all estimates frequently
• Predicted pure signals could help reconstruction below
![Page 24: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/24.jpg)
Feedback between modules
![Page 25: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/25.jpg)
Open problems
• How do do the decompression?– Iterative? Monte Carlo? Low-dim subspace?
• Multiple objects/pure signals:– Deciding how many objects from a module– “binding” problem across modules– Which goes with which?– Layers 2-N need “clones,” one clone per extra
object
![Page 26: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/26.jpg)
Summary: generic sensory model
• Assume inputs result from cascading a simple entangling transformation
• Entangling transformation is cocktail-party with echoes
=
![Page 27: Brain-like software architecture Confessions of an ex-neuroscientist Bill Softky](https://reader030.vdocuments.mx/reader030/viewer/2022033107/56649e615503460f94b5cc54/html5/thumbnails/27.jpg)
Summary: stackable disentangling modules
• Assume one layer of disentangling can be learned and done somehow
• Separate time-series from static echo-kernel structure• Disentangle time-series in batches• Use reconstructions for error-checking and feedback• Propose “API” by which such modules can interact to
solve multi-scale, multi-sensory problems