biology inspired approximate data representation for signal processing, soft computing and
DESCRIPTION
Biology Inspired Approximate Data Representation for Signal Processing, Soft Computing and Control Applications. Emil M. Petriu, Dr. Eng., FIEEE School of Information Technology and Engineering University of Ottawa Ottawa, ON., K1N 6N5 Canada http://www.site.uottawa.ca/~petriu. - PowerPoint PPT PresentationTRANSCRIPT
Biology Inspired Approximate Data Representation for
Signal Processing, Soft Computing and Control Applications
Emil M. Petriu, Dr. Eng., FIEEESchool of Information Technology and Engineering
University of OttawaOttawa, ON., K1N 6N5 Canada
http://www.site.uottawa.ca/~petriu
WISP’2007 - IEEE Int. Symposium on Intelligent Signal Processing, Alcalá de Henares, Spain, 3-5 Oct.2007
Abstract
This paper reviews basics, similarities, and applications of two biology inspired approximate data representation modalities: stochastic data representation and fuzzy linguistic variables.
Stochastic Data Representation
Biological Neurons
Incoming signals to a dendrite may be inhibitory or excitatory.The strength of any input signal is determined by the strength ofits synaptic connection. A neuron sends an impulse down its axonif excitation exceeds inhibition by a critical amount (threshold/offset/bias) within a time window (period of latent summation).
Biological neurons are rather slow (10-3 s) when compared with the modern electronic circuits. ==> The brain is faster than an electronic computer because of its massively parallel structure. The brain has approximately 1011 highly connected neurons (approx. 104 connections per neuron).
Dendrites carry electrical signals in into the neuron body. The neuron body integrates and thresholds the incoming signals.The axon is a single long nerve fiber that carries the signal fromthe neuron body to other neurons. A synapse is the connection between dendrites of two neurons.
Memories are formed by the modification of the synaptic strengths which can change during the entire life of the neural systems.
Body
Axon
Dendrites
Synapse
Looking for a model to prove that algebraic operations with analog variables can be performed by logic gates, von Neuman advanced in 1956 the idea of representing analog variables by the mean rate of random-pulse streams [J. von Neuman, “Probabilistic logics and the synthesis of reliable organisms from unreliable components,” in Automata Studies, (C.E. Shannon, Ed.), Princeton, NJ, Princeton University Press, 1956].
The “random-pulse machine” concept, [S.T. Ribeiro, “Random-pulse machines,” IEEE Trans. Electron. Comp., vol. EC-16, no. 3, pp. 261-276,1967], a.k.a. "noise computer“, "stochastic computing“, “dithering” deals with analog variables represented by the mean rate of random-pulse streams allowing to use digital circuits to perform arithmetic operations. This concept presents a good tradeoff between the electronic circuit complexity and the computational accuracy. The resulting neural network architecture has a high packing density and is well suited for very large scale integration (VLSI).
FS+VFS-V
FS FSXQ
p.d.f. of VR
1
2 FS.
-FS
+FS
1
V
X
0-1
VRQ1-BIT QUANTIZER
X-FS
+FS
XQ
X
0
1
-1
XQ
CLOCK CLK
VRP
ANALOG RANDOM SIGNAL GENERATOR
-FS +FS0R
p(R)1
2 FS
++
VRV
R
Analog/Random-Pulse Conversion
The deterministic component of the random-pulse sequence, conveniently unbiased and rescaled for this purpose to take values +1 and -1 (instead of 1 and respectively 0) , can be calculated as a statistical estimation from the quantization diagram:
E[VRP] = (+1) .p[VR>=0] + (-1) .p[VR<0] = p(VRP) - p(VRP’) = (FS+V)/(2.FS) - (FS-V)/(2.FS )= V/FS;
This finally gives the deterministic analog value V associated with the binary VRP sequence:
V = [p(VRP) - p(VRP')] . FS ; where the apostrophe ( ' ) denotes a logical inversion.
Random-Pulse/Digital Conversion
V*
N-bit SHIFTREGISTER
Up
(n+1) bit UP/DOWNCOUNTER
Down
VRP
Clock
V*
N-bit SHIFTREGISTER
N-bit SHIFTREGISTER
Up
(n+1) bit UP/DOWNCOUNTER
Down
VRP
Clock
Random-pulse/digital converter using the moving average algorithm .
;**
);(11*
01
1
11
NVRPVRPVV
VRPVRPN
VRPN
V
NNN
N
N
ii
N
iiN
32 266 5003.2
1
1.2
x2is
x2ditis
x2RQis4
2
dZis
dHis
dLis
MAVx2RQis
is
(1) Analog Input (2) Analog Input + Dither Noise
(4) Estimation of the Analog Input recovered by the moving-average “Random-Pulse/Digital Conversion”
(3) Random-Pulse Sequence produced by the “Analog/Random-Pulse Conversion”
Sample Index32 266 5003.2
1
1.2
x2is
x2ditis
x2RQis4
2
dZis
dHis
dLis
MAVx2RQis
is
(1) Analog Input (2) Analog Input + Dither Noise
(4) Estimation of the Analog Input recovered by the moving-average “Random-Pulse/Digital Conversion”
(3) Random-Pulse Sequence produced by the “Analog/Random-Pulse Conversion”
Sample Index
Analog/random-pulse and random-pulse/digital conversion of a “step” input signal
1 OUT_OF mDEMULTIPLEXER
RANDOM NUMBERGENERATOR
S1SjSm
CLK
Y = (X1+...+Xm)/m
y
x1
xj
xm
X1
Xj
Xm
Random-Pulse Addition
32 266 5008.2
3.5
1.2
x1is
x1RQis
41.5
MAVx1RQis
dZ1is
x2is 3
x2RQis
44.5
MAVx2RQis 3
dZ2is
x1is x2is 6
SUMRQXis
47.5
MAVSUMRQXis 6
dZSis
dHis
dLis
is
Random-pulse addition
32 144 2569.2
4
1.2
x1is
x1ditis
x1RQis
42
dZis
dHis
dLis
w1is 3.5
dZis 3.5
W1is4
5
x1W1RQis4
6.5
MAVx1W1RQ is 8
dZis 8
is
Random-pulse multiplication
Generalized b-bit analog/stochastic-data conversion and its quantization characteristics
VR
V
RVRQ
CLOCKCLK
VRP
b-BIT
QUANTIZER
X XQ
ANALOG RANDOM
SIGNAL
GENERATOR
-D/2 0
R
p(R)
1/D
+D/2
+
+
.(k+0.5) D(k-0.5) D.
XQ
X
k
k+1
k-1
0
b D.
1/ Dp.d.f.
of VR
D /2D /2
b D. (1- b ) D.
.V= (k- b ) D
k D.
Stochastic Data Representation
The ideal estimation over an infinite number of samples of the stochastic data sequence VRD is:
E[VRD] = (k-1). p[(k-1.5) ≤ VR < (k-0.5) ] + k . p[(k-0.5) ≤ VR< (k+0.5)] = (k-1) . + k . (1-) = k -
The estimation accuracy of the recovered value for V depends on thequantization resolution, the finite number of samples that are actually averaged, and on the statistical properties of the analog dither.
Quantization levels
Relative mean square error
2 72.23
3 5.75
4 2.75
... ...
8 1.23
... ...
analog 1
0 10 20 30 40 50 60 700
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
Moving average window size
Mea
n sq
uare
err
or
1-Bit
2-Bit
Mean square errors function of the moving average window size
RANDOMNUMBERGENERATOR
1-OUT OF-mDEMULTIPLEXER
...
...
CLK
... ... Sm
S1
Si
mX
1X
iX Z = (X +...+X )/mmi
b
b
b
b
b
b
b
Stochastic-data addition.
2-bit stochastic-data multiplier.
YX
101
-110
00010-1
-110
101
000011
000
000
000000
100100
-110
XLSB
XMSB Z
LSB
ZMSB
YLSB
YLSB
0 100 200 300 400 500-2
-1
0
1
2multiplication
0 100 200 300 400 500-2
-1
0
1
2
weightinput
product
Example of 2-bit stochastic-data multiplication.
Correlator Architectures Using Stochastic Data
Representation
V1 VRD1(n.t)
DELAY LINE
MULTIPLIER
STOCHASTIC DATA / DIGITAL
V2
ANALOG / STOCHASTIC DATA
VRD2((n-r).t ) VRD2(n.t )
VRD1(n.t) . VRD2((n-r).t)
CORv1v2 (r.t)
ANALOG / STOCHASTIC DATA
))((2)(11)(1
021 tknVtnV
NtkCOR
N
nVV
Parallel architecture of a 4-point correlator using random-pulse data representation.
for k = 0,1,2,3. ;))(()(1)(1
0tknytnx
NtkCOR
N
nxy
Each calculated correlation point has a 8-bit resolution. In order to reduce the number of interface lines, the 8-bit wide outputs of thefour random-pulse/digital converters are multiplexed. When more modules are connected in larger structures the 1-bit XRQ input lines of all modules are connected together while the 1-bit YRQ output of each module is connected to the 1-bit YRQ input of the next module creating a longer delay line.
Autocorrelation function of a sinusoid calculated by a 32-point parallel random-pulse correlator with a 8-bit/point resolution.
Correlation function of two sinusoids with the same frequency but different amplitudes and phases, calculated by a 32-point parallel random-pulse
correlator with a 8-bit/point resolution.
Correlation function between a sinusoid and another sinusoid of the same frequency but corrupted by white noise, calculated by a32-point parallel random-pulse correlator with 8-bit/point resolution.
Neural Network Architectures Using
Stochastic Data Representation
F
Y = F [ w X ]j=1
m.
j iij
SYN
APS
E
SYN
APS
E
SYN
APS
E
. . .. . . X mX 1 X i
w mjw ijw 1j
Neuron structure
SYNAPSE ADDRESS DECODER
S mpS ijS 11
N-STAGE DELAY LINE
......
wij
DT = w Xij ij.
i
SYNAPSE
MODE
DATIN SYNADD X i
MULTIPLICATION
b
b
b
b
b
... ...
RANDOM-DATA ADDER
DT mj DT ij DT 1j
RANDOM-DATA / DIGITAL
CLK
DIGITAL / RANDOM-DATA
ACTIVATION FUNCTION F
Y = F [ w X ] j j=1
m .
i ij
Multi-bit stochastic-data implementation of a neuron body.
Multi-bit stochastic-data implementation of a synapse
Auto-associative memory NN architecture
P1, t1 P2, t2 P3, t3
Training set
30
P
30x1
30x30
n
30x1
a
30x1W
)*hardlim( PWa
Recovery of 30% occluded patterns
Fuzzy Logic Control
Pioneered by Zadeh in the mid ‘60s fuzzy logic provides the formalism for modeling the approximate reasoning mechanisms specific to the human brain.
“In more specific terms, what is central about fuzzy logic is that, unlike classical logical systems, it aims at modeling the imprecise modes of reasoning that play an essential role in the remarkable human ability to make rational decisions in an environment of uncertainty and imprecision. This ability depends, in turn, on our ability to infer an approximate answer to a question based on a store of knowledge that is inexact, incomplete, or not totally reliable.” [ “Fuzzy Logic,” IEEE Computer Magazine, April 1988, pp. 83-93: ]
Fuzzy Logic
Fuzzy Logic Control
ANALOG (CRISP) -TO-FUZZY INTERFACE FUZZIFICATION
FUZZY-TO- ANALOG (CRISP) INTERFACE DEFUZZIFICATION
SENSORS ACTUATORS
INFERENCE MECHANISM (RULE EVALUATION)
FUZZY RULE BASE
PROCESS
The basic idea of “fuzzy logic control” (FLC) was suggested by L.A. Zadeh, “A rationale for fuzzy control,” J. Dynamic Syst. Meas. Control, vol.94, series G, pp.3-4,1972.
FLC provides a non analytic alternative to the classical analytic control theory. ==> “But what is striking is that its most important and visible application today is in a realm not anticipated when fuzzy logic was conceived, namely, the realm of fuzzy-logic-based process control,” [L.A. Zadeh, “Fuzzy logic,” IEEE Computer Mag., pp. 83-93, Apr. 1988].
Early FLCs were reported by Mamdani and Assilian in 1974, and Sugeno in 1985.
INPUT
OUTPUT
x*
y*
Classical control systems are based on a detailed I/O function OUTPUT= F (INPUT) mapping each high-resolution
quantization interval of the input domain into a high-resolution quantization interval of the output domain
Fuzzy control is based on a much simpler functional description of the desired I/O behavior mapping each low-resolution quantization interval of the input domain into a low-resolution quantization interval of the output domain
INPUT
OUTPUT
y*
x*
Defuzzific
ation
Fuzzification
Membership functions and the quantization characteristics for a 3-set (N, Z, and P) fuzzy partition of the domain where the analog variable x is defined. XFQ are the crisp analog values recovered after a defuzzification of the fuzzy converted value x. It also shows the truncated information XQ recovered from an A/D converter with 3 quantization levels defined over the same domain for x.
The key benefit of FLC is that the desired system behavior can be described with simple “if-then” relations based on very low-resolution models able to incorporate empirical engineering knowledge. FLCs have found many practical applications in the context of complex ill-defined processes that can be controlled by skilled human operators: water quality control, automatic train operation control, elevator control, etc.,
Fuzzy Controller for Truck and Trailer Docking
DOCK
d
NL NM NS PS PM PLZE
AB(
[deg.] -110 -95 -35 -20 -10 0 10 20 35 95 110
NL NM NS PS PM PLZE
GAMMA
[deg.] -85 -55 -30 -15 -10 0 10 15 30 55 85
NEAR LIMITFAR
DIST ( d )
[m] 0.05 0.1 0.75 0.90
INPUT MEMBERSHIP FUNCTIONS
SPEED
[ % ] 16 24 30
STEER (
[deg.] -48 -38 -20 0 20 38 48
LH LM LS ZE RS RM RH
SLOW MED FAST
REV FWD
DIRN
[arbitrary] - +
OUTPUT MEMBERSHIP FUNCTIONS
“STEER / DIRN” RULE BASE
LM/F LS/F RS/F RM/F
LM ZE RM
RM
RH
RH
RHRM RH
RH/F
RH/F
RH/F
RH/F
RH/F
RH/FRH/FRH/FRM/F
LS/R RS/R
RS/R
RS/R RM/R
ZE/R
ZE PS PM PL
LH/F LH/F LH/F
LH
LH
LH
LM
LM
LH
ZE
LH/F
LH/F
LH/F
LH/F
LH/F
LM/F RS/FLS/F
LM/R
LS/R
LS/R
NM
NL
NS
ZE
PS
PM
PL
NL NM NS
GAMMA ()
AB( )
F-R F-R F-R F-R F-R
F-R
F-R
F-R
F-R F-R F-R F-R F-R
F-R
F-R
F-R
There is a hysteresis ring around the center of the rule base table for the DIRN output. This means that when the vehicle reaches a state within this ring, it will continue to drive in the same direction, F (forward) or R (reverse), as it did in the previous state outside this ring.
A hysteresis was purposefully introduced to increase the robustness of the FLC.
DEFFUZIFICATION
The crisp value of the steering angle is obtained by the modified “centroidal” deffuzification (Mamdani inference):
207
47
20
63
63
0
I/O characteristic of th Fuzzy Logic Controller for truck and trailer docking.
LH . LH + LM . LM + LS. LS + ZE . ZE
+ RS . RS + RM . RM + RH . RH ) / (LH + LM + LS + ZE + RS + RM + RL)
There is tenet of common wisdom that FLCs are meant to successfully deal with uncertain data. According to this, FLCs are supposed to make do with “uncertain” data coming from (cheap) low-resolution and imprecise sensors. However, experiments show that the low resolution of the sensor data results in rough quantization of of the controller's I/O characteristic:
207
47
20
63
63
0
0
16
16
0
207
47
1disp
4-bit sensors
7-bit sensors
I/O characteristics of the FLC for truck & trailer docking for 4-bit sensor data () and 7-bit sensor data.
STOCHASTIC-DATA FUZZY LOGIC CONTROLLERS
d
Loading Dock
( , )x y Front Wheel
Back Wheel
(0,0)x
y
The truck backing-up problem
Design a Fuzzy Logic Controller (FLC) able to back up a truck into a docking station from any initial position that has enough clearance from the docking station.
0 4 5 15 20-4-5-15-20-50 50x-position
900 100080060030000-90 27001200 1500 1800
truck angle
00-250-350-450 250 350 450
LE LC CE RC RI
RB RU RV VE LV LU LB
NL NM NS ZE PS PM PL
steering angle
0.0
1.0
1.0
0.0
Membership functions for the truck backer-upper FLC
PS
NS
NM
NM
NL
NL
NL
PM PM
PM
PL PL
NL
NL
NM
NM
NS
PS
NM
NM
NS
PS
NM
NS
PS
PM
PM
PL
NS
PS
PM
PM
PL
PL
RL
RU
RV
VE
LV
LU
LL
LE LC CE RC RI
x
ZE
1 2 3 4 5
6 7
18
31 35343332
30
The FLC is based on the Sugeno-style fuzzy inference.
The fuzzy rule base consists of 35 rules.
0 10 20 30 40 50 60 70-40
-30
-20
-10
0
10
20
30
Time (s)
[deg]
0 10 20 30 40 50 60-50
-40
-30
-20
-10
0
10
20
30
40
Time (s)
[deg]
Time diagram of digital FLC's output during a docking experiment when the input variables, j and x are analog and respectively quantizied with a 4-bit bit resolution
A/D
A/D
Dither
Dither
Low-PassFilter
Low-PassFilter
DigitalFLC
Analog Input
Analog Input
DitheredAnalog Input
High ResolutionDigital Outputs
DitheredDigital Input
DitheredDigital Input
High ResolutionDigital Input
High ResolutionDigital Input
DitheredAnalog Input
FLC architecture using 4-bit stochastic data representation with low-pass filters placed immediately after the input A/D converters
A/D
A/D
Dither
Dither
Low-PassFilter
Low-PassFilter
DigitalFLC
AnalogInput
AnalogInput
DitheredAnalog Input
High ResolutionDigital Output
Low-ResolutionDithered DigitalInput
High ResolutionDigital Output
Low-ResolutionDithered DigitalInput
DitheredAnalog Input
It offers a better performance than the previous one because a final low-pass filter can also smooth the non-linearity caused by the min-max composition rules of the FLC.
FLC architecture using 4-bit stochastic data representation with low-pass filters placed at the FLC’s outputs.
Time diagram of the stochastic FLC's output during a docking experiment when 4-bit A/D converters are used to quantize the dithered inputs and the
low-pass filter is placed at the FLC's output
0 10 20 30 40 50 60 70-50
-40
-30
-20
-10
0
10
20
30
Q [deg]
Time (s)
-50 500
10
20
30
40
50
X
Y
(a)
(b)
(c)
[dock]
initial position
(-30,25)
0
Truck trails for different FLC architectures: (a) analog ; (b) digital without dithering; (c) stochastic data representation with uniform
dithering and 20-unit moving average filter
Stochastic FLCDigital FLC
Analog FLC
Conclusions
Due to its relatively low hardware complexity and high internal noise immunity, the von Neumann stochastic data representation represents an attractive alternative to the analog and high resolution digital data processing techniques for many statistical signal processing and soft computing applications.
Because of the smooth linear transitions in the membership of overlapping fuzzy sets, the fuzzy partition of the analog FLC inputs will not introduce any quantization noise. However, digital FLCs cannot make do with low-resolution data. It is shown that dithering can offer a solution to significantly improve the resolution of the reduced word-length digital FLCs.
• E. Pop, E. Petriu, “Influence of Reference Domain Instability Upon the Precision of Random Reference Quantizer with Uniformly Distributed Auxiliary Source,” Signal Processing, North Holland, Vol. 5, 1983, pp. 87-96.• G. Eatherley, Autonomous Vehicle Docking Using Fuzzy Logic, M.A.Sc. Thesis, 1994. • G. Eatherley, E.M. Petriu, "A Fuzzy Controller for Vehicle Rendezvous and Docking," IEEE Trans. Instr. Meas., Vol. 44, No. 3, pp. 810-814, 1995. • E.M. Petriu, G. Eatherley, “Fuzzy Systems in Instrumentation: Fuzzy Control,” Proc. IMTC/95, IEEE Instr. Meas. Technol. Conf., pp.1-5, Waltham, MA, 1995. • E. Petriu, K. Watanabe, T. Yeap, “Applications of Random-Pulse Machine Concept to Neural Network Design,” IEEE Tr.. Instr. Meas., Vol. 45, No.2, 1996, pp. 665-669.• L. Zhao, Random Pulse Artificial Neural Network Architecture, M.A.Sc. Thesis, University of Ottawa, Canada, 1998 • J. Mao, Reduction of the Quantization Error in Fuzzy Logic Controllers by Dithering, M.A.Sc. Thesis, University of Ottawa, Canada, 1998.• E.M. Petriu, J. Mao, “Fuzzy Sensing and Control for a Truck,” Proc. VIMS-2000, IEEE Workshop on Virtual and Intelligent Measurement Systems, Annapolis, MD, April 2000, pp. 27-32.• E. M. Petriu, L. Zhao, S.R. Das, V.Z. Groza, A. Cornell, “Instrumentation Applications of Multibit Random-Data Representation,” IEEE Tr. Instr. Meas., Vol. 52, No. 1, 2003, pp. 175- 181.• M. Dostaler, Multi-Level Random Data Based Correlator Model, M.A.Sc. Thesis, 2005.
AcknowledgementThis paper represents a synthesis of the work carried out by the author and his collaborators published over the years in conference proceedings and journals:
Acknowledgement
The work reported in this paper was funded in part by Communications and Information Technology Ontario (CITO) and the Natural Sciences and Engineering Research Council (NSERC) of Canada.
Thank you!Thank you!