multi-valued and universal binary neurons: theory and learningbiehl/coll/aizenberg.pdf · the unit...
TRANSCRIPT
![Page 1: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/1.jpg)
Multi-Valued and Universal Binary Neurons:
New Solutions in Neural Networks
Igor AizenbergTexas A&M University-Texarkana,
Department of Computer and Information Sciences
http://www.eagle.tamutedu/faculty/igorhttp://www.freewebs.com/igora
![Page 2: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/2.jpg)
NEURAL NETWORKS
Traditional Neurons
Neuro-Fuzzy Networks
Complex-Valued Neurons
Generalizations of sigmoidalNeurons
Multi-Valued andUniversal Binary Neurons
![Page 3: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/3.jpg)
Multi-Valued and Universal Binary Neurons (MVN and UBN)
MVN’s and UBN’s complex-valued activation functions are phase-dependent: their outputs are completely determined by the weighted sum’s argument and they are always lying on the unit circle.
![Page 4: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/4.jpg)
Initial ideas
Generalization of the principles of Boolean threshold logic for the multiple-valued caseExtension of a set of the Boolean functions that allow learning using a single neuron (overcoming of the Minsky-Papert limitation)Consideration of new corresponding types of neurons
![Page 5: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/5.jpg)
Multi-Valued Neuron
(MVN):learns mappings described by multiple-valued functions (functions of multiple-valued logic including continuous-
valued logic)
Universal Binary Neuron (UBN):learns mappings described by Boolean functions that are not necessarily threshold (linearly separable)
Multi-Valued and Universal Binary Neurons
![Page 6: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/6.jpg)
Contents
Multiple-Valued Logic (k-valued Logic) over the Field of the Complex NumbersMulti-Valued Neuron (MVN)Discrete and continuous-valued MVNLearning Algorithm for MVNThreshold Boolean Functions and P-realizable Boolean FunctionsUniversal Binary Neuron (UBN)Relation between MVN and UBNLearning algorithm for UBN Solving the XOR and parity problems on a single UBNMVN-based multilayer feedforward neural network and its derivative free backpropagation learning algorithmApplicationsFurther work
![Page 7: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/7.jpg)
Multi-Valued Neuron (MVN)
A Multi-Valued Neuron is a neural element with n inputs and one output lying on the unit circle, and with the complex-valued weights.The theoretical background behind the MVN is the Multiple-Valued (k-valued) Threshold Logic over the field of complex numbers
![Page 8: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/8.jpg)
Multiple-Valued Threshold Logic over
the field of Complex numbers and a Multi-Valued Neuron
![Page 9: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/9.jpg)
Multiple-Valued Logic: classical view
The values of multiple-valued(k-valued) logic are traditionally encoded by the integers {0,1, …, k-1}
![Page 10: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/10.jpg)
Multiple-Valued Logic over the field of complex numbers: basic idea
Let M be an arbitrary additive group and its cardinality is not lower than k
Let be a structural alphabet
{ }0 1 1, ,... ,, k kk a a aA A M−= ∈
![Page 11: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/11.jpg)
Multiple-Valued Logic over the field of complex numbers: basic idea
Let be the nth Cartesian power of
Basic Definition: Any function of nvariables is called a function of k-valued logic over group M
nkA
kA
( )1 | :,..., nnk kfx Af x A→
![Page 12: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/12.jpg)
Multiple-Valued Logic over the field of complex numbers: basic idea
If then we obtain traditional k-valued logic.
In terms of previous considerations is a group with
respect to mod k
addition.
{ }0,1,..., 1kA k= −
{ }0,1,..., 1kA k= −
![Page 13: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/13.jpg)
Multiple-Valued Logic over the field of complex numbers: basic idea
Let us take the field of complex numbers as a group MLet , where i is an imaginary unity, be a primitive kth root of unity Let be set
C
exp( 2 )i / kε π=
{ }0 2 1, , ,..., kk ε ε ε ε −Ε = kA
![Page 14: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/14.jpg)
Multiple-Valued (k-valued) logic over the field of complex numbers
(Proposed and formulated by Naum
Aizenberg in 1971-1973)
Important conclusion: Any function of nvariables is a function of k-valued logic over the field of complex numbers according to the basic definition
( )1 | :,..., nnk kff x x Ε →Ε
![Page 15: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/15.jpg)
Multiple-Valued (k-valued) logic over the field of complex numbers
(Proposed and formulated by Naum
Aizenberg in 1971-1973)
/exp( 2 )i kε π=
{0, 1, ..., 1}j k∈ −
exp( 2 / )jj i j kπε→ =
primitive kth
root of unity
regular values of k-valued logici
0
1
k-1
k-2
ε
ε2
εk-1
1one-to-one correspondence
{ }0 2 1, , ,..., kjε ε ε ε ε −∈The
kth
roots of unity
are values
of
k-valued logic over the field of complex numbers
![Page 16: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/16.jpg)
Important advantage
In multiple-valued logic over the field of complex numbers all values of this logic are algebraically (arithmetically) equitable: their absolute values are equal to 1
![Page 17: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/17.jpg)
Discrete-Valued (k-valued) Activation Function
( )= exp 2 / , if
( )/ ( ) ( ) /
ji πj k2 j k arg z 2π j +1 k
P z επ
=
≤ <i
0
1
k-2Z
j-1
J j+1
k-1
Function P
maps the complex plane into the set of the kth
roots of unity
jε
1jε +
![Page 18: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/18.jpg)
Multiple-Valued (k-valued) Threshold FunctionsThe k-valued function
is called a k-valued threshold function, if such a complex-valued weighting vector exists that for all from the domain of the function f:
),...,( 1 nxxf
),...,,( 10 nwww),...,( 1 nxxX =
)...(),...,( 1101 nnn xwxwwPxxf +++=
![Page 19: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/19.jpg)
Multi-Valued Neuron (MVN)
(introduced by I.Aizenberg
& N.Aizenberg
in 1992)
)...(),...,( 1101 nnn xwxwwPxxf +++=f
is a function of k-valued logic
(k-valued threshold function) 1x
nx
),...,( 1 nxxf. . .
P(z)
nn xwxwwz +++= ...110
![Page 20: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/20.jpg)
MVN: main properties
The key properties of MVN: Complex-valued weightsThe activation function is a function of the argument of the weighted sumComplex-valued inputs and output that are lying on the unit circle (kth roots of unity)Higher functionality than the one for the traditional neurons (e.g., sigmoidal)Simplicity of learning
![Page 21: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/21.jpg)
MVN Learning
q sδ ε ε= −
qεi
qs
sε
qε
•Learning is reduced to movement along the unit circle
•No derivative is needed, learning is based on the error-correction rule
- error, which completely determines the weights adjustment
sε-
Desired output
-
Actual output
![Page 22: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/22.jpg)
Learning Algorithm for the Discrete MVN with
the Error-Correction Learning Rule
X
qε
W –
weighting vector; X
-
input vectoris a complex conjugated to X
αr
–
learning rate (should be always equal to 1)r -
current iteration;
r+1 –
the next iteration
is a desired
output (sector)
is an actual
output (sector)
1 ( )( 1)
q srr+ r + ε - ε X
n +W W α
= i
qs
sε
qε
sε
![Page 23: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/23.jpg)
Continuous-Valued Activation Function
(introduced by Igor Aizenberg in 2004)
Continuous-valued case (k ∞):
Arg exp( (arg ))
(
)/ | |
i zP z ez
i zz
= = ==
Function P
maps the complex plane into the unit circle
Z
||/)( zzzP =
i
1
![Page 24: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/24.jpg)
Learning Algorithm for the Continuous MVN with
the Error Correction Learning Rule
XW –
weighting vector; X
-
input vector
is a complex conjugated to Xαr
–
a learning rate (should be always equal to 1)r -
current iteration;
r+1 –
the next iterationZ
– the weighted sum
1 ( 1) | |qr r
rr rW z+ ε - XW
n + zα
+
⎛ ⎞= ⎜ ⎟
⎝ ⎠i
|| zzqε
|| zzq −ε
qε is a desired
output
is an actual
output| |zz | |
q zz
εδ = − -
neuron’s error
![Page 25: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/25.jpg)
A role of the factor 1/(n+1) in the Learning Rule
|| zzq −= εδ -
neuron’s error
nnn xn
wwxn
wwn
ww)1(
~ ; ... ;)1(
~ ;)1(
~11100 +
δ+=
+δ
+=+δ
+=
. ...)1(
...)1()1(
))1(
(...))1(
())1(
(
~...~~~
110
110
11110
110
δδ
δδδ
δδδ
+=++++=
=+
++++
+++
+=
=+
++++
+++
+=
=+++=
zxwxwwn
xwn
xwn
w
xxn
wxxn
wn
w
xwxwwz
nn
nn
nn
nn
The weights after the correction:
The weighted sum after the correction:
![Page 26: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/26.jpg)
Self-Adaptation of the Learning Rate
-is the absolute value of the weighted sum on theprevious (rth) iteration.
rz
is a self-adaptive part of the learning rate1
rz
i
|z|
<1|z|
>1
1
1 ( 1) | |r+qr r
rr r
zW W + ε - Xn+ z zα ⎛ ⎞
= ⎜ ⎟⎝ ⎠
1/|zr
|
is a self-adaptive part of the learning rate
![Page 27: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/27.jpg)
Modified Learning Rules with the Self-Adaptive Learning Rate
1 ( )( 1) | |
q srr r
r
W W + ε - ε Xn+ z
α+ =
1/|zr
|
is a self-adaptive part of the learning rate
1 ( 1) | |qr
r rr
zW W + ε - Xn+ z zα
+
⎛ ⎞= ⎜ ⎟
⎝ ⎠
Discrete MVN
Continuous MVN
![Page 28: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/28.jpg)
Convergence of the learning algorithm
It is proven that the MVN learning algorithm converges after not more than k! iterations for the k -valued activation functionFor the continuous MVN the learning algorithm converges with the precision λ after not more than (π/λ)! iterations because in this case it is reduced to learning in π/λ –valued logic.
![Page 29: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/29.jpg)
MVN as a model of a biological neuron
No impulses Inhibition Zero frequency
Intermediate State Medium frequency
Excitation High frequency
The State of a biological neuron is determined by the frequency of the generated impulses
The amplitude of impulses is always a constant
![Page 30: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/30.jpg)
MVN as a model of a biological neuron
The state of a biological neuron is determined by the frequency of the generated impulses
The amplitude of impulses is always a constant
The state of the multi-valued neuron is determined by the argument of the weighted sum
The amplitude of the state of the multi-valued neuron is always a constant
![Page 31: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/31.jpg)
MVN as a model of a biological neuron
Maximal inhibition
Maximal excitation
2π0π
Intermediate State
Intermediate State
![Page 32: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/32.jpg)
MVN as a model of a biological neuron
Maximal inhibition
Maximal excitation
2π0π
![Page 33: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/33.jpg)
MVN:Learns fasterAdapts betterLearns even highly nonlinear functionsOpens new very promising opportunities for the network designIs much closer to the biological neuronAllows to use the Fourier Phase Spectrum as a source of the features for solving different recognition/classification problemsAllows to use hybrid (discrete/continuous) inputs/output
![Page 34: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/34.jpg)
P-realizable Boolean Functions over
the field of Complex numbers and a Universal Binary Neuron
![Page 35: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/35.jpg)
Boolean Alphabets
{0, 1} –
a classical alphabet
{1,-1} –
an alternative alphabet used here
11 ;1021}1,1{ };1,0{
−→→⇒−=−∈∈
yxxy
![Page 36: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/36.jpg)
Threshold Boolean Functions (a classical definition)
The Boolean function is called a threshold
function, if such a real-valued weighting vector exists that for all from the domain of the function f:
),...,( 1 nxxf
),...,,( 10 nwww),...,( 1 nxxX =
⎩⎨⎧
−=<+++=≥+++
1),...,( if 0, ...1),...,( if ,0...
1110
1110
nnn
nnn
xxfxwxwwxxfxwxww
![Page 37: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/37.jpg)
A threshold function is a linearly separable function
1
1
-1
-1
f(1,1)=
1
f(1,-1)= -1
f(-1,1)= -1
f(-1,-1)=
-1
f (x1
, x2
)
is the
OR function
![Page 38: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/38.jpg)
Threshold Boolean Functions
The threshold function could be implemented on the single neuronThe number of threshold functions is very small in comparison to the number of all functions (104 of 256 for n=3, about 2000 of 65536 for n=4, etc.)Non-threshold functions can not be implemented on a single neuron (Minsky-Papert, 1969)
![Page 39: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/39.jpg)
XOR – a classical non-threshold function
1
1
-1
-1
f(1,1)=1
f(1,-1)= -1
f(-1,1)= -1
f(-1,-1)=1
![Page 40: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/40.jpg)
Is it possible to overcome
the Minsky’s-Papert’s
limitation for the classical perceptron?
Yes !!!
![Page 41: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/41.jpg)
P-realizable Boolean functions (introduced by I. Aizenberg in 1984-1985)
))1(2()arg()2(if,)1()(
/mj+π zj/m =zP j
B
<≤−
π
i
0
1
m-1
m-2Z
1
-1
1-1
-1
1
1=)(zPB
Binary activation function:
![Page 42: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/42.jpg)
P-realizable Boolean functions
The Boolean function is called a P-realizable function, if such a complex-valued weighting vector exists that for all from the domain of the function f:
),...,( 1 nxxf
),...,,( 10 nwww
),...,( 1 nxxX =
)...(),...,( 1101 nnBn xwxwwPxxf +++=
![Page 43: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/43.jpg)
Universal Binary Neuron (UBN) (introduced in 1985)
)...(),...,( 1101 nnBn xwxwwPxxf +++=
1x
nx
),...,( 1 nxxf...
PB(z)
nn xwxwwz +++= ...110
![Page 44: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/44.jpg)
UBN: main properties
The key properties of UBN: Complex-valued weightsThe activation function is a function of the argument of the weighted sumAbility to learn arbitrary (not necessarily threshold) Boolean functions on a single neuronSimplicity of learning
![Page 45: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/45.jpg)
XOR problem
x1 x222110 xwxww
z++=
=)(zPB f(x1, x2)
1 1 1+i 1 1
1 -1 1-i -1 -1
-1 1 -1+i -1 -1
-1 -1 -1-i 1 1
1−=BP
1−=BP
n=2, m=4
–
four sectorsW=(0, 1, i) –
the weighting vector
1
i
-i-1
1=BP
1=BP
![Page 46: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/46.jpg)
Parity problemn=3, m=6 : 6 sectors
W=(0, ε, 1, 1) –
the weighting vector
1
1-1
1
-1 -1
ε
1-1
1X 2X 3X3322
110
xwxwxwwZ
++++=
)(ZPB ),,( 321 xxxf
1 1 1 2+ε 1 1 1 1 -1 ε -1 -1 1 -1 1 ε -1 -1 1 -1 -1 2−ε 1 1 -1 1 1 2+ε− -1 -1 -1 1 -1 ε− 1 1 -1 -1 1 ε− 1 1 -1 -1 -1 2−ε− -1 -1
![Page 47: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/47.jpg)
Dependence of the number of sectors on the number of inputs
A proven estimation for m, to ensure P-realizibility of all functionsEstimation for self-dual and even functions (experimental fact, mathematically open problem)P-realizability of all functions may be reduced to P-realizability of self-dual and even functions
n
mn 222 ≤≤
nmn 32 ≤≤
![Page 48: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/48.jpg)
Relation between P-realizable Boolean functions and k-valued threshold functions
If the Boolean function f is P-realizable with a separation of the complex plane into m sectors then at least one m-valued threshold function fm (partially defined on the Boolean domain) exists, whose weighting vector is the same that for the Boolean function f.The learning of P-realizable function may be reduced to the learning of partially defined m-valued threshold function
![Page 49: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/49.jpg)
Learning Algorithm for UBN
q
= s-1 (mod
m),
if
Z
is
closer
to
(s-1)st
sector
q = s+1 (mod
m), if
Z
is
closer
to
(s+1)st
sector
1( )
1
qr
r+ r
sε - ε+ X ,(n+
W)
W α=
s+1 i s s-1
Semi-supervised/semi-
self-adaptive learning
![Page 50: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/50.jpg)
Main Applications
![Page 51: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/51.jpg)
MVN- based Multilayer Feedforward Neural Network (MLMVN) and a Backpropagation Learning
Algorithm
Suggested and developed by Igor Aizenberg et al in 2004-2006
![Page 52: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/52.jpg)
MVN-
based Multilayer Feedforward Neural Network
Hidden layers Output layer
1
2
N
![Page 53: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/53.jpg)
MLMVN: Key PropertiesDerivative-free learning algorithm based on the error-correction ruleSelf-adaptation of the learning rate for all the neuronsMuch faster learning than the one for other neural networksA single step is always enough to adjust the weights for the given set of inputs independently on the number of hidden and output neuronsBetter recognition/prediction/classification rate in comparison with other neural networks, neuro-fuzzy networks and kernel based techniques including SVM
![Page 54: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/54.jpg)
MLMVN: Key Properties
MLMVN can operate with both continuousand discrete inputs/outputs, as well as with the hybrid inputs/outputs: continuous inputs discrete outputsdiscrete inputs continuous outputshybrid inputs hybrid outputs
![Page 55: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/55.jpg)
A Backpropagation Derivative- Free Learning Algorithm
kmT -
a desired output of the kth
neuron from the mth
(output) layer
kmY -
an actual output of the kth
neuron from the mth
(output) layer
kmkmkm YT −=δ* -
the network error for the kth
neuron from output layer
*1km
mkm s
δ=δ -
the error
for the kth
neuron from output layer
11 += −mm Ns -the number of all neurons on the previous layer (m-1, to which the error is backpropagated) incremented by 1
![Page 56: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/56.jpg)
A Backpropagation Derivative- Free Learning Algorithm
∑∑++
=
−++
=
+++
δ=δ=δ11
1
111
1
1121
)(1)(11 jj N
i
ijkij
j
N
i
ijkijij
kjkj w
sw
ws
The error
for the kth
neuron from the hidden (jth) layer, j=1, …, m-1
1 ;,...,2 ,1 11 ==+= − smjNs jj
-the number of all neurons on the previous layer (previous to j, to which the error is backpropagated) incremented by 1
The error backpropagation:
![Page 57: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/57.jpg)
A Backpropagation Derivative- Free Learning Algorithm
Correction rule for the neurons from the mth
(output) layer (kth
neuron of mth
layer):
kmkmkmkm
imkmkmkm
ikmi
nCww
niYnCww
δ+
+=
=δ+
+= −
)1(~
,...,1 ,~)1(
~
00
1
![Page 58: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/58.jpg)
A Backpropagation Derivative- Free Learning Algorithm
Correction rule for the neurons from the 2nd
through m-1st
layer (kth
neuron of the jth
layer (j=2, …, m-1):
kjkj
kjkjkj
ikjkj
kjkji
kji
znC
ww
niYzn
Cww
δ+
+=
=δ+
+=
||)1(~
,...,1 ,~||)1(
~
00
![Page 59: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/59.jpg)
A Backpropagation Derivative- Free Learning Algorithm
11
110
10
11
111
||)1(~
,...,1 ,||)1(
~
kk
kkk
ikk
kki
ki
znCww
nixzn
Cww
δ+
+=
=δ+
+=
Correction rule for the neurons from the 1st
hidden
layer:
![Page 60: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/60.jpg)
Criteria for the convergence of the learning process
Learning should continue until either minimum MSE/RMSE
criterion will be satisfied or zero-error will be reached
![Page 61: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/61.jpg)
MSE criterion
λ≤=δ ∑∑∑==
N
ss
N
s kskm E
NW
N 11
2* 1)()(1
λ is a maximum possible MSE for the training data
N is
the number of patterns in the training set
is the network square error for the sth
pattern
* 2( )s kmsk
E δ=∑
![Page 62: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/62.jpg)
RMSE criterion
λ≤=δ ∑∑∑==
N
ss
N
s kskm E
NW
N 11
2* 1)()(1
λ is a maximum possible RMSE for the training data
N is
the number of patterns in the training set
is the network square error for the sth
pattern
* 2( )s kmsk
E δ=∑
![Page 63: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/63.jpg)
MLMVN Learning: Example
1 1
2 2
3 3
(exp(4.23 ),exp(2.10 )) ,
(exp(5.34 ),exp(1.24 )) ,
(exp(2.10 ),exp(0 )) .
X i i T
X i i T
X i i T
= →
= →
= →
Suppose, we need to classify three vectors belonging to three different classes:
( ) ( ) ( )1 2 3exp 0.76 , exp 2.56 , exp 5.35 .T i T i T i= = =
1 2 3, ,T T T
[arg( ) 0.05,arg( ) 0.05], 1, 2,3,j jT T j− + =
Classes are determined in such a way that the argument of the desired output of the network must belong to the interval
![Page 64: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/64.jpg)
MLMVN Learning: Example
Thus, we have to satisfy the following conditions:
( ) ( ) arg arg 0.05iArg zjT - e ≤
iArg ze
, where
is the actual output.
and for the mean square error
20.05 0.0025E ≤ =
.
![Page 65: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/65.jpg)
MLMVN Learning: Example
Let us train the 2 1 MLMVN
(two hidden neurons and the single neuron in the output layer
x1
x2
( )21, xxf
11
12
21
![Page 66: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/66.jpg)
MLMVN Learning: ExampleThe training process converges after 7 training epochs.
Update of the outputs:
![Page 67: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/67.jpg)
MLMVN Learning: Example
E p o c h
1 2 3 4 5 6 7
M S E
2.4213 0.1208 0.2872 0.1486 0.0049 0.0026 0.0009
20.05 0.0025E ≤ =
![Page 68: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/68.jpg)
MLMVN: Simulation Results
![Page 69: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/69.jpg)
Simulation Results: Benchmarks
All simulation results for the benchmark
problems are obtained using the network with n
inputs
n S 1 containing a single hidden layer with S
neurons
and a single neuron in the output layer:
Hidden layer
Output layer
![Page 70: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/70.jpg)
The Two Spirals Problem
![Page 71: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/71.jpg)
Two Spirals Problem: complete training of 194 points
Structure of thenetwork A network type The results
2 40 1
MLMVN Trained completely, no errors
A traditional MLP, sigmoid activation function
still remain 4% errors
2 30 1
MLMVN Trained completely, no errors
A traditional MLP, sigmoid activation function
still remain 14% errors
![Page 72: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/72.jpg)
Two Spirals Problem: cross-validation (training of 98 points
and prediction of the rest 96 points)
The prediction rate is stable for all the networks from 2 26 1 till 2 40 1:
68-72%
The prediction rate of 74-75%
appears 1-2 times per 100 experiments with the network 2 40 1
The best known result obtained using the Fuzzy Kernel Perceptron (Nov. 2002) is 74.5%,
But it is obtained using more complicated and larger network
![Page 73: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/73.jpg)
The “Sonar” problem
208 samples:60 continuous-valued inputs1 binary output
The “sonar”
data is trained completely using the simplest possible MLMVN
60 2 1
(from 817 till 3700 epochs in 50 experiments)
![Page 74: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/74.jpg)
The “Sonar”
problem: prediction
104 samples in the training set104 samples in the testing set
The prediction rate is stable for 100 experiments with the simplest possible MLMVN
60 2 1:
88-93%
The best result for the Fuzzy Kernel Perceptron (Nov., 2002) is 94%.The best result for SVM is 89.5%.However, both FKP and SVM used are larger and their trainingis more complicated task.
![Page 75: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/75.jpg)
Mackey-Glass time series prediction
Mackey-Glass differential delay equation:
),()(1.0)(1
)(2.0)(10 tntx
txtx
dttdx
+−−+−
=ττ
The task of prediction is to predict )6( +tx
from )18( ),12( ),6( ),( −−− txtxtxtx
.
![Page 76: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/76.jpg)
Mackey-Glass time series prediction
Training Data: Testing Data:
![Page 77: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/77.jpg)
Mackey-Glass time series prediction
RMSE Training: RMSE Testing:
![Page 78: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/78.jpg)
Mackey-Glass time series predictionTesting Results:
Blue curve
–
the actual series; Red curve
–
the predicted series
![Page 79: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/79.jpg)
Mackey-Glass time series prediction
# of neurons on the hidden
layer 50 50 40
ε- a maximum possible RMSE 0.0035 0.0056 0.0056
Actual RMSE for the training set (min - max) 0.0032 - 0.0035 0.0053 – 0.0056 0.0053 – 0.0056
Min 0.0056 0.0083 0.0086 Max 0.0083 0.0101 0.0125
Median 0.0063 0.0089 0.0097 Average 0.0066 0.0089 0.0098
RMSE for the testing
set SD 0.0009 0.0005 0.0011 Min 95381 24754 34406 Max 272660 116690 137860
Median 145137 56295 62056
Number of training epochs Average 162180 58903 70051
The results of 30 independent runs:
![Page 80: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/80.jpg)
Mackey-Glass time series prediction
Comparison of MVN to other models:
MLMVNmin
MLMVNaverage
GEFREXM. Rosso, Generic Fuzzy
Learning, 2000 min
EPNet You-Liu,
Evolution. system, 1997
ANFISJ.S. R. Jang,
Neuro-Fuzzy, 1993
CNNE M.M.Islam
et all., Neural
Networks Ensembles,
2003
SuPFuNISS.Paul et
all, Fuzzy-Neuro, 2002
Classical Backprop.
NN 1994
0.0056 0.0066 0.0061 0.02 0.0074 0.009 0.014 0.02
MLMVN outperforms all other networks in:
•The number of either hidden neurons or supporting vectors
•Speed of learning
•Prediction quality
![Page 81: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/81.jpg)
Real World Problems:
Solving Using MLMVN
![Page 82: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/82.jpg)
Classification of gene expression microarray data
![Page 83: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/83.jpg)
“Lung”
and “Novartis”
data sets
4 classes in both data sets“Lung”: 197 samples with 419 genes“Novartis”: 103 samples with 697 genes
![Page 84: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/84.jpg)
K-folding cross validation with K=5
“Lung”: 117-118 samples of 197 in each training set and 39-40 samples of 197 in each testing set“Novartis”: 80-84 samples of 103 in each training set and 19-23 samples of 103 in each testing set
![Page 85: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/85.jpg)
MLMVN
Structure of the network: n 6 4 (n inputs, 6 hidden neurons and 4 output neurons)The learning process converges very quickly (500-1000 iterations that take up to 1 minute on a PC with Pentium IV 3.8 GHz CPU are required for both data sets)
![Page 86: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/86.jpg)
Results: Classification Rates
MLMVN: 98.55% (“Novartis)MLMVN: 95.945% (“Lung”)kNN (k=1) 97.69% (“Novartis”)kNN (k=2) 92.55% (“Lung”)
![Page 87: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/87.jpg)
Generation of the Genetic Code using MLMVN
![Page 88: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/88.jpg)
Genetic code
There are exactly four different nucleic acids: Adenosine (A), Thymidine (T), Cytidine (C) and Guanosine (G)Thus, there exist 43=64 different combinations of them “by three”. Hence, the genetic code is the mappingbetween the four-letter alphabet of the nucleic acids (DNA) and the 20-letter alphabet of the amino acids (proteins)
![Page 89: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/89.jpg)
Genetic code as a multiple- valued function
Let be the amino acidLet be the nucleic acidThen a discrete function of three variables
, 1, ..., 20jG j =
{ }, , , , 1, 2, 3ix A G C T i∈ =
1 2 3( , )jG f x , x x=
is a function of a 20-valued logic, which is partially defined on the set of four-valuedvariables
![Page 90: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/90.jpg)
Genetic code as a multiple- valued function
A multiple-valued logic over the field of complex numbers is a very appropriate model to represent the genetic code functionThis model allows to represent mathematically those biological properties that are the most essential (e.g., complementary nucleic acids A T; G Cthat are stuck to each other in a DNA double helix only in this order
![Page 91: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/91.jpg)
DNA double helix
The complementary nucleic acids
A T; G C
are always stuck to each other in pairs
![Page 92: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/92.jpg)
Representation of the nucleic acids and amino acids
ND
T
F
SA
AY
G
WM
I
H
Q
T K
ERGC
LV
PC
The complementary nucleic acids
A T; G Care located such that
A=1; G=i
T
= -A= -1
and
C= -
G
= -i
All amino acids
are distributed along the unit circle in the logical way, to insure their closeness to those nucleic acids that form each certain amino acid
![Page 93: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/93.jpg)
Generation of the genetic code
The genetic code can be generated using MLMVN 3 4 1 (3 inputs, 4 hidden neurons and a single output neuron – 5 neurons)There best known result for the classical backpropagation neural network is generation of the code using the network 3 12 2 20 (32 neurons)
![Page 94: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/94.jpg)
Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN
![Page 95: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/95.jpg)
Problem statement: capturing
Mathematically a variety of capturing principles can be described by the Fredholm integral of the first kind
where x,t ℝ2, v(t) is a point-spread function (PSF) of a system, y(t) is a function of a real object and z(x) is an observed signal.
2
2( ) ( , ) ( ) , ,z x v x t y t dt x t= ∈∫
Mic
rosc
opy
Tomog
raphy
∈
Ph
oto
![Page 96: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/96.jpg)
Image deblurring: problem statement
Mathematically blur is caused by the convolution of an image with the distorting kernel.Thus, removal of the blur is reduced to the deconvolution.Deconvolution is an ill-posed problem, which results in the instability of a solution. The best way to solve it is to use some regularization technique.To use any kind of regularization technique, it is absolutely necessary to know the distorting kernel corresponding to a particular blur: so it is necessary to identify the blur.
![Page 97: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/97.jpg)
Image deblurring: problem statement
The observed image given in the following form:
where “*”
denotes a convolution operation, y
is an image, υ is point-spread function
of a system (PSF) , which is exactly
a distorting kernel, and ε is a noise.
In the continuous frequency domain
the model takes the form:
where is a representation of the signal z
in the Fourier domain
and denotes a Fourier transform.
( ) ( )( ) ( )z x v y x xε= ∗ + ,
( ) ( ) ( ) ( )Z V Yω ω ω ε ω= ⋅ + ,
2( ) { ( )}Z F z x Rω ω= , ∈ ,{}F ⋅
![Page 98: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/98.jpg)
Blur Identification
We use MLMVN to recognize Gaussian, motion and rectangular (boxcar) blurs.We aim to identify simultaneously both blur (PSF), and its parameters using a single neural network.
![Page 99: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/99.jpg)
Considered PSFPSF in time domain
PSF in frequency domain
Gaussian Motion Rectangular
![Page 100: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/100.jpg)
Considered PSF
The Gaussian PSF:
is a parameter (variance)
The uniform linear motion:
h
is a parameter (the length of motion)
The uniform rectangular:
h
is a parameter (the size of smoothing area)
2 21 2
2 2
1( ) exp2
t tv t
πτ τ⎛ ⎞+
= −⎜ ⎟⎝ ⎠
2 21 2 1 2
1 , / 2, cos sin ,( )otherwise,0,
t t h t tv t h φ φ⎧⎪ + < == ⎨⎪⎩
1 22
1 , , , ( ) 2 2
otherwise,0,
h ht tv t h
⎧ < <⎪= ⎨⎪⎩
2τ
![Page 101: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/101.jpg)
Degradation in the frequency domain:
True Image Gaussian Rectangular HorizontalMotion
VerticalMotion
Images and log of their Power Spectra log Z
![Page 102: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/102.jpg)
Training Vectors
We state the problem as a recognition of the shape of V , which is a Fourier spectrum of PSF υ and its parameters from the Power-Spectral Density, whose distortions are typical for each type of blur.
![Page 103: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/103.jpg)
Training Vectors
( )( ) ( )( ) ( )
1 2, min
max min
log logexp 2 ( 1) ,
log logk k
j
Z Zx i K
Z Z
ωπ
⎛ ⎞−⎜ ⎟= ⋅ −⎜ ⎟−⎜ ⎟⎝ ⎠
),...,( 1 nxxX =The training vectors are formed as follows:
for
(| ( ) |)Log Z ω
(0,0)ω
1 2 2
1 2
2 1
1,..., / 2 1, for , 1,..., / 2 1,/ 2,..., 2, for 1, 1,..., / 2 1,
1,...,3 / 2 3, for 1, 1,..., / 2 1,
j L k k k Lj L L k k Lj L L k k L
= − = = −⎧⎪ = − = = −⎨⎪ = − − = = −⎩
LxL is a size of an image
the length of the pattern vector is n= 3L/2-3
![Page 104: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/104.jpg)
Examples of training vectors
True Image Gaussian Rectangular
HorizontalMotion
VerticalMotion
![Page 105: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/105.jpg)
Neural Network
Hidden layers Output layer
1
2
N
Blur 1
Blur 2
Blur N
Training (pattern) vectors
![Page 106: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/106.jpg)
Output Layer Neuron
Reservation of domains on the unit circle for the output neuron
![Page 107: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/107.jpg)
Simulation
{ }1, 1.33, 1.66, 2, 2.33, 2.66, 3 ;τ ∈
Experiment 1 (2700 training pattern vectors corresponding to 72 images): six types of blur with the following parameters:MLMVN structure: 5 35 6
1) The Gaussian blur is considered with 2) The linear uniform horizontal motion blur of the lengths 3, 5, 7, 9; 3) The linear uniform vertical motion blur of the length 3, 5, 7, 9; 4) The linear uniform diagonal motion from South-West to North-
East blur of the lengths 3, 5, 7, 9; 5) The linear uniform diagonal motion from South-East to North-
West blur of the lengths 3, 5, 7, 9; 6) rectangular has sizes 3x3, 5x5, 7x7, 9x9.
![Page 108: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/108.jpg)
ResultsClassification Results
Blur
MLMVN, 381 inputs,5 35 6,
2336 weights in total
SVMEnsemble from
27 binary decision SVMs,
25.717.500 support vectors in total
No blur 96.0% 100.0%Gaussian 99.0% 99.4%Rectangular 99.0% 96.4
Motion horizontal 98.5% 96.4Motion vertical 98.3% 96.4
Motion North-East Diagonal 97.9% 96.5Motion North-West Diagonal 97.2% 96.5
![Page 109: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/109.jpg)
Restored images
Blurred noisy image:rectangular 9x9
Restored
Blurred noisy image:Gaussian, σ=2
Restored
![Page 110: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/110.jpg)
Main Publications1. N.N. Aizenberg, Yu. L.Ivaskiv and D.A. Pospelov "About one Generalization of the
Threshold Function", Doklady Akademii Nauk SSSR (The Reports of the Academy of Sciences of USSR), vol. 196, No. 6, 1971, pp. 1287–1290. (in Russian).
2. N.N. Aizenberg, Yu.L. Ivaskiv, D.A. Pospelov and G.F. Hudiakov "Multi-Valued Threshold Functions. 1. The Boolean Complex-Threshold Functions and their Generalization", Kibernetika (Cybernetics), No. 4, 1971, pp. 44-51 (in Russian).
3. N.N. Aizenberg, Z. Toshich. A Generalization of the Threshold Functions, // Proceedings of the Electronic Department University of Belgrade, No.399, 1972, pp.97-99.
4. N.N. Aizenberg, Yu.L. Ivaskiv, D.A. Pospelov and G.F. Hudiakov "Multi-Valued Threshold Functions. 2. Synthesis of the Multi-Valued Threshold Element" Kibernetika(Cybernetics), No. 1, 1973, pp. 86-94 (in Russian).
5. N.N. Aizenberg and Yu.L. Ivaskiv Multiple-Valued Threshold Logic, Naukova Dumka Publisher House, Kiev, 1977 (in Russian).
6. I.N. Aizenberg "Model of the Element with a Complete Functionality", Izvestia AN SSSR. Technicheskaya ("Journal of Computer and System Sciences International "), 1985, No. 2., pp. 188-191 (in Russian).
7. I.N. Aizenberg "The Universal Logical Element over the Field of the Complex Numbers",Kibernetika (Cybernetics and Systems Analysis), No.3, 1991, pp.116-121 (in Russian).
![Page 111: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/111.jpg)
Main Publications8. N.N. Aizenberg and I.N. Aizenberg "Model of the Neural Network Basic Elemets (Cells)
with Universal Functionality and various of Harware Implementation", Proc. of the 2-ndInternational Conference "Microelectronics for Neural Networks", Kyrill & MethodyVerlag, Munich, 1991, pp. 77-83.
9. N.N.Aizenberg and I.N.Aizenberg "CNN Based on Multi-Valued Neuron as a Model ofAssociative Memory for Gray-Scale Images", Proceedings of the Second IEEEInternational Workshop on Cellular Neural Networks and their Applications, TechnicalUniversity Munich, Germany October 14-16, 1992, pp.36-41.
10. N.N.Aizenberg, I.N. Aizenberg and G.A.Krivosheev “Method of Information Processing”Patent of Russia No 2103737, G06 K 9/00, G 06 F 17/10, priority date 23.12.1994.
11. N.N. Aizenberg, I.N. Aizenberg and G.A. Krivosheev "Multi-Valued Neurons: Learning,Networks, Application to Image Recognition and Extrapolation of Temporal Series",Lecture Notes in Computer Science, vol. 930 (J.Mira, F.Sandoval - Eds.), Springer-Verlag, 1995,pp.389-395.
12. I.N. Aizenberg. and N.N. Aizenberg “Universal binary and multi-valued neuronsparadigm: Conception, Learning, Applications”, Lecture Notes in Computer Sciense, vol.1240 (J.Mira,R.Moreno-Diaz, J.Cabestany - Eds.), Springer-Verlag, 1997, pp. 463-472.
13. I.N. Aizenberg “Processing of noisy and small-detailed gray-scale images using cellularneural networks”, Journal of Electronic Imaging, vol. 6, No. 3, July, 1997, pp. 272-285.
![Page 112: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/112.jpg)
Main Publications
14. I. Aizenberg., N. Aizenberg, J. Hiltner, C. Moraga, and E. Meyer zu Bexten., “Cellular Neural Networks andComputational Intelligence in Medical Image Processing”, Journal of Image and Vision Computing, Vol. 19, March,2001, pp. 177-183.
15. I. Aizenberg., E. Myasnikova, M. Samsonova., and J. Reinitz , “Temporal Classification of DrosophilaSegmentation Gene Expression Patterns by the Multi-Valued Neural Recognition Method”, MathematicalBiosciences , Vol.176 (1), 2002, pp. 145-159.
16. I. Aizenberg and C. Butakoff, “Image Processing Using Cellular Neural Networks Based on Multi-Valued andUniversal Binary Neurons”, Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology,Vol. 32, 2002, pp. 169-188.
![Page 113: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/113.jpg)
A book
Igor Aizenberg, Naum Aizenberg and Joos
Vandewalle
“Multi-Valued and Universal Binary Neurons:
Theory, Learning and Applications”,
Kluwer, 2000
![Page 114: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/114.jpg)
The latest publications1.
Aizenberg I., Paliy D., Zurada J., and Astola J., "Blur Identification by Multilayer Neural Network based on Multi-Valued Neurons", IEEE Transactions on Neural Networks, accepted, to appear: April, 2008.
2.
Aizenberg I., "Solving the XOR and Parity n
Problems Using a Single Universal Binary Neuron", Soft Computing, vol. 12, No 3, February, 2008, pp. 215-222.
3.
Aizenberg I. and Moraga C. "Multilayer Feedforward Neural Network based on Multi-Valued Neurons and a Backpropagation Learning Algorithm", Soft Computing, Vol. 11, No 2, February, 2007, pp. 169-183.
4.
Aizenberg I. and Moraga C., "The Genetic Code as a Function of Multiple- Valued Logic Over the Field of Complex Numbers and its Learning using
Multilayer Neural Network Based on Multi-Valued Neurons", Journal of Multiple-Valued Logic and Soft Computing, No 4-6, November 2007, pp. 605-
618.
5.
Aizenberg I. Advances in Neural Networks. Class notes
for the graduate course. University of Dortmund. 2005.
![Page 115: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/115.jpg)
Some Prospective ApplicationsSolving different recognition and classification problems, especially those, where the formalization of the decision rule is a complicated problemClassification and prediction in biologyTime series predictionModeling of complex systems including hybrid systems that depend on many parametersNonlinear filtering (MLMVN can be used as a nonlinear filter)Etc. …
![Page 116: Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow](https://reader034.vdocuments.mx/reader034/viewer/2022052001/6013f12cfc66d475cd381038/html5/thumbnails/116.jpg)
CONCLUSION
The best results are expected in the near
future…
If it be now, 'tis not to come; if it be not to come, it will be now;
if it be not now, yet it will come: the readiness is all…
Shakespeare