topology optimization in spatially distributed …
TRANSCRIPT
TOPOLOGY OPTIMIZATION IN SPATIALLY
DISTRIBUTED CELLULAR NEURAL NETWORK
by
Varsha Bhambhani
A thesis submitted to the Faculty of the University of Delaware in partialfulfillment of the requirements for the degree of Master of Science in MechanicalEngineering
Winter 2012
c© 2012 Varsha BhambhaniAll Rights Reserved
TOPOLOGY OPTIMIZATION IN SPATIALLY
DISTRIBUTED CELLULAR NEURAL NETWORK
by
Varsha Bhambhani
Approved:Herbert G. Tanner, Ph.D.Professor in charge of thesis on behalf of the Advisory Committee
Approved:Anette M. Karlsson, Ph.D.Chair of the Department of Mechanical Engineering
Approved:Babatunde A. Ogunnaike, Ph.D.Interim Dean of the College of Engineering
Approved:Charles G. Riordan, Ph.D.Vice Provost for Graduate and Professional Education
ACKNOWLEDGEMENTS
I would like to thank everyone who contributed to making the thesis Topology
Optimization in Spatially Distributed Cellular Neural Network a reality.
First and foremost, I am obliged to my advisor, Dr. Herbert G. Tanner
(Assistant professor, Mechanical Engineering Department, University of Delaware,
Newark, DE), for the time he spent in mentoring, leading, supporting, and encour-
aging me, which helped me to shape my thoughts in the right direction not only
on this project, but also for my future endeavor. His expertise in controls, wisdom,
guidance, patience, passion for challenges, perseverance in research, and constant
encouragement have boosted up my morale and given me the competence in the
subject of controls. This thesis would have remained a dream had it not been for
his valuable suggestions, guidance, and training. His thoughtful insights and endless
efforts, sharp perceptiveness, and penetrating questions have always showed me the
right direction at the right time. I will forever be indebted to him and I would like
to thank him for this and much more.
The members of my thesis committee, Dr. Ioannis Poulakakis and Dr.
Christopher Rasmussen, have generously given their time and expertise to better
my work. I thank them for their contribution and their good-natured support. I
gratefully acknowledge the funding sources that made my M.Sc work possible. This
work was financially supported by NSF grant number 0822845.
I am indebted to my many colleagues who supported me, especially I would
like to thank my colleague and friend Luis Ariel Valbuena Reyes for his great help
and cooperation in the experimental validation on CoroBots. I would also like
iii
to thank my colleagues and friends, Adithya Boddu, Sridhar Shah and Prasanna
Kanappam, for their immense help in the lab and for their support and valuable
suggestions. All the time spent with them will be cherished for a long time. Ef-
forts, cooperation, and help from other important members of Cooperative Robots
Laboratory, Jie Fu, Costas, Laura and Chetan Rawal are gratefully acknowledged.
Their wonderful company helped maintain a jovial atmosphere in the lab while doing
serious research at the same time.
I am grateful to my friends and family members who have always been my
supporters, especially I am indebted to my friends Rajesh Kappera, Yaw Boakye-
Hwedie, Melissa Peterson, Smita Joshi and Archana Chandran. I simply cannot
find words to express my gratitude to my brothers, Mr. Kamal Bhambhani and Dr.
Akhilesh Bhambhani. They have always encouraged me and have always been a
source of motivation in my life. They help me and support me unconditionally. I
admire the cooperation and understanding of my sister-in-law Mrs. Priya Bhamb-
hani and Mrs. Harshita Bhambhani and am thankful to Laksh, his happy face and
smile have always brightened my spirits. My parents, Mr. K. L. Bhambhani and
Mrs. Rashmi Bhambhani, have always supported me and understood me, not only
as their daughter, but as a person, and it is to them that I dedicate this thesis.
Varsha Bhambhani
iv
TABLE OF CONTENTS
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiLIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixLIST OF ALGORITHMS . . . . . . . . . . . . . . . . . . . . . . . . . . . xABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Informal problem statement, approach and overview . . . . . . . . . . 21.3 Objective and contribution . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Thesis organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 TECHNICAL BACKGROUND AND LITERATURE SURVEY . 7
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 The cnn basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.3 Stability, attractiveness and robustness . . . . . . . . . . . . . . . . . 14
2.3.1 Quantification of stability properties . . . . . . . . . . . . . . 152.3.2 Robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 Analysis and design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.5 Impact of time delays in associative memories . . . . . . . . . . . . . 20
3 COMBINED DESIGN OF TOPOLOGY AND LINK WEIGHTS 24
3.1 Preliminaries and mathematical model . . . . . . . . . . . . . . . . . 253.2 Design method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2.1 The sequential trimming method . . . . . . . . . . . . . . . . 30
v
3.2.2 The randomized algorithm method . . . . . . . . . . . . . . . 33
3.3 Conclusion and future work . . . . . . . . . . . . . . . . . . . . . . . 35
4 NUMERICAL RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.1 A design example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.2 Sequential dilution of network connectivity . . . . . . . . . . . . . . . 404.3 Randomized dilution of network connectivity . . . . . . . . . . . . . . 424.4 More memorized information: less accurate recollection . . . . . . . . 444.5 Conclusion and future work . . . . . . . . . . . . . . . . . . . . . . . 45
5 REAL TIME HARDWARE IMPLEMENTATION . . . . . . . . . 47
5.1 The experimental testbed . . . . . . . . . . . . . . . . . . . . . . . . 485.2 Discretized algorithm implementation . . . . . . . . . . . . . . . . . . 515.3 Results and discussions . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6 CONCLUSION AND FUTURE WORK . . . . . . . . . . . . . . . . 55
6.1 Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.2 Contribution of thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 556.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
APPENDIXCNN STABILITY CRITERIA . . . . . . . . . . . . . . . . . . . . . . 62
vi
LIST OF FIGURES
2.1 A two-dimensional cnn of size 4× 4 with the squares representingbasic building blocks called cells and solid lines representing theinterconnection links. . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 A circuit representation of cell C(i, j) in a cnn. . . . . . . . . . . . 11
4.1 A 24× 24 cnn interconnection structure where high cost edges arestretched and highlighted and the local links are represented bydashed lines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Patterns memorized by the cnn. A first implementation of thenetwork included only patterns 1 through 4. A second included allof them, and the two implementations were compared in terms ofperformance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.3 Network performance, parameterized by the number of linksremoved. The horizontal axis marks the level of noise injected to theinput of the network, and the vertical axis is the average recallprobability of the network for that particular level of noise. . . . . . 42
4.4 Network recall probability for a set of four memory vectors. Theoverlapping curves suggest that in terms of performance, the originalnetwork and the ones with diluted connectivity are essentiallyindistinguishable, irrespectively of the method used for optimization. 43
4.5 Network’s recall probability for set of 10 memory vectors:Comparison among the performance of original network, oneoptimized using a sequential and a random optimization algorithm 44
5.1 A CoroBot at Cooperative Robots Laboratory at University ofDelaware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.2 CoroBot experimental set-up representing the design example . . . 52
vii
5.3 Error norm:discretized system states (simulation to experimental) . 53
viii
LIST OF TABLES
4.1 Cumulative stability parameter K as a result of successive deletionof high cost edges by sequential topology dilution.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.1 Functioning of power selector switch in CoroBot . . . . . . . . . . . 48
ix
List of Algorithms
1 Topology optimization through sequential dilution . . . . . . . . . . . . . 32
2 Topology optimization through randomized methods . . . . . . . . . . . 34
x
ABSTRACT
A new network topology optimization approach to cellular neural network
design, as a method for realizing associative memories using sparser networks is
conceptualized. This type of optimization allows recurrent neural networks to be
implemented in a spatially distributed fashion, that is, with components of the
network residing in different physical locations. This could find application in ad-
dressing the problem of dynamic allocation of a team of robots to a collection of
spatially distributed tasks which is relevant for large scale environmental monitor-
ing and surveillance. Spatially distributed sensing allows for greater coverage of the
environment than a single large vehicle with multiple sensors would permit in many
cases.
In this work, we try to answer the question of how could the design process be
different if the network topology was also part of the design. A sparser cellular neural
network topology can be achieved without significantly degrading the performance
of the network, by selectively deleting those weights from the optimized network
which contribute the least to ability of the network to recall the desired patterns.
This approach is particularly useful where neural links incur varying costs, such as
implementation of associative memories over wireless sensor networks. The cellular
neural networks interconnection topology is diluted, without significantly degrading
its performance, where performance is quantified by the average recall probability
of the patterns engraved into the networks associative memory. The average recall
probability is a measure of performance of the designed network in presence of noise
and is defined as the ratio of number of recovered memory patterns (perturbed
xi
initial condition vectors which result in same output as the stored memory vector)
to the total number of perturbed initial condition vectors. Since the average recall
probability cannot be assessed prior to testing, the optimization algorithm uses
the networks stability parameters as a measure of quality of memorization, and
optimization proceeds by selectively removing costly links that contribute the least
to the magnitude of these parameters.
Two different approaches to implementing the optimization of the networks
topology are implemented and compared. The first one is a sequential process in
which a single link is removed each time, specifically the one the removal of which
incurs the least performance cost compared to all other existing high-cost links.
This method ignores the possibility that a non-obvious combination of links may
produce better results through the links simultaneous removal. This phenomenon
has been observed in simulation studies which validated the proposed method. To
validate further the optimization, but more importantly, to ensure that the overall
approach does not depend on the particular method used for the combinatorial
optimization we also implemented an alternative approach which is based on the
randomized optimization. In this approach a random sample of a sufficient number
of i.i.d possible topology is generated. In other words, each random topology in
the sample has the same probability distribution as the others and all are mutually
independent.
An example is used to demonstrate that irrespectively of the combinatorial
algorithm used, the approach yields sparser associative memories that in general
trade off performance for cost, and in many cases the performance of the diluted
network is on par with the original system. In our numerical tests, the two meth-
ods yield comparable results, which do not differ significantly in terms of resulting
network performance. Performance is quantified in terms of the network recall
probability, and in the proposed optimization algorithm approach is captured by
xii
the neural networks stability parameters. Further, we apply the ideas developed
so far to control network communication in actual robots to experimentally verify
our simulation results. Experimental testing has shown that spatially distributed
implementations of cnn on CoroBots are indeed feasible, and that for some cases,
the communication delays related to the communication between the different com-
ponents of the network are not significant enough to affect the performance and
stability properties of the dynamical system. It is shown that the error between
simulation of the discrete-time dynamics and experimental results practically coin-
cide, with a maximum error difference of the order of 10−4 . Thus the proposed
combinatorial optimization methods performed almost equally well in practice as in
simulations.
xiii
Chapter 1
INTRODUCTION
1.1 Motivation
Recent technological advances made in the field of micro electro-mechanical
systems (mems), combined with the progress made in communication systems, have
made the deployment of mobile sensor networks (msns) a reality. Multi-agent msns
that process the sensing information locally and make autonomous decisions based
on the information they collect, can greatly facilitate sensor driven tasks such as re-
mote sensing, search and terrain mapping. Furthermore, the growing ease in building
small, low-power, inexpensive mobile devices now makes it feasible to spread out a
network of sensors in a number of environments. Spatially distributed sensing al-
lows for greater coverage of the environment than a single large vehicle with multiple
sensors would permit in many cases. The later may require considerable power to
operate, lack robustness to component failure and cannot adapt the configuration or
resolution of the sensor array. Also multi-agent system may be useful in overcoming
environmental obstacles like obstructions, line of sight constraints etc.
An autonomous network has the capability to form and manage itself while
its components (nodes) move about or go out of action. This makes the system
capable of reconfiguration and self-organization, which is greatly desired due to the
unpredictable network conditions and application requirements. Localized decision-
making capability (as compared to a centralized system) offers benefits in terms of
communication bandwidth, which means less energy depletion and less expensive in-
frastructure to run the network. Locally processing as much information as possible
1
not only minimizes the total number of bits transmitted and saves communication
cost, but it also ensures faster responses. Thus a decentralized system offers the
added benefits of robustness, modularity and scalability.
1.2 Informal problem statement, approach and overview
Autonomous msns have a dynamic topology, where the nodes can move freely,
join or leave the network at any time without having to send any warning informa-
tion in the networks running process. The nodes in the network have autonomous
decision-making capabilities that are based on the information they collect. To en-
able sensor networks to identify autonomously what they are observing is not an
easy task and has been of great interest to researchers.
In msns, the individual sensor nodes are often low cost, low power devices
that have limited data processing capabilities, memory, communication rate and
battery life. For extending the lifetime of such networks, energy conservation is crit-
ically important since recharging or replacing the batteries attached to the sensors
once they are deployed is often not possible. Also as a consequence of limited com-
putational power and bandwidth, some data transmissions in the msn will probably
experience large delays and drop of data packets due to limited memory size. Hence
it is advisable to use the resources efficiently to maintain high quality of service of
the network for an extended period of time. However, in a large scale system of
nodes, the hardware and networking technologies used in the underlying msns may
differ from one system to another. The platform heterogeneity may pose a challenge
to make full use of the resources available in the integrated system. It is also difficult
to achieve real-time and reliable communication between different nodes.
The dynamic network topology of msns is another major challenge in the
sense that when an intermediate node dies, the remaining network should still be
able to guarantee real-time and reliable communication by exploiting appropriate
2
protocols and algorithm. Furthermore, different sensors for different kinds of phys-
ical variables, e.g., temperature, humidity, location, speed, generate traffic signal
flows with different characteristics in message size and sampling rate, and thus there
is a need to support service differentiation and breaking down the bigger network
to a group of subnetwork classified according to the type of tasks they do or the
amount of resource they consume would be of great importance and would ensure
cost efficiency.
Recently, researchers have shown interest in using intelligent tools especially
neural networks in energy efficient approaches of msns, due to their useful properties
such as parallel computation, distributed storage, scalability and robustness. There
is a great analogy and comparability between msns and neural networks. In this
thesis, we want to enable the msns to do local processing by implementing a neural
network on it.
This thesis work presents an approach to the optimization of the given net-
work topology in which communication links between nodes may incur variable costs.
We selectively “trim” network links in an effort to trade network performance for
smaller commination cost. A sparser network is designed without significantly de-
grading its performance, wherein, a larger network is broken down into a group
of subnetworks (this may allow for service differentiation and may support plat-
form heterogeneity in an efficient fashion). Furthermore, sparser network means
less competition for bandwidth, processing, memory and battery consumption at
nodal level.
We perform combinatorial optimization on the neural network’s topology to
obtain a sparser network, in which the links between the components of the network
that reside in different physical locations are minimized. The approach builds on
existing computationally efficient tools for the design of cellular neural networks and
uses the concept of the network’s stability parameters to assess the performance of
3
the network prior to testing.
1.3 Objective and contribution
Reported approaches on the design of sensor networks (without time delays)
start with, and build on some given network topology. The choice of this starting
point is justified from the fact that typically the physical platform on which the
network is implemented is fixed. In this thesis, we raise the question of how could
the design process be different if the network topology was also part of the design.
The objective of this work is to develop a methodology for the design of cellular
neural networks cnns with interconnection topologies optimized and suitable for
spatially distributed implementation. The novelty of the proposed approach lies in
the formulation of the combinatorial optimization problem in a way that trades-off
network performance for communication overhead, and the use of this method for
the physical implementation of associative memories across different interconnected
processors.
1.4 Thesis organization
Chapter 2 reviews the fundamental properties of the cnn model and real-
izing associative memories via cnns. This includes a brief introduction to sensor
networks, neural networks, associative memories, literature search on existing design
(synthesis) methods for neural network as associative memories and cellular neural
networks — their architecture, stability and robustness, synthesis and design and
effect of time delays in cnns.
Chapter 3 aims at the development of sparser cnn topology using the con-
strained combinatorial optimization algorithm. This involves an introduction to
terms like the stability parameter and the network recall probability. Given the
predetermined network topology and the set of memory vectors to be stored, first
4
the network is designed (solving for connection matrix and bias vector) by formu-
lating an optimization problem, with non-linear constraints, which is transformed
into a generalized eigen value problem (gevp) with linear matrix inequality (lmi)
constraints. Networks designed using this method are expected to exhibit less spu-
rious patterns and higher recall probabilities. In the following two sections, we test
two different approaches to performing the combinatorial optimization. The first
one is a sequential process in which a single link is removed each time, specifically
the one the removal of which incurs the least performance cost compared to all
other existing high-cost links. To validate the results of this optimization method
we also implement an alternative approach which is based on the randomized opti-
mization. In this approach a random sample of a sufficient number of independent
and identically distributed (i.i.d) possible topologies is generated.
In Chapter 4, we present a numerical example, a cnn that is made up of
n = 24 cells interconnected locally and the topology represented by the index matrix
S. The network is divided into three small sub-networks and binary patterns are
stored as associative memories. A section is devoted to each kind of dilution method
i.e dilution by sequential trimming and by randomized method approach. Simulation
results are provided and comparative analysis is done. It is concluded that the
performance of the neural network is not adversely affected by a moderate dilution
of connectivity.
Chapter 5 presents the real-time communication using CoroBot platforms.
This involves a brief description of the CoroBot mobile robot platforms and server
client model for wireless communication. Each of the subnetworks described in
the designed example, is realized on a different CoroBot, and the high cost links
between the subnetworks are implemented over wireless channels. The objective is
to experimentally test the hypothesis that a cnn can be implemented in a spatially
distributed way, and that the unavoidable communication delays related to wireless
5
communication do not necessarily destroy the stability properties of the dynamical
system.
Chapter 6 concludes the thesis by summarizing our findings with remarks on
the future research work.
1.5 Future work
In this thesis, we show that such spatially distributed implementations of
neural networks over mobile sensor networks are indeed feasible, and that for some
cases, the communication delays related to the communication between the different
components of the net- work are not significant enough to affect the performance
and stability properties of the dynamical system. Furthermore, in our lab setup
experiments on CoroBots, if an application imposes larger communication delays
which cannot be accommodated by the length of the discrete-time step, one possible
solution would be to “stretch” the time axis of the discrete-time system, “freezing”
the evolution of the discrete-time system between steps for a period that allows the
dissemination of information between subnetworks. In this way, convergence speed
is traded-off for increased robustness to communication delays, and stability of the
network (in terms of convergence to memory vectors) is not affected.
A possible area for future research would be to analyze global robust stability
of uncertain cellular neural networks with discrete and distributed delays. This
section focusses on effects of time delays in cnns, a brief overview and possible
direction for future research should be discussed.
6
Chapter 2
TECHNICAL BACKGROUND AND LITERATURE
SURVEY
2.1 Introduction
For the past two decades, there has been increasing interest among researchers
in problems related to neural network technology and artificial neural networks. A
neural network is a computational model based on some aspects of neurobiology and
that could perform “smart” tasks similar to those performed by the human brain.
As described in [9], a neural network is a large scale analog circuit which processes
signals in real time. Few important characteristics of neural networks are asyn-
chronous parallel processing, continuous-time dynamics, and global interaction of
network elements. This subject has attracted immense research interest due to the
challenging nature of the problem and owing to many of its impressive applications
that have been proposed for various fields such as linear and non-linear program-
ming, optimization, associative memory, pattern recognition, speech recognition and
computer vision.
As described in [38], the evolution (in time) of the states of an array of neuron-
like elements toward their equilibrium points can be interpreted as the evolution of
an imperfect pattern toward the correct/stored pattern, and it is comparable to the
recollection of information in an associative memory. An associative memory is a
content addressable structure that maps specific input representations to specific
output representations. In other words, if a pattern is presented to an associative
7
memory, it returns the pattern stored which is “closer” to the one presented. The two
main requirements for the associative memories are first that every given memory
should be an equilibrium of the network and second that the equilibria corresponding
to the memory vectors have to be asymptotically stable. Artificial neural networks
have been used as a model for associative memory since the 80s, and study of such
systems has been of great interest to many researchers in recent years [38]. One of
the most fascinating property of associative memories is their capability to tolerate
noisy and/or partial input in recalling the information.
Extensive research on synthesis of associative memory has been done and
several methods that make use of Hopfield or similar type of neural networks have
been proposed[1, 6, 20, 21, 22, 55, 23, 7, 37, 38, 39, 40, 53]. Hopfield networks are a
special kind of recurrent neural networks that have been shown to implement asso-
ciative memories effectively. They use Hebbian rule [19] to make connection matrix
(weight matrix) store a set of patterns and they have at most storage capacity of
roughly 0.15 times the number of neurons. The network design method proposed by
Hopfield [20, 21], considered that the memory patterns to be stored are represented
by vectors at which an energy function for the network assumes a local minimum.
Thus, the patterns similar to a given stored pattern will approach this stored pat-
tern, since the network is designed such that all of its trajectories seek the local
minima of the energy function. However, implementing an associative memory is
not an easy task because storing each desired pattern as a vector at which an energy
function for the network assumes a local minimum (i.e., storing each desired pattern
as an asymptotically stable equilibrium of the network) is difficult. Moreover, con-
trolling the extent of the basins of attraction of each stored pattern [11, 26, 27, 46]
and minimizing the number of extraneous stored patterns, called spurious states are
two challenging tasks.
8
In literature [24, 26, 42, 40, 44, 43, 53, 41, 50, 60, 61, 28, 29, 17], many de-
sign techniques have been proposed for continuous-time and discrete time neural
networks to realize associative memories. For example, the outer product method
consents for adding new storage patterns and there is no need to recompute the
entire interconnection matrix T. However, restrictions such as only few desired pat-
terns can be stored as memory vectors and the limited capacity which allows to store
only 0.15N asymptotically stable equilibrium points, pose difficulty in implementa-
tion. Another design technique, called the projection learning rule guarantees that a
network designed by this method will always store a given vector as an equilibrium
point (which may not be asymptotically stable). The network can store effectively
up to N vectors as equilibria, where N is order of the network. Whereas the eigen-
structure method [42, 43, 44] allows the network to store equilibrium points which
outnumber the order of the network, and also guarantees that all of the desired
patterns are stored as asymptotically stable equilibrium points. The synthesis of
discrete time neural networks described by hupercubes [43, 45] provides a means
of implementing neural network by serial processors and special digital hardware.
Networks designed by the outer product method, the projection learning rule and
the eigen-structure method are globally stable (i.e., all trajectories converge to some
equilibrium point) and they have a symmetric interconnecting structure. Later work
by Michell and Farrell [40, 39, 38] yields networks with non-symmetric interconnec-
tion matrix that guarantee to store all desired patterns as asymptotically stable
equilibrium points. However these are not globally stable networks and can store
up to N linearly independent patterns corresponding to asymptotically stable equi-
librium points.
Another interesting topic is the problem of realizing associative memories
via cellular neural networks (cnns). A cnn is a special class of neural network in
which every cell is connected to only neighboring units. Here, a cell is the basic
9
circuit unit of cnn and it contains linear and non-linear circuit elements, which
typically are linear capacitors and resistors, linear and non-linear controlled sources
and independent sources. A cnn’s continuous time feature allows real-time signal
processing and its local interconnection feature makes it suitable for VLSI imple-
mentation. Many design algorithms [34, 35, 36] which yield cnn-based associative
memories with symmetric weight matrices have been proposed in literature and
eigen structure method has been generalized for the synthesis of cnns. However,
as mentioned in [47], designing neural associative memories based only on stability
criteria does not yield networks with satisfactory performance and additional infor-
mation is needed to address other performance indices such as the size of attraction
domain for each desired memory vector and number of spurious patterns. This the-
sis builds on the work of [47] and addresses the problem of finding the parameters of
the cnns that can work as associative memories with improved performance. The
designed network has less spurious patterns and larger attraction domains for the
desired memory vectors. Further, we selectively “trim” network links in an effort
to trade network performance for smaller communication cost. In the next section,
we explain the architecture and non-linear dynamics of cnn as described in [9], and
Figure 2.1: A two-dimensional cnn of size 4 × 4 with the squares representingbasic building blocks called cells and solid lines representing the inter-connection links.
10
define the Lyapunov function and equilibrium points of a cnn.
Section 2.2 presents the cnn circuit architecture and in-depth analysis of its
dynamic properties. This is followed by explanation of the stability, attractiveness
and robustness properties of the cnn model. In Section 2.4, we briefly discuss the
different analysis and design techniques provided in literature, including sparse cnn
structure and its significance. Finally impact of time delays in assosciative memory
is discussed in Section 2.5.
2.2 The cnn basics
A cnn is a network of regularly spaced circuit clones called cells (shown in
Fig. 2.1). As seen in the figure, any cell in a cnn is connected only to its neighboring
cells. However, because of the propagation effects of the continuous-time dynamics
of the cnn, cells not directly connected together may affect each other indirectly.
For a M ×N cnn having M rows and N columns, the cell on the ith row and jth
column is denoted by C(i, j) and its r-neighborhood (r is positive integer number)
is defined by:
Nr(i, j) = C(k, l) | max| k − i |, | l − j | ≤ r, 1 ≤ k ≤M ; 1 ≤ l ≤ N (2.1)
Figure 2.2: A circuit representation of cell C(i, j) in a cnn.
11
Figure 2.2 shows circuit representation of a cell in cnn, where u, x and y
denote input, state and output respectively of the cell. The node voltage vxij of cell
C(i, j) is called state of the cell and its initial condition is assumed to be less than or
equal to one. The node voltage vuij is called the input of cell C(i, j) and is assumed
to be constant with magnitude less than or equal to one. The node voltage vyij is
called the output, Eij is the independent voltage source and I is the independent
current source.
State equation:
Cdvxij(t)
dt= − 1
Rx
vxij(t) + ΣC(k,l)∈Nr(i,j)A(i, j; k, l)vykl(t)
+ΣC(k,l)∈Nr(i,j)B(i, j; k, l)vukl(t) + I (2.2a)
Output equation:
vyij(t) =1
2(| xij(t) + 1 | − | xij(t)− 1 |) (2.2b)
Input equation:
vuij = Eij (2.2c)
Constraint equation:
| vxij(0) |≤ 1, | vuij |≤ 1 (2.2d)
Parameter Assumptions:
A(i, j; k, l) = A(k, l; i, j), C > 0, Rx > 0 (2.2e)
Each cell C(i, j) contains one independent voltage source Eij, one indepen-
dent current source I, one linear capacitor C, two linear resistors Rx and Ry,
and at most 2m linear voltage controlled current sources which are coupled to its
neighbor cells via the controlling input voltage vukl , and the feedback from the
output voltage vykl of each neighbor cell C(k, l), where m is equal to number of
12
neighbors. Also Ixy(i, j; k, l)=A(i, j; k, l)vykl and Ixu(i, j; k, l)=B(i, j; k, l)vukl for all
C(k, l) ∈ Nr(i, j). A cnn is completely characterized by the set of all non-linear
differential equations associated with the cells in the circuit [9].
Furthermore, by assuming that B(i, j; k, l) = 0 in ( 2.2a), the state of the
cnn can be expressed as
dvxijdt
= − 1
Rx
vxij + µA(i, j; k, l)f(vxij) + I (2.3)
or in the formdvxijdt
= g1(vxij , t, µ). (2.4)
Then, by definition, the equilibrium points of the system (2.4) will be v∗xij , for which
g1(v∗xij , t, µ) = 0. Thus in 0 ≤ vxij(t) ≤ 1, the equilibrium points of a typical cell of
a cnn C(i, j) are defined as
v∗xij =IRx
1− µARx
, 1 ≤ i ≤M, 1 ≤ j ≤ N. (2.5)
Based on definition of equilibrium point of a cnn, the stable system equilib-
rium point of a cnn is a state vector with components v∗xij , 1 ≤ i ≤ M, 1 ≤ j ≤ N ,
for which µ < 1A(i,j;k,l)Rx
[54]. In matrix vector form the above system can be repre-
sented as shown in [9] by
x = −x+ T sat (x) + b, (2.6a)
y = sat (x) (2.6b)
where, x = [x11, x12, . . . , x21, . . . , xMN ]T ∈ Rn is the stack vector of all neuron states,
y = [y11, y12, . . . , y21, . . . , yMN ]T ∈ Hn is the output vector (Hn is the n-dimensional
hypercube [−1,+1]n), T = [Tij] ∈ Rn×n is the network connection weight matrix,
b ∈ Rn is the network’s bias vector and for vector arguments, the saturation function
is defined elementwise as sat (x) , 12
(|x+ 1| − |x− 1|).
If Bn represents the set of bipolar vectors in Hn, namely those whose elements
are either +1 or −1, then for i = 1, . . . , n, the initial condition vectors of (2.6) should
13
always satisfy |xi(0)| ≤ 1. The interconnection topology information of the network
can be described by an adjacency matrix S, and thus a weight Tij is non-zero only
if Sij = 1. Further the qualitative properties of the cnn model are described as:
1. xequi ∈ Rn is an equilibrium point of (2.6) if x(0) = xequi implies x(t) = xequi,
for all time t > 0.
2. Equilibrium point xequi of (2.6) is stable if, for any ε > 0, there exits δ > 0
such that
‖x(0)− xequi‖ < δ implies that ‖x(t)− xequi‖ < ε,for all t > 0.
3. An equilibrium point xequi of (2.6) is asymptotically stable if it is stable and
there exists δ > 0 such that
x(t)→ xequi as t→∞ if ‖x(0)− xequi‖ < δ.
4. α ∈ Hn is a memory vector of system (2.6) if there exists an asymptotically
stable equilibrium point β ∈ Rn of above system such that α = sat(β).
5. System (2.6) is globally stable if, for every initial condition x(0) ∈ Hn, the
system output state converges to some equilibrium point.
In next section, we further explain the stability of cnn in detail followed by the
attractiveness and robustness properties.
2.3 Stability, attractiveness and robustness
cnns are not only important to researchers because of their collective dynam-
ics but also because they find applications [16] in performing parallel computation.
In carrying out the computation, the cnns are subjected to various stochastic per-
turbations and it is crucial to understand how these perturbations affect the network.
Consequently, it is very important to know whether the networks are stable or not
under the perturbations. As a result, the stability criteria for cnn have become an
attractive research problem of prime importance.
14
2.3.1 Quantification of stability properties
Generally speaking, a dynamical system is said to be stable if the response
to an external system is bounded. As explained in Section 2.2, if the trajectory
of a cnn starting from initial state remains near equilibrium state, than cnn is
said to be stable. If it tends to return to the equilibrium state, cnn is said to be
asymptotically stable. Furthermore, stable system equilibrium point of a cnn is a
state vector with components v∗xij , 1 ≤ i ≤M , 1 ≤ j ≤ N , for which µ < 1A(i,j;k,l)Rx
and limt−>∞vyij(t) = 0 or 1 [54]. Since any stable neural network system equilibrium
point is a limit point of a set of trajectories of ( 2.3), such an attracting limit point
is said to have a basin of attraction. In other words, parameter µ determines the
limit points and basins of attraction of a cnn and in turn can affect the stability
of cnn. More research work on the estimation of the basin of attraction for the
stable equilibrium points in a cnn can be found in [46], where the authors suggest a
method based on determining the so-called tree of regions connected with each stable
equilibrium point, and which gives more insight into how the basins of attractions
are situated. The method is also useful in determining which basins of attraction
this (binary) initial condition belongs to.
Another important and trusted technique for analyzing the stability of cnn
is Lyapunov’s method by determining an appropriate Lyapunov function [9, 54] for
a cnn. The Lyapunov function E(t) of a cnn represented by ( 2.7) is an indicative
of stability of the system and is defined by the scalar function:
E(t) = −1
2
∑ij
∑kl
A(i, j; k, l)vyij(t)vykl(t) +1
2Rx
∑ij
vyij(t)2
−∑ij
∑kl
B(i, j; k, l)vyij(t)vukl −∑ij
Ivyij(t). (2.7)
For the system to be stable, E(t) must be bounded and should be a monotonically
decreasing function.
15
Significant amount of research has been done on establishing relation between
the system parameter matrix values and the stability of system. For example, [33]
shows that with each diagonal element of weight T set to 1, the following properties
can be established:
1. System has only binary memory vectors.
2. For any memory vector α ∈ Bn, none of the vertices γ ∈ Bn with Hamming
distance H(γ, α) = 1 can be memory vector and if x(0) = γ, H(γ, α) = 1, and
if γ and α differ in the ith bit, then xi(t) moves towards αi at t = 0.
cnns satisfying these diagonal constraint usually have less spurious patterns and
larger attraction domain for desired memory vectors than other cases. According to
the stability analysis of cnn by [34, 35, 36], if α ∈ Bn and β , Tα+ b are such that
αiβi = αi(∑j=1ton
Tijαj + bi) > 1, i = 1, ..., n (2.8)
then (α, β) is a pair of memory vector and an asymptotically stable equilibrium
point of the system, Furthermore, if α ∈ Bn and β , Tα + b are such that
αiβi = αi(∑j=1ton
Tijαj + bi) < 1, i = 1, ..., n (2.9)
then α ∈ Bn cannot be a memory vector.
Another interesting approach found in literature for quantifying stability of
cnn is the notion of the stability parameter. The concept of stability parameters
appears in some earlier work on associative memory networks [2, 30, 15], and is
proposed as a measure of quality of memorization. Specifically, it has been demon-
strated [13, 27] that the sign as well as the magnitude of these numbers are related
to the size of the attraction regions of the desired memory patters. Although the
applicability of this concept as a universal measure of memorization quality, (specif-
ically, the use of their size as a direct measure of the absolute sizes of the attraction
16
regions) has long been debated [11], these parameters are generally accepted as a
reasonably good metric for the network’s performance [31]. In this thesis, we use
the magnitude of the stability parameters Kiµ
Kiµ = α(µ)i hi =
α(µ)i
∑j cijα
(µ)j
||ci||2, (2.10)
to quantify the quality of memorization in the network, where cij = Tij, ‖ci‖ =√∑j c
2ij, and superscipt (·)(µ) indexes the set of desired memory vectors.
The stability parameters and the degree of symmetry of connection matrix
affect the system dynamics and the attraction domain sizes. Although positive
values for the stability parameters indicate a stable system, one has to be careful
in assigning further physical meaning to their value, and using them for comparing
the performance of the neural network.
2.3.2 Robustness
Because of the possible perturbations and implementation errors, it is essen-
tial to consider the perturbed form of ( 2.6)
x = −(In + ∆A)x+ (T + ∆T )sat (x) + (b+ ∆b), (2.11a)
y = sat (x) (2.11b)
where ∆A = diag[∆a1 . . .∆an] with 1 + ∆ai > 0 for i = 1, . . . , n, ∆T ∈ Rn×n, and
∆b ∈ RN . In 1996, [36] established that if α ∈ Bn is memory vector of system ( 2.6),
and if αi(∑
j=1ton Tijαj + bi) > ξ > 0, i = 1, ..., n, then α is also the memory vector
of the perturbed system ( 2.11) if the following condition [47, 33] is satisfied:
‖∆A‖+ ‖∆T‖+ ‖∆b‖ < ξ. (2.12)
A metric called average recall probability [36, 47] is a measure of performance
of the designed network in presence of noise and is defined as the ratio of number
of recovered memory patterns (perturbed initial condition vectors which result in
17
same output as the stored memory vector) to the total number of perturbed ini-
tial condition vectors. Although this quantification method measures the network’s
performance accurately and unambiguously, one shortcoming is that it can only be
applied after the network has been designed and tested on a significant number of
test inputs. In other words, it does not allow the designer to predict the network’s
performance without experimentation. This motivates the use of a different per-
formance metric which can be utilized in the design process. Toward this end, we
selected and used the networks stability parameters defined by ( 2.10).
2.4 Analysis and design
In many past approaches on the design of cnns, some network topology is
given to start with and build on. For example, [36, 35] generalized the eigenstructure
method for the synthesis of non-symmetric cellular neural network with a predeter-
mined local interconnection structure that can store a set of desired bipolar vectors
as memory points. The linear combination η of the desired memory vectors is also
a reachable memory vector of the synthesized network. This design method was
extended to sparsely connected neural networks, provided that the interconnecting
structure includes self feedback for all neurons. Similarly [47] started with a given
interconnection structure of a cnn, characterized by the index matrix S, and a
given set of desired binary memory vectors α1, ..., αm ∈ Bn, to design the synthesis
problem to derive the parameters (Tij and bi values) of a cnn with improved perfor-
mance. The optimization problem had non-linear constraints and was transformed
to a gevp problem which was solved using lmi tool box in matlab.
The fact that typically the physical platform on which the network is im-
plemented is given, justifies the choice of the network’s topology. In this work, we
try to answer the question of how could the design process be different if the net-
work topology was also part of the design. In scenarios where implementing and
maintaining certain neuron connections are more expensive than others, one may be
18
forced to strike a balance between connectivity cost and network performance. Sim-
ple stability analysis of neural networks implementing associative memories suggests
similar links between the network of connections and the stability of the dynami-
cal system as in consensus networks. However, in the case of neural networks, the
topologies used are typically standard: either complete graphs (in Hopfield net-
works) [20, 22, 55, 23] or grid-like structures (in cnns). cnns have sparse intercon-
nections; sparse (or diluted) associative neural networks attract a lot of attention
for a number or reasons: they are more suitable for hardware implementation than
their fully connected counterparts; they require less memory and are faster to oper-
ate during computer simulations; biological neural networks like neurons in human
brain seem to resemble sparse networks. Sparsity can enable spatially distributed
implementations (parts of the system in different places) and it makes the system
more robust.
Yet, available design methodologies do not place specific conditions on the
network’s structure. In addition, the cost of communication (delayed or otherwise)
between neurons is commonly ignored. However if neurons communicate at a non-
trivial cost, there is a cost benefit in designing networks that perform just as well,
but with less communication. This thesis suggests a method to optimize the net-
work topology of cnns in which communication links between neurons may incur
variable cost by selectively “trimming” network links in an effort to trade network
performance for smaller communication cost. Building on existing efficient design
tools, we perform combinatorial optimization on a portion of a cnn that includes
the most expensive interconnection links. This produces a sparser cnn, the perfor-
mance of which can be comparable to the original network. We tested two different
approaches to performing the combinatorial optimization. The first one is a sequen-
tial process in which a single link is removed each time, specifically the one the
removal of which incurs the least performance cost compared to all other existing
19
high-cost links. Although the search in each step of this process is exhaustive, it has
to be noted that this method ignores the possibility that a non-obvious combination
of links may produce better results through the link’s simultaneous removal. This
phenomenon has been observed in simulation studies which validated the proposed
method. To validate further the optimization, but more importantly, to ensure that
the overall approach does not depend on the particular method used for the combi-
natorial optimization we also implemented an alternative approach which is based
on the randomized optimization [57]. In this approach a random sample of a suffi-
cient number of i.i.d possible topology is generated. In other words, each random
topology in the sample has the same probability distribution as the others and all are
mutually independent. In our numerical tests, the two methods yield comparable
results, which do not differ significantly in terms of resulting network performance.
Performance is quantified in terms of the network recall probability, and in the pro-
posed optimization algorithm approach is captured by the neural network’s stability
parameters. On the other hand, a naive implementation of a “branch-and-bound”
approach, where links are divided into “promising” and “not-promising” for deletion
groups according to their associated K value, and only the “promising” possibilities
are explored in the subsequent steps, will generally fail. This is because due to the
combinatorial nature of the problem, an edge whose sole deletion has an adverse
effect on the stability parameters may even improve the value of K when combined
with additional edge removals. Experiments are performed on real-time hardware in
loop corobot platforms where three co-robots communicate among each other using
the proposed algorithm.
2.5 Impact of time delays in associative memories
This section focusses on effects of time delays in cnns, a brief overview and
possible direction for future research should be discussed. Time delays are often
encountered in various cellular neural networks (introduced by switching speed of
20
amplifiers and communication time in electronic circuits of cnn)and often result
in oscillations, instability, and poor performance of networks. In particular, having
an upper bound on the time delay that the network can afford is important both
theoretically and application-wise and the problem has been studied extensively
[56, 18, 59, 8]. cnns with delay τ (dcnns) [52] are described by the state equations
of the form
Cdvxij(t)
dt= − 1
Rx
vxij(t) +∑
C(k,l)∈Nr(i,j)
A(i, j; k, l)vykl(t) +∑
C(k,l)∈Nr(i,j)
(2.13a)
AT (i, j; k, l)vykl(t− τ) +∑
C(k,l)∈Nr(i,j)
B(i, j; k, l)vukl(t)
+I
The output equation is
vyij(t) =1
2(| xij(t) + 1 | − | xij(t)− 1 |) (2.13b)
while the input is
vuij = Eij = constant. (2.13c)
In the above, Nr(i, j) is the r neighborhood of cell C(i, j). The space invari-
ance property for dcnn is expressed by:
A(i, j; k, l) = A(i− k, j − l) (2.14a)
B(i, j; k, l) = B(i− k, j − l) (2.14b)
AT (i, j; k, l) = AT (i− k, j − l) (2.14c)
where A0 and A1 are the cloning templates and AT is the delay cloning template.
Assuming Rx = C = 1, (2.13) can be represented compactly as
x(t) = −x(t) + A0y(t) + A1y(t− τ) +Bu+ I. (2.15)
21
The symmetry of DCNN is dependent on symmetry of the network when τ = 0 or
on symmetry of A0 + A1. For in-depth explanation of dynamics of dcnns readers
are to referred to [10].
The stability properties of dcnn can be analyzed using a suitable Lyapunov
function. A Lyapunov function for a dcnn can be given as
V (xt) = y′(t)Py(t)−∫ 0
−τ[y′(t+ θ)− y′(t)]
A′1f(θ)A1[y(t+ θ)− y(t)]dθ (2.16)
where f(θ) is a scalar function continuous with it derivative on [−τ, 0], f ∈ C1([−τ, 0],R)
and P = −I + A0 + A1. The Lyapunov function has a constant limit when time
tends to infinity. A sufficient condition to ensure complete stability of delay sym-
metric dcnn for any initial condition ∈ C([−τ, 0], RM×N), is that A1 is invertible,
A1 < 2/3τ and P symmetric.
Later work on global stability of dcnns [32] was based on the Lyapunov
technique and LaSalle’s invariance principle as well as some facts of positive definite
matrices and matrix inequalities. The authors of [32] assumed that the (2.15) has
an equilibrium point x∗ for a given u, which is shifted to the origin. Let us define
z(t) = x(t)− x∗ and the (2.15) becomes
z(t) = −z(t) + A0φ(z(t)) + A1φ(z(t− τ)) (2.17)
where φi(zi(t)) = yi(zi(t) + x∗i ) − yi(x∗i ) and φi(0) = 0. For a DCNN defined by
(2.17), the origin is the unique equilibrium point and it is globally asymptotically
stable if there is a constant β ≥ 0 such that the following conditions are true:
1. −(A+ A1 = βI) is positive definite.
2. ‖A1‖2 ≤√
1 + β.
Parameter β approximately compensates for the trade-off between the matrix defini-
titeness condition on the feedback matrix, and the norm inequality condition on the
22
delayed feedback matrix. The less restrictive the norm condition on the feedback
matrix A1 is with respect to β, the less definiteness is required on the matrix A+A1.
The stability condition is not affected by the delay parameter. The problem of global
robust stability analysis for a class of uncertain delayed cnns with discrete and dis-
tributed delays was studied [47], and a novel stability criterion based on Lyapunov
stability theory and the LMI framework was derived. In this thesis, we ignore the
delays, assuming that they do not have any impact on system stability. However,
it would be interesting to investigate also the possibility of extending the results to
more general delay models, such as time-varying or distributed ones.
23
Chapter 3
COMBINED DESIGN OF TOPOLOGY AND LINK
WEIGHTS
In this chapter, a new network topology optimization approach for cnns has
been formulated. This type of optimization allows recurrent neural networks to
be implemented in a spatially distributed fashion, that is, with components of the
network residing in different physical locations. In this approach the number of
neural links connecting the spatially distributed subnetworks, assumed expensive
to implement and maintain are minimized. The cnn’s interconnection topology is
diluted without significantly degrading its performance. Performance is quantified
by the average recall probability of the patterns engraved into the network’s asso-
ciative memory. As explained in Chapter 2, the average recall probability [36, 47]
is a measure of performance of the designed network in presence of noise and is
defined as the ratio of number of recovered memory patterns (perturbed initial con-
dition vectors which result in same output as the stored memory vector) to the total
number of perturbed initial condition vectors. Since the average recall probability
cannot be assessed prior to testing, the optimization algorithm uses the network’s
stability parameters as a measure of quality of memorization, and optimization pro-
ceeds by selectively removing those links connecting distant network components,
which contribute the least to the magnitude of these parameters. Two different ap-
proaches to implementing the optimization of the network’s topology are compared.
In the following chapters, an example is used to demonstrate that irrespectively of
24
the combinatorial optimization algorithm used, the proposed method yields sparser
associative memories that in general trade off performance for cost. In many cases
the performance of the diluted network is on par with the original system. Sim-
ulations and experiments on mobile robots communicating wirelessly, validate the
performance of the designed associative memories.
This Chapter is organized as follows: Section 3.1 briefly introduces the fun-
damentals of cnns, and formulates the synthesis problem for cnns as a constrained
optimization problem. In Section 3.2, we solve the topology optimization problem
by diluting the original network, represented by a weighted adjacency matrix S, by
judiciously removing some of the long distance links identified in S, in a way that
the quality of memorization is not severely affected. This section is divided into two
subsections. Subsection 3.2.1 aims at the development of an optimization method
where we iteratively “trim” that particular long distance link whose deletion results
in the minimum decrease in the value of our objective function. Subsection 3.2.2
focusses on selecting the desired topology out of a sufficiently large sample of com-
binations of deleted links using a randomized algorithm that returns a network
topology achieving a probable near maxima of the objective function. Finally in
Section 3.3, concluding remarks are given.
3.1 Preliminaries and mathematical model
Consider a two-dimensional, continuous time zero-input M × N cnn [9] in-
troduced in Section 2.2 and represented by (2.4). The dynamics of this system can
be represented mathematically as
xij = −xij +∑
C(k,l)∈Nr(i,j)
Wij,klykl + dij, yij = sat (xij) , (3.1)
where 1 ≤ i ≤ M , 1 ≤ j ≤ N , and sat (xij) , 12
(|xij + 1| − |xij − 1|) . Here xij
and yij are the state and output of the (i, j)th cell respectively, and Nr(i, j) is an r-
neighborhood of the (i, j)th cell defined asNr(i, j) , {(k, l) : max {|k − i| , |l − j|} ≤
25
r}, for 1 ≤ k ≤ M, 1 ≤ l ≤ N . Wij,kl represents the connection weight from cell
C(k, l) to cell C(i, j) and dij is the bias term for cell C(i, j). With W = [Wij,kl] ∈
RMN×MN expressed as an n × n matrix T = [Tij] = W and with d = [dij] ∈ RMN
expressed as an n dimensional vector b = [bi] = d, (3.1) can be expressed in matrix
form by equation (2.6) as
x = −x+ T sat (x) + b,
y = sat (x)
where x = [x11, x12, . . . , x21, . . . , xMN ]T ∈ Rn is the stack vector of all neuron states ,
y = [y11, y12, . . . , y21, . . . , yMN ]T ∈ Hn is the output vector (Hn is the n-dimensional
hypercube [−1,+1]n), T = [Tij] ∈ Rn×n is the network connection weight matrix,
b ∈ Rn is the network’s bias vector and for vector arguments, the saturation function
is defined elementwise.
If Bn represents the set of bipolar vectors in Hn, namely those whose elements
are either +1 or −1, then for i = 1, . . . , n, the initial condition vectors of (2.6) should
always satisfy |xi(0)| ≤ 1. The interconnection topology information of the network
can be described by an adjacency matrix S, with the understanding that a weight
Tij is non-zero only if Sij = 1. Vector α ∈ Hn is a memory vector for (2.6) if the
latter has an asymptotically stable equilibrium point β ∈ Rn such that α = sat (β)
[9].
The synthesis problem for a cnn can be stated as follows:
Problem 1 (Synthesis): Given a cnn interconnection structure (2.6), ex-
pressed by an adjacency matrix S, and the set of desired bipolar memory vectors
α1, . . . αm ∈ Bn, find the network weights Tij and bias parameters bi so that the
obtained neural network can store all desired memory patterns.
The adjacency matrix S determines which Tij, typically encodes a lattice
structure (a weight Tij is non-zero only if Sij = 1) and can be selected arbitrarily
26
as long as as long as certain conditions are satisfied [9, 34, 47]. If α ∈ Bn and
β = Tα + b are such that
αiβi = αi( n∑j=1
Tijαj + bi)> 1, ∀i = 1, . . . , n,
then (α, β) is a pair of a memory vector and an asymptotically stable equilibrium
point of system (2.6). Furthermore if α ∈ Bn and β = Tα + b are such that for
any i = 1, . . . , n, αiβi = αi(∑n
j=1 Tijαj + bi)< 1, then α ∈ Bn cannot be a memory
vector. System (2.6) is globally stable if T is symmetric.
In addition to the aforementioned stability criterion, specific stability and
robustness properties for these networks are established in terms of the elements of
the network’s weight matrix T and bias vector b [9, 34, 47]:
Let α ∈ Bn be a memory vector of system (2.6) and let k ≥ 1 be an integer.
If T = T − In and b satisfy
αi
(n∑j=1
(Tijαj + bi)
)> 2(k − 1) max
1≤j≤n
∣∣∣Tij∣∣∣ , (3.3)
for i = 1, . . . , n, then any binary vector α∗ ∈ Bn such that 1 ≤ h(α∗, α) ≤ k
(h(α∗, α) ,∑
i |α∗i−αi| denotes the Hamming distance) has the following properties
[47]:
1. α∗ is not a memory vector (asymptotically stable equilibrium point for (2.6)).
2. if x(0) = α∗ and α∗i 6= αi, then xi(t) converges to αi.
The proof of above theorem is provided in appendix 6.3.
Perturbed system [36, 47] which takes into account the implementation error
can be described by
x = −(In + ∆A)x+ (T + ∆T )sat(x) + (b+ ∆b), y = sat(x) (3.4)
27
where ∆A = diag[∆a1, . . . ,∆an] with 1 + ∆ai > 0, i = 1, . . . , n, ∆T ∈ Rn×n and
∆b ∈ Rn. A memory vector α of the (2.6) which satisfies αi
(∑nj=1(Tijαj + bi)
)>
µ > 0, i = 1, . . . , n will also be a memory vector of (3.4) provided that ‖A‖infty +
‖T‖infty + ‖b‖infty < µ. From (3.3) and above robustness condition, authors of
[36, 47] concluded that increasing k results in increase of both the attractivity and
the robustness of the stored memory vector α ∈ Bn and decrease of the probability
of existence of spurious patterns in vertices near α.
The synthesis problem can be formulated as a gevp [47]. For i, j = 1, . . . , n,
and k = 1, . . . ,m,
min(−δ), s.t. (3.5a)
(−δ)diag[2q1, . . . , 2qn]− diag[−p1, . . . ,−pn] > 0 (3.5b)
α(k)i
(n∑j=1
Tijα(k)j + bi
)− pi > 0, (3.5c)
qi − Tij > 0, (3.5d)
Tij + qi > 0, (3.5e)
Tii = 0, , (3.5f)
Tij = T Tij = Tij|S (3.5g)
L < qi < U, (3.5h)
where pi and qi for i = 1, . . . , n are additional “slack” variables used to cast the
design problem as a lmi [5], and L and U are the lower and upper bounds for the
design variables in the gevp.
As explained in Section 2.3, we use the magnitude of the stability parameters
Kiµ to quantify the quality of memorization in the network,
Kiµ = α(µ)i hi =
α(µ)i
∑j cijα
(µ)j
||ci||2,
28
where cij = Tij, ‖ci‖ =√∑
j c2ij, and superscipt (·)(µ) indexes the set of desired
memory vectors. The stability parameters Kiµ, (one for each pair of neuron node
and memory vector) are numbers which have been proposed as a measure of quality
of memorization and it has been hypothesized that they are linked to the size of the
attraction regions of the neural network [31, 15, 11]. The magnitude of the stabil-
ity parameters and the degree of symmetry of the connection matrix T affect the
properties of the attractors in the system dynamics. Positive values for the stability
parameters is an indication that a pattern corresponds to a locally asymptotically
stable equilibrium.
3.2 Design method
This thesis suggests a method to optimize the network topology of cnns in
which communication links between neurons may incur variable cost. The method
selectively “trims” network links in an effort to trade network performance for
smaller communication cost. Building on existing efficient design tools (computed
T and b through gevp), we perform combinatorial optimization on a portion of a
cnn that includes the most expensive interconnection links. This produces a sparser
cnn, the performance of which can be comparable to the original network.
A cnn is considered as a collection of sub-networks wherein information flow
across sub-networks is more expensive compared to communication within each sub-
network. We are interested in minimizing communication cost, while maintaining
the functionality and performance of the whole network above a certain thresh-
old. Given the (unweighted) adjacency matrix S of the cnn, along with the set
of desired bipolar memory vectors α1, ...αm ∈ Bn, the first step is to determine the
network parameters Tij and bi through the solution of the gevp (3.5). The resulting
network maximizes the recall probability of the patterns it has been designed for,
without considering the cost of using the different network links. The next step is
to dilute the connectivity of S, by removing some of the expensive links identified
29
in S, in a way so that the quality of memorization is not severely affected. The
communication costs are captured by the weights of the weighted adjacency matrix
S. Balancing performance against communication cost is achieved through (combi-
natorial) optimization over the network links, subject to the stability constraints in
(2.10).
The problem addressed in this thesis is the following variant of Problem 1:
Problem 2 (Topology optimization): Given (2.6), implemented on network
expressed by a weighted graph G with adjacency matrix S, with the set of desired
bipolar memory vectors α1, . . . αm ∈ Bn, determine the connection weights Tij and
bias parameters bi of a subgraph G, so that the cnn on the subgraph stores all
desired memory patterns as reachable memory vectors, and recalls them (almost)
as well as that implemented on original network.
We implemented two different approaches to performing the combinatorial
optimization. The first one is a sequential process in which a single link is removed
each time, specifically the one the removal of which incurs the smaller performance
reduction compared to all other existing high-cost links. With the desire to validate
the result of the first optimization method and to verify that the dilution concept
applies regardless of how the topology is optimized, we also implemented an alter-
native approach which is based on the randomized optimization. In this approach
a random sample of a sufficient number of i.i.d possible topologies is generated.
The two design procedures based on combinatorial optimization of cellular neural
network topology is described in following two subsections.
3.2.1 The sequential trimming method
The optimization process is iterative. For a given (intermediate) topology
S, the algorithm determines the neural network weights Tij and biases bi for i, j =
1 . . . , n, and based on the selected patterns α(µ)i for µ = 1, . . . ,m to be memorized,
an n × m stability parameter matrix is formed by repeated application of (2.10):
30
K = [Kiµ]i=1,...,n;µ=1,...,m. For the given patterns α1...αm to be stored effectively in
the network’s memory, all stability parameters Kiµ must be nonnegative and as
large as possible. For the network topology encoded in S, the value of the following
objective function is evaluated:
K =n∑i=1
m∑µ=1
Kiµ. (3.6)
Thus K, being the sum of all nodes’ stability parameters for all chosen memory
vectors, quantifies the collective ability of the network to recall all desired memories.
Remark 1 : Several different performance metrics based on different norms of
the stability parameter matrix K, such as the minimum row (or column) sums, the
(absolute) minimum element of K, etc, have been tested as alternatives to (3.6).
When the average recall probability of each design was evaluated, it was determined
that the sum of all stability parameters captured more accurately the ability of the
network to recall memory patters.
A naive implementation of a “branch-and-bound” approach, where links are
divided into “promising” and “not-promising” for deletion groups according to their
associated K value, and only the “promising” possibilities are explored in the sub-
sequent steps, will generally fail. This is because due to the combinatorial nature
of the problem, an edge whose sole deletion has an adverse effect on the stability
parameters may even improve the value of K when combined with additional edge
removals.
In detail, for a given cost threshold ν, and performance threshold κ, the high-
cost edges that are candidates for deletion are identified in the residual adjacency
matrix
R =1
2
(sign(S − νS) + S
),
where the sign function is evaluated element-wise on the matrix argument. Since
every element in the index matrix is either 1 or 0, a cost threshold value of ν = 2
31
Algorithm 1 Topology optimization through sequential dilution
Require: Matrices α, S, S, constants ν, κ.Ensure: Matrix A of K values for each edge that may be removed.
1: R← 12
(sign(S − νS) + S
).
2: n← rowlength(α)3: m← columnlength(α)4: A← [0]n×m5: C ← [0]n×m6: while max(R) 6= 0 do7: For each (i, j) such that R(i, j) 6= 0, do8: S ′ ← {S : S(i, j)← 0, S(j, i)← 0}9: Compute T and b, given S ′ and α(µ)
10: K ←∑n
i=1
∑mµ=1Kiµ
11: for µ = 1 to m do12: for t = 1 to n do13: C(t, µ)← α
(µ)t
(∑nj=1 Ttjα
(µ)j + bt
)14: end for15: end for16: if K > κ ∧ minC > 1 then17: A(i, j)← K ; A(j, i)← K18: end if19: R(i, j)← 0; R(j, i)← 020: end while
ensures that when a link is trimmed, the corresponding element in R changes to a
non-zero value. For every nonzero (i, j) element in R, we remove the associated (i, j)
edge from S. If (2.2a) is satisfied then we compute and store the resulting value
of K. The high-cost edge associated with the highest stored K value is marked for
deletion, and the step is repeated for the topology where neurons i and j are not
linked. Further there is a trade-off between the system stability parameter K value
and the number of expensive links trimmed which is decided based on the κ value.
A lower κ value may result in trimming greater number of expensive links.
The pseudo-code in Algorithm 1 gives an outline of the process for evaluating
the edges which are candidates for deletion.
Upon completion, Algorithm 1 provides the (nonzero) performance indices of
32
the network produced after each potential expensive edge which may be trimmed.
The (i, j) edge with the highest K value in A is removed from the network and
the process is repeated until the while loop of Algorithm 1 is no longer executed;
then either all edges with cost above the threshold ν are removed, or their removal
violates the stability condition (3.3), or results in an unacceptable performance
metric. Although, in general, performance deteriorates as more neuron links are
removed, with some modest dilution one may actually increase the objective function
using a sparser topology.
3.2.2 The randomized algorithm method
This section suggests a way of finding probable near maxima of the objective
function K. For this, let us consider a measurable function h : X → R, where X
is a measurable subset of some finite dimensional Euclidean space. For finding the
exact maximum of h(.)
h∗ = supx∈Xh(x) (3.7)
suppose PX is the given probability measure on X, and β ∈ (0, 1) is a given number.
A number h0 ∈ R is said to be a probable near maximum [14] of h(.) to level α if
h∗ ≥ h0, PX{x ∈ X : h(x) > h0} ≤ α. (3.8)
Given the unweighted adjacency matrix S, the weighted adjacency matrix S
and the residual adjacency matrix R, we assign a (uniform) probability of deletion
(1− p) to every non-zero element of R and then draw ξ i.i.d sample matrices where
any element in R may appear with probability p. The integer ξ is selected such that
[57]:
ξ ≥ log(1/δ)
log[1/(1− α)], (3.9)
where, α, δ ∈ (0, 1) are the level and confidence parameters of the randomized
algorithm, respectively. In other words, it can be said with confidence at least 1− δ
33
Algorithm 2 Topology optimization through randomized methods
Require: Matrices α, S, S, p,δ, α, ν, κ = 0.Ensure: Near maxima of K and corresponding topology
1: R← 12
(sign(S − νS) + S
).
2: ξ ← lg(1/δ)lg[1/(1−α)]
3: Row ← rowlength(S)4: Column← columnlength(S)5: n← rowlength(α)6: m← columnlength(α)7: C ← [0]n×m8: while limit < ξ + 1 do9: RandMat← randsrc(n,m, [0, 1; (1− p), p])
10: for countI = 1 to Row do11: for countJ = 1 to Column do12: if R(countI, countJ) = 1 then13: S(countI, countJ)← RandMat(countI, countJ)14: else15: S(countI, countJ)← S(countI, countJ)16: end if17: end for18: end for19: Compute T and b, given S and α(µ)
20: K ←∑n
i=1
∑mµ=1Kiµ
21: for µ = 1 to m do22: for t = 1 to n do23: C(t, µ)← α
(µ)t
(∑nj=1 Ttjα
(µ)j + bt
)24: end for25: end for26: if minC > 1 then27: limit = limit+ 128: end if29: if κ ≤ K then30: κ← K; Topology ← S.31: end if32: end while
that h0 is a probable near maximum to level α. Thus the resulting network given by
S has the same connections as S except that the high cost edges appear randomly.
34
Then we check for the stability condition to make sure that the sample topology
can hold the desired patterns. We repeat the process until we have ξ such random
networks. For each network Si for i = 1toξ, we determine the maximum cumulative
stability parameter using (3.6). Let this be represented by Ki. Then near maximum
of the stability parameter is given by
˘K = max1≤i≤ξKi, (3.10)
The process of random generation of ξ networks and determination of near maxima
of K is pseudo-coded in Algorithm 2.
3.3 Conclusion and future work
In this chapter, a detailed explanation of design of cellular neural network
with sparsity was provided. Two different algorithms to dilute the network topology
were presented with the motive of trading off network performance for communi-
cation cost. In our present work, the adjacency matrix S and diluted network S
both are symmetric in nature. A possible direction of future research may include
performance of non-symmetric cnn. Also the current system is not considering time
delays and hence another interesting area of future research could be optimization
of topology of cnns with time varying delays [49, 48].
In Chapter 4, an example is used to demonstrate that irrespectively of the
combinatorial algorithm used, the approach yields sparser associative memories that
in general trade off performance for cost, and in many cases the performance of
the diluted network is on par with the original system. Furthermore, extensive
simulations are performed to investigate the behavior of the cnn before and after
topology optimization and comparisons are made.
35
Chapter 4
NUMERICAL RESULTS
In previous chapter, we provided a detailed explanation of the design of
sparser cnn such that the topology of the neural network is part of its design.
In this chapter, we will show that the dilution of connectivity of the network can
affect the ability of the neural network to recall patterns. For this, we simulate a
two-dimensional continuous time zero-input 24× 24 cnn represented by (2.6).
This chapter is divided into five sections. In Section 4.1, a design example
is presented to illustrate the validity of the proposed combinatorial optimization al-
gorithms for cnn topology optimization. Section 4.2 focusses on implementation of
sequential trimming method to dilute the original unoptimized cnn and the perfor-
mance of proposed design method is quantified in terms of average recall probability,
as a function of noise contaminating its input. This is followed by Section 4.3 in
which the network topology is optimized by means of an alternative approach called
randomized algorithmic method which guarantees that the probability of finding
a better topology is below a given threshold. A comparative analysis of the two
proposed methods is done. In Section 4.4, we demonstrate the fact that associative
memories with more stored patterns are less robust to noise (more likely to fail to
recollect accurately) compared to the one that stores less information. Finally, Sec-
tion 4.5 concludes this chapter by summarizing our work and presenting remarks on
possible future research work.
36
4.1 A design example
To demonstrate the applicability of the optimization approach proposed in
this thesis, we consider a design example, in which we use a cnn that is made up
of 24 cells interconnected as shown in Figure 4.1.
10 20 30 40 50 60 70 800
10
20
30
40
50
60
70Three sub networks connected by high cost edges
Figure 4.1: A 24 × 24 cnn interconnection structure where high cost edges arestretched and highlighted and the local links are represented by dashedlines.
The interconnection structure of this cnn is represented by adjacency matrix
S, given by
S =
0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 01 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 1 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 1 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 1 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 01 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 01 1 1 0 0 0 1 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 00 1 1 1 0 0 0 1 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 00 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 1 0 0 0 0 0 0 00 0 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 1 0 0 0 0 0 00 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 00 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 00 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 1 0 0 00 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 1 0 00 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 1 00 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 10 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 1 10 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0
. (4.1)
37
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(a) Pattern 1
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(b) Pattern 2
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(c) Pattern 3
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(d) Pattern 4
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(e) Pattern 5
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(f) Pattern 6
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(g) Pattern 7
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(h) Pattern 8
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(i) Pattern 9
1 2 3 4 5 6 7
1
1.5
2
2.5
3
3.5
4
4.5
5
(j) Pattern 10
Figure 4.2: Patterns memorized by the cnn. A first implementation of the networkincluded only patterns 1 through 4. A second included all of them,and the two implementations were compared in terms of performance.
Here, a link is present if S(i, j) = 1 and the neurons are disconnected other-
wise. As we see in Figure 4.1, the network is divided into three small sub-networks
38
which are linked by “expensive” connections marked by solid lines. The subnetworks
separated by the solid edges are assumed to be implemented in different physical
locations. Furthermore, the cost of each connection is represented in form of an
element of a weighted adjacency matrix S by
S =
0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 01 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 00 1 0 2 0 0 0 1 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 00 0 2 0 1 0 0 0 2 1 1 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 1 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 01 1 0 0 0 0 0 1 0 0 0 0 2 2 0 0 0 0 0 0 0 0 0 01 1 1 0 0 0 1 0 1 0 0 0 2 2 2 0 0 0 0 0 0 0 0 00 1 1 2 0 0 0 1 0 1 0 0 0 2 2 2 0 0 0 0 0 0 0 00 0 2 1 1 0 0 0 2 0 1 0 0 0 2 1 1 0 0 0 0 0 0 00 0 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 1 0 0 0 0 0 00 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 00 0 0 0 0 0 2 2 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 00 0 0 0 0 0 2 2 2 0 0 0 1 0 1 0 0 0 1 1 1 0 0 00 0 0 0 0 0 0 2 2 2 0 0 0 1 0 1 0 0 0 1 1 2 0 00 0 0 0 0 0 0 0 2 1 1 0 0 0 2 0 1 0 0 0 2 1 1 00 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0 1 1 10 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 1 10 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 1 1 2 0 0 0 1 0 2 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 2 1 1 0 0 0 2 0 1 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0
. (4.2)
Here, Sij = 2 if the associated (i, j)th edge is a “long distance” edge. In
this example, we wish to store the ten patterns α1, . . . α10 shown in Figure 4.2 as
memory patterns. In each subfigure, 24 boxes are used to represent each memory
vector, and in each box, 1 and −1 represent white and black respectively. To start
with, only top four memory vectors are considered as associative memory vectors
for storage. Each memory vector corresponds to a 6× 4 array with black and white
boxes represented numerically by −1 and 1 respectively, and is read left to right
from top left corner to produce a single column, stack vector of binary values. The
network parameters, namely the bias vector b and the weight matrix T = T + In,
are obtained by solving the gevp with the bounds in system (3.5) set to L = 1 and
U = 10. For solving (3.5), the “LMI Toolbox” in matlab is used. The next stage
in simulation is to optimize the topology by diluting “long distance” edges marked
by solid lines in Figure 4.1.
In the next section, we dilute the cnn represented by S by implementing
sequential “trimming” of “long distance” links as explained in Section 3.2.1.
39
4.2 Sequential dilution of network connectivity
To reduce the communication cost, connection edges from the given set of
expensive edges marked by solid lines in Figure 4.1 are considered for possible
removal. These edges can be numerically identified by setting a cost threshold ν
(below which a connection is thought to be cheap) and evaluating the residual matrix
R = 12
(sign(S − νS) + S
). Successive deletion of “long distance” edges results in
a different K value. Table 4.1 lists the objective function value resulting from the
removal of different sets of expensive edges. For example, deletion of nine costly
edges in the sequence shown in the table will result in maximum stability parameter
sum of 108.3511 in the example considered.
The performance of proposed design method is quantified in terms of average
recall probability of the neural network, as a function of noise contaminating its
input. We estimate the recall probability for each noise level by generating a set of
50 different 24 × 1 arrays of randomly generated elements within a given interval.
The elements of the noise array are drawn uniformly from the interval [−k, k], and
the level of noise is characterized by the positive constant k (noise factor). We then
contaminate each memory vector with each one of these arrays to produce 50 differ-
ent perturbed versions of each memory vector, at each different noise level quantified
by k. Such a perturbed memory vector is denoted ~α(µ) where µ ranges in 1, . . . , 4
for initial run. Each element of a perturbed input vector ~α(µ), is then saturated
within the [−1, 1] interval, to ensure that the input vector belongs to Hn. For each
memory vector ~α(µ), all 50 perturbed version of it are fed to the network, and the
number of times for which the network converged to α(µ) is recorded. From these
numbers, and after repeating the process for all µ = 1, . . . , 4 the recall probability
of the network for the particular level of noise is calculated.1
1 It should be noticed here that the set of noise added to each initial conditionvector αµ to obtain the corresponding set of perturbed condition vectors thatis then fed to the network, determines the performance of the network. Larger
40
Table 4.1: Cumulative stability parameter K as a result of successive deletion ofhigh cost edges by sequential topology dilution.
No. of deleted edges Node A Node B K
1 8 13 108.32672 10 9 108.27023 9 14 108.27064 8 14 108.26825 9 15 108.28676 7 13 108.26117 3 10 108.27748 3 4 108.26609 7 14 108.267210 10 15 108.351111 4 9 108.260912 9 16 108.257213 8 15 108.251114 16 21 107.830015 22 15 107.270616 21 22 103.768117 15 16 100.7607
Figure 4.3 shows the recall probability of the cnn with the top four memory
patterns of Figure 4.2 stored in the network, as increasingly more neural connections
are severed. There is always a gradual decrease in performance as the amount of
noise injected in the pattern increases. For moderate to severe network dilution, it
is seen that the recall probability of the sparse network remains close to that of the
network with the original topology, and only after 16 out of the 17 high cost links
are deleted we see a noticeable change in the performance.
perturbation results in poorer recall probability.
41
0 5 10 15 20 25 30 350.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
Noise factor value
Avera
ge re
call p
robab
lity
Comparing Recall Probablities of the CNN with Successive Deletion of Edges
Original Network Topology1 edge deleted2 edges deleted3 edges deleted4 edges deleted5 edges deleted6 edges deleted7 edges deleted8 edges deleted9 edges deleted10 edges deleted11 edges deleted12 edges deleted13 edges deleted14 edges deleted15 edges deleted16 edges deletedAll edges deleted
Figure 4.3: Network performance, parameterized by the number of links removed.The horizontal axis marks the level of noise injected to the input ofthe network, and the vertical axis is the average recall probability ofthe network for that particular level of noise.
4.3 Randomized dilution of network connectivity
Our numerical tests suggest that the type of algorithm used for the combina-
torial optimization of network connectivity has little or no effect on the outcome of
the design process. In this section, we compare the original (not optimized) network
model, with the neural network having its topology improved sequentially, and the
one derived by a randomized optimization approach. The comparison is made in
terms of the average recall probability, and results of the comparative study are
shown in Figure 4.4.
It can be seen that the original network topology has roughly same recall
probability graph as the sparser topology derived by the two proposed optimization
approaches. As said before, the sequential dilution algorithm will trim as many
42
Figure 4.4: Network recall probability for a set of four memory vectors. Theoverlapping curves suggest that in terms of performance, the originalnetwork and the ones with diluted connectivity are essentially indis-tinguishable, irrespectively of the method used for optimization.
expensive links while keeping the performance metric above the threshold. For ex-
ample, in the present case the network is trimmed till it reaches the maximum cumu-
lative stability parameter value for the sparer network so obtained (K = 108.3511).
If this threshold is set to a relatively small value, however, then the algorithm will
progressively remove a much longer set of links and there will be a noticeable de-
crease in network performance as shown in Figure 4.3. The objective of optimization
is thus to balance communication cost versus average recall probability.
For our second optimization approach, the specified level α and the confi-
dence parameter δ values are set to 0.01 and 0.005 respectively. The probability
of occurrence of ones in the random matrix “RandMat” of Algorithm 2, p is set
to 0.9 and the simulation is run ξ = 527 number of times. The probable near
maximum of K obtained using this method was 108.9599 with deletion of 5 costly
edges [(14, 9), (10, 9), (13, 7), (15, 9)and(10, 3)]. As shown in the Figure. 4.4, both
43
optimization methods result in similar recall probability.
4.4 More memorized information: less accurate recollection
Intuitively, when more memory patterns are stored in an associative memory,
and given that all memories have to share the same state space, the regions of
attraction of each individual memory vector is reduced. The implication of this fact
is that the associative memory with more stored patterns is less robust to noise (is
less likely to recollect accurately) compared to the one that stores less information.
The numerical tests reported in this section verify this hypothesis, but also show
that the process of dilution of network connectivity has the same overall effect on
the network, irrespectively of how much information is engraved in the memory of
the system.
Figure 4.5: Network’s recall probability for set of 10 memory vectors: Comparisonamong the performance of original network, one optimized using asequential and a random optimization algorithm
Toward this end, we now consider all ten input patterns shown in Figure 4.2.
The objective is to test the ability of the network to store additional patterns with
44
lean interconnection topologies. We compare the recall probability of the original
(not optimized) network, the neural network obtained from the application of the
sequential dilution Algorithm 1, and the one designed using the randomized Algo-
rithm 2. Results indicate that while the recall probability of the networks where ten
memory vectors are stored are clearly lower compared to that of the networks with
only four memory vectors (Figure 4.4), for the same number of stored patterns, the
performance of the neural network is not adversely affected by a moderate dilution
of connectivity. All three cnn topologies, where ten memories are stored, behave
equally well.
4.5 Conclusion and future work
In this chapter, we simulated a cnn design example and implemented the
two proposed topology optimization approaches namely the “sequential dilution
method” and the “ randomized optimization method.” It is found that the original
network topology (unoptimized) has roughly same recall probability graph as the
sparser topology derived by the two proposed optimization approaches and that the
type of algorithm used for the combinatorial optimization of network connectivity
has little or no effect on the outcome of the design process. We also show that the
associative memory with more stored patterns is less robust to noise (is more likely
to fail to recollect accurately) compared to the one that stores less information. It
should be noted that the present design example considered ten 24 × 1 as desired
memory vectors to be stored in the cnn. Efforts were made to simulate large array
of memories. However, due to computational limitations and large simulation time,
this still remains a difficult task and is one of the major challenges.
In Chapter 5, we will see an experimental verification of our proposed design
methods on mobile robots communicating wirelessly. The combinatorial optimiza-
tion method performed almost equally well in practice as in simulations. Also ran-
dom delay has serious impacts on networked communication systems such as the
45
deterioration of the performance, instability, etc. However, for our purpose, we have
considered time delays to be negligible as compared to the time scale of the cnn
dynamics.
46
Chapter 5
REAL TIME HARDWARE IMPLEMENTATION
A majority of this research relies on computer simulations for performance
evaluations of our proposed topology optimization methods. However, evaluating
the effect of real life parameters like understanding the impact of wireless path loss
and interference, are difficult to model in simulation and/or analytical platforms.
In this chapter we apply the ideas developed so far to control network communi-
cation in actual robots to experimentally verify our simulation results. We express
the ideas through a physical realization of the example introduced in Section 4.1.
Although several experimental platforms have been proposed to address these needs
[58, 4, 3], differences in cost, capabilities, and applicability motivate the use of new
experimental platforms to suit targeted experiments. System composition, com-
munication, and control structure all play major roles in accomplishing tasks with
multi-robot systems. We used CoroBot - programmable mobile robot platform from
CoroWare for our experimentations. It is a capable, expandable and affordable mo-
bile robot platform designed to minimize the complexity of robot development. The
low-cost CoRobots enable repeatable experiments without complete reliance on hu-
man subjects for mobility and is low-cost, easily maintainable, extensible, flexible
to support, initially an outreach demonstration, and ultimately an array of research
experiments for validation of our simulation results.
In Section 5.1, we detail the architecture, hardware, communication inter-
face, and software tools used for the CoroBots. This is followed by a demonstration
47
of the application of CoroBots for implementing cnn design example 4.1. Proposed
research experiments that we will be conducting for wireless network experimenta-
tion are explained in Section 5.2. Section 5.3 concludes the chapter by summarizing
the results and discussing the possible future research.
5.1 The experimental testbed
This section describes the physical implementation of a cnn in a spatial
distributed way, on a group of three mobile robots interconnected through wi-fi. A
CoroBot is a four-wheeled robot from CoroWare that has an on board mini-ITX
computer and an optional 4 DOF arm. It is a mobile robot platform that consists of
primary sensor (a laser ranger finder, pan-tilt camera, and/or fixed camera, wheel
encoders and bumper sensors) and the optional 4 degree of freedom arm with a
gripper sensor. It is equipped with a PC class CPU (1.5 GHz Via C-7), expansive
program storage space and CPU capacity to run additional software and can be
purchased running Linux, Windows 7 or in a dual-boot configuration. Its processor
is capable enough to execute learning algorithms, image processing, localization and
mapping tasks. CoroBot comes with 12 digital inputs, 8 analog inputs, 8 digital
outputs, I2C port and extra USB ports which helps in easy mount of additional
hardware components such as GPS, environmental sensors, etc. It can communicate
both wireless(Wi-Fi) and through cable connection.
The front of the CoroBot has the robotic arm. There is also an infrared
Table 5.1: Functioning of power selector switch in CoroBot
Position Mode of CoroBot
Upper Operate off tethered power, enable battery chargingCentermost Off/disable battery chargingLower No charge as run off battery power
48
(a) Front side of CoroBot (b) Right side of CoroBot
(c) Left side of CoroBot (d) Rear side of CoroBot
Figure 5.1: A CoroBot at Cooperative Robots Laboratory at University ofDelaware
range sensor in front that detects the distance from the front of the CoroBot and
the CoroBot Control Panel displays this information. The right side of CoroBot
has a power button near its front and a reset button near its back. The CoroBot’s
left side has a battery charging connector on the left, a tethered power on the right
and there is a three position switch called power selector switch in the center. As
49
explained in Table 5.1, when the power selector switch is in the up position (even
though the CoroBot may be powered off), some power is still available and supports
the robotic arm in place (the arm cannot be manually adjusted). However, with the
switch in the center position and the robot powered off, there is no residual power
available to support the robotic arm, and it can be manually moved to any position
[12]. The rear of the robot has a rear infrared sensor located on the base and the
connection ports are located on the upper deck in between the batteries (Figure 5.1).
The upper deck contains the motherboard, batteries, wireless network adapter and
PCI slot for user-added expansion boards and the lower deck has a servo controller
board [12].
The CoroBot can generally be connected to a wired network as any standard
or desktop computer or it can also be connected to a local wireless network. In our
experiments, each of the three subnetworks described in section 4.1 is realized on
a different CoroBot, and the high cost links between the subnetworks are imple-
mented over wireless channels. As stated in works [51] and Katayama et al. [25],
the traditional local control loop is expected to expand to tomorrows control over
large communication network with variable delay times. Random delay has serious
impacts on networked control systems (NCS) such as the deterioration of the perfor-
mance, instability, etc. However, for our purpose, we have considered time delays to
be negligible as compared to the time scale of the cnn dynamics. The objective is
to experimentally test the hypothesis that a cnn can be implemented in a spatially
distributed way, and that the unavoidable communication delays related to wireless
communication do not necessarily destroy the stability properties of the dynamical
system.
50
5.2 Discretized algorithm implementation
For the purpose of experimentation, we discretize system (2.6) using differ-
ence equations [35] as
xi((k + 1)h) = [xi(kh) + h
n∑j=1
Tijyj(kh) +biai
(eaih − 1)]e−aih, (5.1a)
yi(kh) = sat(xi(kh)), (5.1b)
k = 0, 1..., (5.1c)
i = 1....n (5.1d)
where h is the step duration, k indexes the current time step and i marks
the particular component of the state vector. System (5.1) represents a time-
discretization of the continuous-time dynamics (2.6) under the assumption of a small
time step. Obviously, as h grows, the trajectories of (2.6) and (5.1) diverge, and
eventually the cnn as implemented physically becomes unstable. In the case tested
here, the time step required was sufficiently small for the delays related to commu-
nication be accommodated. If an application imposes larger communication delays
which cannot be accommodated by the length of the discrete-time step, one possible
solution would be to “stretch” the time axis of the discrete-time system, “freezing”
the evolution of the discrete-time system between steps for a period that allows the
dissemination of information between subnetworks. In this way, convergence speed
is traded-off for increased robustness to communication delays, and stability of the
network (in terms of convergence to memory vectors) is not affected. Indeed, it
is verified in our simulations that relatively small values for h result in trajecto-
ries for the state of the cnn which are almost indistinguishable from those of the
continuous-time equations.
Each CoroBot is assigned a fixed IP address on the network router. Here,
each CoroBot on the network acts as a client as well as a server at the same time,
51
Figure 5.2: CoroBot experimental set-up representing the design example
so when it is transmitting information to other CoroBots on the network, it be-
haves as a client, whereas when receiving information (nodes, states, status flags)
from other CoroBots, it behaves as a server that creates a new thread per each in-
coming communication request from other Corobots. The communication channel
is not continuous, it is closed eachtime after some information is transmitted and
re-established eachtime before a new information has to be transmitted. Thus the
network shown in Figure. 4.1 is captured by three CoroBots each representing the
three sub-networks linked by costly edges (as discussed in Section 4.1). The net-
work topology is same as represented by connectivity shown in (4.1) and the cost
of connections are represented by the elements of the weighted adjacency matrix S
(4.2). The objective is to experimentally test the validity of the system represented
by (2.6) using the two proposed optimization methods and ensure that the commu-
nication delays are not significant to destabilize the system. The experimental set
52
Figure 5.3: Error norm:discretized system states (simulation to experimental)
up of three CoroBots is shown in Figure 5.2.
In addition, and for the time step duration that was found sufficient for our
experimental implementation (h = 0.05 sec), the error between simulation of the
discrete-time dynamics (5.1) and experimental results practically coincide, with a
maximum error difference of the order of 10−4 as shown in Figure. 5.3. Here, x-axis
is the time vector and y axis is the error norm.
5.3 Results and discussions
Experimental testing has shown that spatially distributed implementations
of cnn on CoroBots are indeed feasible, and that for some cases, the communica-
tion delays related to the communication between the different components of the
network are not significant enough to affect the performance and stability prop-
erties of the dynamical system. It is shown that the error between simulation of
53
the discrete-time dynamics and experimental results practically coincide, with a
maximum error difference of the order of 10−4. Thus the proposed combinatorial
optimization methods performed almost equally well in practice as in simulations.
54
Chapter 6
CONCLUSION AND FUTURE WORK
6.1 Reflections
In this thesis we formulated a novel combinatorial optimization approach to
design of a sparse cellular neural network. This method is applicable to networks
where links may have varying costs. We found that it is possible to selectively sever
links between neurons of a cellular neural network without significantly affecting
its ability to recall the patterns engraved in its memory. Building up on existing
design tools for cellular neural networks, we optimized the cnn where the objective
function is built using the network’s stability parameters. The type of optimization
algorithm used is immaterial. This type of network optimization yields sparser in-
terconnection topologies, which in turn allows for physical implementations of these
networks which are spatially distributed, and in which the links that connect neu-
rons that reside in different physical locations are as small as possible. Numerical
and experimental testing on CoroBots has shown that such spatially distributed
implementations are indeed feasible, and that for some cases, the communication
delays related to the communication between the different components of the net-
work are not significant enough to affect the performance and stability properties
of the dynamical system.
6.2 Contribution of thesis
In literature, most of the reported approaches on the design of sensor net-
works (without time delays) start with, and build on some given network topology.
55
However in this thesis, we raise the question of how could the design process be
different if the network topology was also part of the design. The objective of this
work is to develop a methodology for the design of cellular neural networks cnns
with interconnection topologies optimized and suitable for spatially distributed im-
plementation. The novelty of the proposed approach lies in the formulation of the
combinatorial optimization problem in a way that trades-off network performance
for communication overhead, and the use of this method for the physical implemen-
tation of associative memories across different interconnected processors.
6.3 Future work
It should be noted that in this thesis , we have considered time delays to
be negligible compared to the time scale of the cnn dynamics. We experimen-
tally show that such spatially distributed implementation of neural networks mobile
sensor networks are indeed feasible, and for some cases, the communication de-
lays related to the communication between different components of the network are
not significant enough to affect the performance and stability properties of the dy-
namical system. However, time delays may become significant depending on the
communication medium (like water). A possible area for future research would be
to analyze global robust stability of uncertain cellular neural networks with discrete
and distributed delays. This might involve the determination of an upper bound
on the permissible time delay and its impact on stability, attractiveness (basins of
attraction) and robustness of the cnn.
56
BIBLIOGRAPHY
[1] S. V. B. Aiyer, M. Niranjan, and F. Fallside. A theoretical investigation intothe performance of the hopfield model. IEEE Transactions on Neural Networks,1:204–215, 1990.
[2] Shun-Ichi Amari. Characteristics of randomly connected threshold-element net-works and network systems. Proceedings of the IEEE, 59(1):35–47, 1971.
[3] M. A. Batalin and G. S. Sukhatme. Coverage, exploration and deploymentby a mobile robot and communication network. Telecommunication Systems,26:181–196, 2004.
[4] P. Blaer and P. K. Allen. Topbot:automated network topology detection witha mobile robot. IEEE International Conference on Robotics and Automation,2:1582–1587, 2003.
[5] S. Boyd, L. Elghaoui, E. Feron, and V. Balakrishnan. Linear Matrix Inequalitiesin Systems and Control Theory, volume 15. Studies in Applied and NumericalMathematics, Society for Industrial and Applied Mathematics, July 1994.
[6] J. Bruck. On the convergence properties of the hopfield model. IEEE Proceed-ings, 78:1579–1585, 1990.
[7] J. Buhmann and K. Schulten. Associative recognition and storage in a modelnetwork of physiological neurons. Biological Cybernetics, 54:319–335, 1986.
[8] J. Cao and L. Wang. Periodic oscillatory solution of bidirectional associativememory networks with delays. Physical Review E (Statistical Physics, Plasmas,Fluids, and Related Interdisciplinary Topics), 61:1825–1828, 2000.
[9] L. O. Chua and L. Yang. Cellular neural networks: theory and applications.IEEE Transactions on Circuits and Systems, 35:1257–1290, 1988.
[10] P. P. Civalleri, M. Gilli, and L. Pandolfi. On stability of cellular neural networkswith delay. IEEE Transactions on Circuits and Systems-I, Fundamental Theoryand Applications, 40:157–165, 1993.
57
[11] A. C. C. Coolen. On the relation between stability parameters and sizes ofdomains of attraction in attractor neural networks. Europhysics Letters, 16:73–78, 1991.
[12] CoroWare. Corobot user’s guide. http://www.robotshop.com/PDF/
corobot-user-guide.pdf, 2007. [Online].
[13] B. M. Forrest. Content-addressability and learning in neural networks. Journalof Physics A: Mathematical and General, 21(1):245–255, 1988.
[14] Y. Fujisaki and Y. Kozawa. Probabilistic robust controller design: probablenear minimax value and randomized algorithms. Proceedings of IEEE Confer-ence on Decision and Control, pages 1938–1943, 2003.
[15] E. Gardner. The space of interactions in neural network models. Journal ofPhysics A: Mathematical and General, 21(1):257–270, 1988.
[16] M. Gilli, M. Biey, and P. Checco. Equilibrium analysis of cellular neural net-works. IEEE Transactions on circuits and Systems-I, Fundamental Theory andApplications, 51:903–912, 2004.
[17] D. Gobovic and M. E. Zaghloul. Analog cellular neural network with applicationto partial differential equations with variable mesh size. IEEE InternationalSymposium on Circuits and Systems, 6:359–362, 1994.
[18] K. Gopalsamy and Xue zhong He. Stability in asymmetric hopfield nets withtransmission delays. Physica D: Nonlinear Phenomena, 76:344–358, 1994.
[19] D. O. Hebb. The Organization of Behavior. Wiley: New York, 1949.
[20] J. J. Hopfield. Neural networks and physical systems with emergent collectivecomputational abilities. Proceedings of National Academy of Sciences, 79:2554–2558, 1982.
[21] J. J. Hopfield. Neurons with graded response have collective computationalproperties like those of two-state neurons. Proceedings of National Academy ofSciences, 81:3088–3092, 1984.
[22] J. J. Hopfield and D. W. Tank. ‘neural’ computation of decisions in optimizationproblems. Biological Cybernetics, 52:141–152, 1985.
[23] J. J. Hopfield and D. W. Tank. Computing with neural circuits: a model.Science, 233:625–633, 1986.
[24] S. Hui and S. H. Zak. Dynamical analysis of the brain-state-in-a-box (bsb)neural models. IEEE Transactions on Neural Networks, 3:86–94, 1992.
58
[25] T. Katayama, T. McKelvey, A. Sano, C. Cassandras, and M. Camp. Trends insystems and signals. Proceedings of 16th IFAC World Congress, 2005.
[26] J. D. Keeler. Basins of attraction of neural network models. AIP ConferenceProceedings 151 on Neural Networks for Computing, pages 259–264, 1986.
[27] Thomas B. Kepler and L.F. Abbott. Domains of attraction in neural networks.Journal de Physique, 49(10):1657–1662, 1988.
[28] S. Kotsiantis and D. Kanellopoulos. Discretization technique: A recent sur-vey. GESTS International Transactions on Computer Science and Engineering,32:47–58, 2006.
[29] A. V. Kozlov and D. Koller. Nonuniform dynamic discretization in hybridnetworks. Thirteenth Annual Conference on Uncertainty in AI (UAI97), pages314–325, 1997.
[30] W. Krauth and M. Mezard. Learning algorithms with optimal stability in neuralnetworks. Journal of Physics A: Mathematical and General, 20(11):L745–L752,1987.
[31] K. E. Kurten. Adaptive architectures for hebbian network models. Journal dePhysique I, 2:615–624, 1992.
[32] T. L. Liao and F. C. Wang. Global stability for cellular neural networks withtime delay. IEEE Transactions on Neural Networks, 11:1481–1484, 2000.
[33] D. Liu and Z. Lu. A new synthesis approach for feedback neural networks basedon the perceptron training algorithm. IEEE Transactions on Neural Networks,8:1468–1482, 1997.
[34] D. Liu and A. N. Michel. Cellular neural networks for associative memories.IEEE Transactions on Circuits and Systems, 40:119–121, 1993.
[35] D. Liu and A. N. Michel. Sparsely interconnected neural networks for associa-tive memories with applications to cellular neural networks. IEEE Transactionson Circuits and Systems-II, Analog and Digital Signal Processing, 41:295–307,1994.
[36] D. Liu and A. N. Michel. Robustness analysis and design of a class of neuralnetworks with sparse interconnecting structure. Neurocomputing, 12:59–76,1996.
[37] A. N. Michel and J. A. Farrell. Associative recall of memory without errors.Physical Review, 35:380–392, 1987.
59
[38] A. N. Michel and J. A. Farrell. Associative memories via artificial neural net-works. IEEE Control Systems Magazine, 1990:6–17, 1988.
[39] A. N. Michel and J. A. Farrell. A synthesis procedure for hopfields continuous-time associative memory. IEEE Transactions on Circuits and Systems, 37:877–884, 1990.
[40] A. N. Michel, J. A. Farrell, and H. F. Sun. Analysis and synthesis techniquesfor hopfield type synchronous discrete time neural networks with applicationto associative memory. IEEE Transactions on Circuits and Systems, 37:1356–1366, 1990.
[41] A. N. Michel and D. L. Gray. Analysis and synthesis of neural networks withlower block triangular interconnecting structure. IEEE Transactions on Cir-cuits and Systems, 37:1267–1283, 1990.
[42] A. N. Michel, J. H. Ji, and W. Porod. Qualitative analysis and synthesis of aclass of neural networks. IEEE Transactions on Circuits and Systems, 8:976–986, 1988.
[43] A. N. Michel, J. H. Ji, and W. Porod. Analysis and synthesis of a class of neuralnetworks: linear systems operating on a closed hypercube. IEEE Transactionson Circuits and Systems, 36:1405–1422, 1989.
[44] A. N. Michel, J. H. Ji, and W. Porod. Analysis and synthesis of a class of neuralnetworks:variable structure systems with infinite gain. IEEE Transactions onCircuits and Systems, 36:713–731, 1989.
[45] A. N. Michel, J. Si, and G. Yen. Analysis and synthesis of a class of discrete-time neural networks described on hypercubes. IEEE Transactions on NeuralNetworks, 2:32–46, 1991.
[46] V. M. Mladenov, D. M. W. Leenaerts, and F. H. Uhlmann. Estimation ofthe basin of attractions in cnns. IEEE Transactions on Circuits and Systems,45:571–574, 1998.
[47] J. Park and Y. Park. An optimization approach to design of cellular neuralnetworks. International Journal of Systems Science, 31:1585–1591, 2000.
[48] J. H. Park. Global exponential stability of cellular neural networks with varyingdelays. Applied Mathematics and Computation, 183:1214–1219, 2006.
[49] J. H. Park. A new stability analysis of a class of cellular neural networks.Applied Mathematics and Computation, 181:200–205, 2006.
60
[50] R. Perfetti. A neural network to design neural networks. IEEE Transactionson Circuits and Systems, 38:1099–1103, 1991.
[51] M. Pohjola. PID Controller Design in Networked Control Systems. 2006.
[52] T. Roska and L. O. Chua. Cellular neural networks with nonlinear and delay-type template elements. 1990 IEEE International Workshop on Cellular NeuralNetworks and their Applications, CNNA− 90, pages 12–25, 1990.
[53] M. E. Savran and O. Morgul. On the design of dynamic associative neuralmemories. IEEE International Joint Conference on Neural Networks, 2:1166–1171, 1991.
[54] A. Slavova. Dynamic properties of cellular neural networks. Journal of AppliedMathematics and Stochastic Analysis, 6:107–116, 1993.
[55] D. W. Tank and J. J. Hopfield. Simple ‘neuron’ optimization networks: Ana/d converter, signal decision circuit, and a linear programming circuit. IEEETransactions on Circuits and Systems, 33:533–541, 1986.
[56] P. van den Driessche and Xingfu Zou. Global attractivity in delayed hopfieldneural network models. SIAM Journal on Applied Mathematics, 58:1878–1890,1998.
[57] M. Vidyasagar. Randomized algorithms for robust controller synthesis usingstatistical learning theory. Automatica, 37:1515–1528, 2001.
[58] B. White, J. Lepreau, and S. Guruprasad. Lowering the barrier to wirelessand mobile experimentation. Computer Communication Review (ACM SIG-COMM), 33:47–52, 2003.
[59] H. Ye, A. N. Michel, and K. Wang. Global stability and local stability ofhopfield neural networks with delays. Physical Review E (Statistical Physics,Plasmas, Fluids, and Related Interdisciplinary Topics), 50:4206–4213, 1994.
[60] G. Yen and A. N. Michel. A learning and forgetting algorithm in associativememories: results involving pseudo-inverses. IEEE Transactions on Circuitsand Systems, 38:1193–1205, 1991.
[61] G. Yen and A. N. Michel. A learning and forgetting algorithm in associativememories: the eigenstructure method. IEEE Transactions on Circuits andSystems- II, 39:212–225, 1992.
61
APPENDIX
CNN STABILITY CRITERIA
Theorem Let α ∈ Bn be a memory vector of (2.6), and let k ≥ 1 be an integer, if
T = T − I and b satisfy
αi( n∑j=1
Tijαj + bi)> 2(k − 1)max1≤j≤m|Tij|, i = 1 . . . , n (6.1)
then any binary vector α∗ ∈ Bn such that 1 ≤ H(α∗, α) ≤ khas following properties
1. α∗ is not a memory vector of (2.6)
2. if x(0) = α∗ and α∗i 6= αi, then xi(t) converges to αi.
Proof
Let α∗ ∈ Bn be any binary vector satisfying 1 ≤ H(α∗, α) ≤ k and let α∗ 6= α.
Without loss of generality, we assume that α = 1 and α∗ = −1. Then, δ , α∗ − α
satisfies
∣∣ n∑j=1
Tijδj∣∣ =
∣∣Ti1δ1 + . . .+ 0× δi + . . . Tinδn∣∣ ≤ 2(k − 1)max1≤j≤n|Tij| (6.2)
Now∑n
j=1 Tijα∗j + bi =
∑nj=1 Tijαj + bi +
∑nj=1 Tijδj
≥∑n
j=1 Tijαj + bi − 2(k − 1)max1≤j≤n|Tij| > 0
This implies that
62
α∗i(∑n
j=1 Tijα∗j + bi
)= α∗i
(α∗i +
∑nj=1 Tijα
∗j + bi
)= 1−
∑nj=1 Tijα
∗j + bi < 1
α∗ cannot be a memory vector of (2.6).
Now let system 2.6 start from x(0) = α∗, then ith component satisfies
˙xi(0) = −xi(0) +∑n
j=1 Tijsat (xj(0)) + bi = −xi(0) +(sat(xi(0)) +∑n
j=1 Tijsat (xj(0)) + bi)
= α∗i + (−α∗i +∑n
j=1 Tijα∗j ) + bi =
∑nj=1 Tijα
∗j + bi
Above must be greater than 0, and hence xi(t) evolves towards αi = 1 from −1.
Other case can be shown similarly.
63