mathematics of neural networks - home - springer978-1-4615-6099-9/1.pdf · vi mathematics of neural...
TRANSCRIPT
MATHEMATICS OF NEURAL
NETWORKS
Models, Algorithms and Applications
OPERATIONS RESEARCH/COMPUTER SCIENCE INTERFACES
SERIES
Ramesh Sharda, Series Editor ConocolDuPont Chair of Management of Technology
Oklahoma State University Stillwater, Oklahoma U.S.A.
Other published titles in the series:
Brown, Donald/Scherer, William T. University of Virginia Intelligent Scheduling Systems
Nash, Stephen G'/Sofer, Ariela George Mason University The Impact of Emerging Technologies on Computer Science and Operations Research
Barth, Peter Max-Planck-Institut fur Infonnatik, Gennany
Logic-Based 0-/ Constraint Programming
Jones, Christopher V. University of Washington Visualization and Optimization
Barr, Richard S./ Helgason, Richard V./ Kennington, Jeffery L. Southern Methodist University Interfaces in Computer Science and Operations Research: Advances in Metaheuristics, Optimization, and Stochastic Modeling Technologies
MATHEMATICS OF NEU RAL N ETWORKS
Models, Aigorithms and Applications
EDITED SY
Stephen W ELLACOTT University of Srighton
United Kingdom
• John C MASON
University of Huddersfield
United Kingdom
• lain J ANDERS ON
University of Huddersfield
United Kingdom
SPRINGER SCIENCE+BUSINESS MEDIA, LLC
Library of Congress Cataloging-in-Publication Data
A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN 978-1-4613-7794-8 ISBN 978-1-4615-6099-9 (eBook) DOI 10.1007/978-1-4615-6099-9
© Springer Science+Business Media New York 1997 Originally published by Kluwer Academic Publishers 1997 Softcover reprint of the hardcover 1 st edition 1997
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC
Printed on acid-free paper.
CONTENTS
PREFACE XXI
Part I INVITED PAPERS 1 1 N-TUPLE NEURAL NETWORKS
N. M. Allinson and A. R. Kolcz 3 2 INFORMATION GEOMETRY OF NEURAL
NETWORKS -AN OVERVIEW-Shun-ichi Amari 15
3 Q-LEARNING: A TUTORIAL AND EXTENSIONS George Cybenko, Robert Gray and Katsuhiro Moizumi 24
4 ARE THERE UNIVERSAL PRINCIPLES OF BRAIN COMPUTATION? Stephen Grossberg 34
5 ON-LINE TRAINING OF MEMORY-DRIVEN ATTRACTOR NETWORKS Morris W. Hirsch 41
6 MATHEMATICAL PROBLEMS ARISING FROM CONSTRUCTING AN ARTIFICIAL BRAIN J. G. Taylor 47
Part II SUBMITTED PAPERS 59 7 THE SUCCESSFUL USE OF PROBABILITY
DATA IN CONNECTIONIST MODELS J. R. Alexander Jr. and J. P. Coughlin 61
8 WEIGHTED MIXTURE OF MODELS FOR ON-LINE LEARNING P. Edgar An 67
9 LOCAL MODIFICATIONS TO RADIAL BASIS NETWORKS I. J. Anderson 73
v
VI MATHEMATICS OF NEURAL NETWORKS
10 A STATISTICAL ANALYSIS OF THE MODIFIED NLMS RULES E.D. Aved'yan, M. Brown and C.J. Harris 78
11 FINITE SIZE EFFECTS IN ON-LINE LEARNING OF MULTI-LAYER NEURAL NETWORKS. David Barber, Peter Sollich and David Saad 84
12 CONSTANT FAN-IN DIGITAL NEURAL NETWORKS ARE VLSI-OPTIMAL V. Beiu 89
13 THE APPLICATION OF BINARY ENCODED 2ND DIFFERENTIAL SPECTROMETRY IN PREPROCESSING OF UV -VIS ABSORPTION SPECTRAL DATA N Benjathapanun, W J 0 Boyle and K T V Grattan 95
14 A NON-EQUIDISTANT ELASTIC NET ALGORITHM Jan van den Berg and Jock H. Geselschap 101
15 UNIMODAL LOADING PROBLEMS Monica Bianchini, Stefano Fanelli, Marco Gori and Marco Protasi 107
16 ON THE USE OF SIMPLE CLASSIFIERS FOR THE INITIALISATION OF ONE-HIDDEN-LAYER NEURAL NETS Jan C. Bioch, Robert Carsouw and Rob Potharst 113
17 MODELLING CONDITIONAL PROBABILITY DISTRIBUTIONS FOR PERIODIC VARIABLES Christopher M Bishop and Ian T Nabney 118
18 INTEGRO-DIFFERENTIAL EQUATIONS IN COMPARTMENTAL MODEL NEURODYNAMICS Paul C. Bressloff 123
19 NONLINEAR MODELS FOR NEURAL NETWORKS Susan Brittain and Linda M. Haines 129
20 A NEURAL NETWORK FOR THE TRAVELLING SALESMAN PROBLEM WITH A WELL BEHAVED ENERGY FUNCTION Marco Budinich and Barbara Rosario 134
Contents Vll
21 SEMIPARAMETRIC ARTIFICIAL NEURAL NETWORKS Enrico Capobianco 140
22 AN EVENT-SPACE FEEDFORWARD NETWORK USING MAXIMUM ENTROPY PARTITIONING WITH APPLICATION TO LOW LEVEL SPEECH DATA D.K. Y. Chiu, D. Bockus and J. Bradford 146
23 APPROXIMATING THE BAYESIAN DECISION BOUNDARY FOR CHANNEL EQUALISATION USING SUBSET RADIAL BASIS FUNCTION NETWORK E.S. Chng, B. Mulgrew, S. Chen and G. Gibson 151
24 APPLICATIONS OF GRAPH THEORY TO THE DESIGN OF NEURAL NETWORKS FOR AUTOMATED FINGERPRINT IDENTIFICATION Carol G. Crawford 156
25 ZERO DYNAMICS AND RELATIVE DEGREE OF DYNAMIC RECURRENT NEURAL NETWORKS A. Delgado, C. Kambhampati and K. Warwick 161
26 IRREGULAR SAMPLING APPROACH TO NEUROCONTROL: THE BAND-AND SPACE-LIMITED FUNCTIONS QUESTIONS Andrzej Dzielinski and Rafal Zbikowski 166
27 UNSUPERVISED LEARNING OF TEMPORAL CONSTANCIES BY PYRAMIDAL-TYPE NEURONS. Michael Eisele 171
28 NUMERICAL ASPECTS OF MACHINE LEARNING IN ARTIFICIAL NEURAL NETWORKS S. W. Ellacott and A. Easdown 176
29 LEARNING ALGORITHMS FOR RAM-BASED NEURAL NETWORKS Alistair Ferguson, Laurence C Dixon and Hamid Bolouri 181
30 ANALYSIS OF CORRELATION MATRIX MEMORY AND PARTIAL MATCH-IMPLICATIONS FOR COGNITIVE
Vlll MATHEMATICS OF NEURAL NETWORKS
PSYCHOLOGY Richard Filer and James Austin 186
31 REGULARIZATION AND REALIZABILITY IN RADIAL BASIS FUNCTION NETWORKS Jason A.S. Freeman and David Saad 192
32 A UNIVERSAL APPROXIMATOR NETWORK FOR LEARNING CONDITIONAL PROBABILITY DENSITIES D. Husmeier, D. Allen and J. G. Taylor 198
33 CONVERGENCE OF A CLASS OF NEURAL NETWORKS Mark P. Joy 204
34 APPLICATIONS OF THE COMPARTMENTAL MODEL NEURON TO TIME SERIES ANALYSIS S. K asderidis and J. G. Taylor 209
35 INFORMATION THEORETIC NEURAL NETWORKS FOR CONTEXTUALLY GUIDED UNSUPERVISED LEARNING Jim Kay 215
36 CONVERGENCE IN NOISY TRAINING Petri K oistinen 220
37 NON-LINEAR LEARNING DYNAMICS WITH A DIFFUSING MESSENGER Barl Krekelberg and John G. Taylor 225
38 A VARIATIONAL APPROACH TO ASSOCIATIVE MEMORY Abderrahim Labbi 230
39 TRANSFORMATION OF NONLINEAR PROGRAMMING PROBLEMS INTO SEPARABLE ONES USING MULTILAYER NEURAL NETWORKS Bao-Liang Lu and Koji Ito 235
40 A THEORY OF SELF-ORGANISING NEURAL NETWORKS S P Luttrell 240
41 NEURAL NETWORK SUPERVISED TRAINING BASED ON A DIMENSION REDUCING METHOD G.D. Magoulas, M.N. Vrahatis, T.N. Grapsa and G.S. Androulakis 245
Contents IX
42 A TRAINING METHOD FOR DISCRETE MULTILAYER NEURAL NETWORKS G.D. Magoulas, M.N. Vrahatis, T.N. Grapsa and G.S. Androulakis 250
43 LOCAL MINIMAL REALISATIONS OF TRAINED HOPFIELD NETWORKS S. Manchanda and G.G.R. Green 255
44 DATA DEPENDENT HYPERPARAMETER ASSIGNMENT Glenn Marion and David Saad 259
45 TRAINING RADIAL BASIS FUNCTION NETWORKS BY USING SEPARABLE AND ORTHOGONALIZED GAUSSIANS J. C. Mason, 1. J. Anderson, G. Rodriguez and S. Seatzu 265
46 ERROR BOUNDS FOR DENSITY ESTIMATION BY MIXTURES Ronny Meir and Assaf J. Zeevi 270
47 ON SMOOTH ACTIVATION FUNCTIONS H. N. Mhaskar 275
48 GENERALISATION AND REGULARISATION BY GAUSSIAN FILTER CONVOLUTION OF RADIAL BASIS FUNCTION NETWORKS Christophe Molina and Mahesan Niranjan 280
49 DYNAMICAL SYSTEM PREDICTION: A LIE ALGEBRAIC APPROACH FOR A NOVEL NEURAL ARCHITECTURE Yves Moreau and Joos Vandewalle 285
50 STOCHASTIC NEURODYNAMICS AND THE SYSTEM SIZE EXPANSION Toru Ohira and Jack D. Cowan 290
51 AN UPPER BOUND ON THE BAYESIAN ERROR BARS FOR GENERALIZED LINEAR REGRESSION Cazhaow S. Qazaz, Christopher K. I. Williams and Christopher M. Bishop 295
52 CAPACITY BOUNDS FOR STRUCTURED NEURAL NETWORK ARCHITECTURES Peter Rieper, Sabine Kroner and Reinhard Moratz 300
53 ON-LINE LEARNING IN MULTILAYER NEURAL NETWORKS David Saad and Sara A. Solla 306
x MATHEMATICS OF NEURAL NETWORKS
54 SPONTANEOUS DYNAMICS AND ASSOCIATIVE LEARNING IN AN ASSYMETRIC RECURRENT RANDOM NEURAL NETWORK M. Samuelides, B. Doyon, B. Cessac and M. Quoy 312
55 A STATISTICAL MECHANICS ANALYSIS OF GENETIC ALGORITHMS FOR SEARCH AND LEARNING Jonathan L. Shapiro, Adam Prugel-Bennett and Magnus Rattray 318
56 VOLUMES OF ATTRACTION BASINS IN RANDOMLY CONNECTED BOOLEAN NETWORKS Sergey A. Shumsky 323
57 EVIDENTIAL REJECTION STRATEGY FOR NEURAL NETWORK CLASSIFIERS A. Shustorovich 328
58 DYNAMICS APPROXIMATION AND CHANGE POINT RETRIEVAL FROM A NEURAL NETWORK MODEL J. Smid and P. Vol! 333
59 QUERY LEARNING FOR MAXIMUM INFORMATION GAIN IN A MULTI-LAYER NEURAL NETWORK Peter Sollich 339
60 SHIFT, ROTATION AND SCALE INVARIANT SIGNATURES FOR TWO-DIMENSIONAL CONTOURS, IN A NEURAL NETWORK ARCHITECTURE David McG. Squire and Terry M. Caelli 344
61 FUNCTION APPROXIMATION BY THREE-LAYER ARTIFICIAL NEURAL NETWORKS Shin Suzuki 349
62 NEURAL NETWORK VERSUS STATISTICAL CLUSTERING TECHNIQUES: A PILOT STUDY IN A PHONEME RECOGNITION TASK G. Tambouratzis, T. Tambouratzis and D. Tambouratzis 355
63 MULTISPECTRAL IMAGE ANALYSIS USING PULSED COUPLED NEURAL NETWORKS G L Tarr, X Clastres, L Freyss, M Samuelides, C Dehainaut and W Burckel 361
Contents Xl
64 REASONING NEURAL NETWORKS Rua-Huan R. Tsaih 366
65 CAPACITY OF THE UPSTART ALGORITHM Ansgar H. L. West and David Saad 372
66 REGRESSION WITH GAUSSIAN PROCESSES Christopher K. 1. Williams 378
67 STOCHASTIC FORWARD-PERTURBATION, ERROR SURFACE AND PROGRESSIVE LEARNING IN NEURAL NETWORKS Li-Qun Xu 383
68 DYNAMICAL STABILITY OF A HIGH-DIMENSIONAL SELF -ORGANIZING MAP Howard Hua Yang 389
69 MEASUREMENTS OF GENERALISATION BASED ON INFORMATION GEOMETRY Huaiyu Zhu and Richard Rohwer 394
70 TOWARDS AN ALGEBRAIC THEORY OF NEURAL NETWORKS: SEQUENTIAL COMPOSITION R. Zimmer 399
J. R. Alexander Jr. Towson State University, Towson, MD 21032, USA. Email: e7 [email protected]
D. Allen Centre for Neural Networks, Department of Mathematics, King's College London, London WC2R 1LS UK.
N. M. Allinson Department of Electrical Engineering and Electronics, UMIST, Manchester M60 1QD, UK. Email: [email protected]
Shun-ichi Amari University of Tokyo, Tokyo, Japan, RIKEN Frontier Research Program, Wako-City, Saitama, Japan. Email: [email protected]
P. Edgar An Department of Ocean Engineering, Florida Atlantic University, Boca Raton, FL 33431, USA. Email: [email protected]
I. J. Anderson School of Computing and Mathematics, University of Huddersfield, Queensgate, Huddersfield HD1 3DH, UK. Email: [email protected]
G.S. Androulakis Department of Mathematics, University of Patras, GR-261.10 Patras, Greece. Email: [email protected]
CONTRIBUTORS
James Austin Advanced Computer Architecture Group, Department of Computer Science, University of York, Heslington, Y01 5DD, UK.
E.D. Aved'yan Russian Academy of Sciences, Institute of Control Sciences, 65 Profsoyuznaya str., Moscow 117342, Russia. Email: [email protected]
David Barber Dept. Comp, Sci. and Appl. Maths., Aston University, Birmingham B4 7ET, UK. Web: http://www.ncrg.aston.ac.uk/
V. Beiu Los Alamos National Laboratory, Division NIS-I, Los Alamos, New Mexico 87545, USA. Email: [email protected]
Jan van den Berg Department of Computer Science, Erasmus University Rotterdam, The Netherlands. Email: [email protected]
N Benjathapanun Department of Electrical, Electronic and Information Engineering City University, Northampton Square, London EC1V OHB, UK.
Monica Bianchini Dipartimento di Sistemi e Informatica, Universita degli Studi di Firenze, Via di Santa Marta, 3, 50139 Firenze, Italy. Email: [email protected]
Contributors
Jan C. Bioch Department of Computer Science, Erasmus University Rotterdam, The Netherlands. Email: [email protected]
Christopher M. Bishop Neural Computing Research Group, Dept. of Computer Science and Applied Mathematics, Aston University, Birmingham B4 7ET, UK. Email: [email protected]. uk
D. Bockus Dept. Computing and Information Science, University of Guelph, Ontario NIG 2Wl, Canada.
Hamid Bolouri Engineering Research and Development Centre, University of Hertfordshire, College Lane, Hatfield, Herts, ALI0 9AB, UK.
W J 0 Boyle Department of Electrical, Electronic and Information Engineering City University, Northampton Square, London ECIV OHB, UK.
J. Bradford Brock University, St. Catharines, Ontario, Canada.
Paul C. Bressloft' Department of Mathematical Sciences, Loughborough University, Leics. LEU 3TU, UK. Email: [email protected]
Susan Brittain University of Natal, Pietermaritzburg, South Africa.
M. Brown ISIS research group, Dept of Electronics and Computer Science, Southampton University, Highfield, Southampton, S017 IBJ, UK. Email: [email protected]
Marco Budinich Dipartimento di Fisica & INFN, Via Valerio 2, 34127, Trieste, Italy. Email: [email protected]
William Burckel Phillips Laboratory, Kirtland Air Force Base Albuquerque, New Mexico, USA.
Terry M. Caelli School of Computing, Curtin University of Technology, Perth, Western Australia. Email: [email protected]
Enrico Capobianco Stanford University, Stanford, CA 94305, USA. Email: [email protected]
Robert Carsouw Department of Computer Science, Erasmus University Rotterdam, The Netherlands.
B. Cessac University of New-Mexico, Department of Electrical and Computer Engineering, Albuquerque, NM 87131, USA.
S. Chen Dept. of E.E.E., The University of Portsmouth, UK.
Xavier Clast res Centre d'Etudes et de Recherches de Toulouse, 2 avenue Edouard Belin, BP 4025, 31055 Toulouse Cedex, France.
D.K.Y. Chiu
Xlll
Dept. Computing and Information Science, University of Guelph, Ontario NIG 2Wl, Canada. Email: [email protected]
XIV
E.S Chng National University of Singapore, Institute of System Science, Heng Mui Keng Terrace, Kent Ridge, 119597 Singapore. Email: [email protected].
J. P. Coughlin Towson State University, Towson, MD 21032, USA.
Jack D. Cowan Departments of Mathematics and Neurology, The University of Chicago, Chicago, IL 60637, USA. Email: [email protected]
Carol G. Crawford U.S. Naval Academy, Department of Mathematics, Annapolis MD 21402, USA. Email: [email protected]
George Cybenko Thayer School of Engineering, 8000 Cummings Hall, Dartmouth College, Hanover, NH 03755 USA. Email: [email protected]
Laurence C Dixon Engineering Research and Development Centre, University of Hertfordshire, College Lane, Hatfield, Herts, AL10 9AB, UK.
Christopher Dehainaut Phillips Laboratory, Kirtland Air Force Base Albuquerque, New Mexico, USA.
A. Delgado National University of Columbia, Elec. Eng. Dept., AA No. 25268, Bogota, D C, Columbia SA. Email: [email protected]
B. Doyon Unite INSERM 230, Service de Neurologie, CHU Purpan, 31059 Toulouse Cedex, France.
CONTRIBUTORS
Andrzej Dzielinski Department of Mechanical Engineering, James Watt Building, University of Glasgow, Glasgow G12 8QQ, Scotland, UK. Email: [email protected]
A. Easdown School of Computing and Mathematical Sciences, University of Brighton, Brighton BN1 4GJ, UK. Email: [email protected]
Michael Eisele The Salk Institute, Computational Neurobiology Laboratory, PO Box 85800, San Diego, CA 92186 - 5800, USA. Email: [email protected]
S.W. Ellacott School of Computing and Mathematical Sciences, University of Brighton, Brighton BN1 4GJ, UK. Email: [email protected]
Stefano Fanelli Dipartimento di Matematica, Universita. di Roma, "Tor Vergata" Via della Ricerca Scientifica, 00133 Roma, Italy. Email: [email protected]
Alistair Ferguson Engineering Research and Development Centre, University of Hertfordshire, College Lane, Hatfield, Herts, AL10 9AB, UK. Email: [email protected]
Richard Filer Advanced Computer Architecture Group, Department of Computer Science, University of York, Heslington, YOl 5DD, UK. Email: [email protected]. uk
Contributors
Jason A.S. Freeman Centre for Neural Systems, University of Edinburgh, Edinburgh EH8 9LW, UK. Email: [email protected]
Laurent Freyss Centre d'Etudes et de Recherches de Toulouse, 2 avenue Edouard Belin,BP 4025, 31055 Toulouse Cedex, France.
Jock H. Geselschap Department of Computer Science, Erasmus University Rotterdam, The Netherlands.
G. Gibson Biomathematics and Statistics, The University of Edinburgh, UK.
Marco Gori Dipartimento di Sistemi e Informatica, U niversita. degli Studi di Firenze, Via di Santa Marta, 3, 50139 Firenze, Italy. Email: [email protected]
T.N. Grapsa Department of Mathematics, University of Patras, GR-261.10 Patras, Greece. Email:[email protected]
K T V Grattan Department of Electrical, Electronic and Information Engineering City University, Northampton Square, London EC1V OHB, UK.
Robert Gray Thayer School of Engineering, 8000 Cummings Hall, Dartmouth College, Hanover, NH 03755 USA.
G.G.R. Green Dept. of Physiological Sciences, University of Newcastle upon Tyne, Newcastle upon Tyne, NE1 7RU, UK. Email: [email protected]
xv
Stephen Grossberg Boston University, Department of Cognitive and Neural Systems, and Center for Adaptive Systems, 677 Beacon Street, Boston, Massachusetts 02215 USA
Linda M. Haines University of Natal, Pietermaritzburg, South Africa. Email: [email protected]
C.J. Harris ISIS research group, Dept of Electronics and Computer Science, Southampton University, Highfield, Southampton, S017 lBJ, UK.
Morris W. Hirsch Department of Mathematics, University of California at Berkeley, CA 94720-3840 USA. Email: [email protected]
D. Husmeier Centre for Neural Networks, Department of Mathematics, King's College London, London WC2R 1LS, UK. Email: [email protected]
Koji Ito Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology, 4259, Nagatsuda, Midori-ku, Yokohama 226, Japan. Email: [email protected]
Mark P. Joy Centre for Research in Information Engineering, School of Electrical, Electronic and Information Enineering, South Bank University, 103, Borough Rd., London SE1 OAA, UK. Email: [email protected]
C. Kambhampati Cybernetics Department, University of Reading, Whiteknights, Reading RG6 6AY, UK.
XVI
S. Kasderidis Department of Mathematics, King's College London, London WC2R 1LS, UK.
Jim Kay Dept. of Statistics, University of Glasgow, Mathematics Building, Glasgow G12 8QW, UK. Email: [email protected]
Petri Koistinen Rolf Nevanlinna Institute, P.O. Box 4, FIN-00014 University of Helsinki, Finland. Email: [email protected]
A. R. Kolcz Department of Electrical Engineering and Electronics, UMIST, Manchester M60 1QD, UK.
Bart Krekelberg Centre for Neural Networks, King's College London London WC2R 2LS, UK Email: [email protected]
Sabine Kroener Technische Informatik I, TU Hamburg-Harburg, Harburger Schl06str. 20, D-21071 Hamburg, Germany. Email: [email protected]
Abderrahim Labbi Dept. of Computer Science, University of Geneva, 24 Rue General Dufour, 1121 Geneva 4, Switzerland. Email: [email protected]
Bao-Liang Lu The Institute of Physical and Chemical Research (RIKEN), 3-8-31 Rokuban,Atsuta-ku, Nagoya 456, Japan. Email: [email protected]
CONTRIBUTORS
S P Luttrell Defence Research Agency, Malvern, Worcs, WR14 3PS, UK. Email: [email protected]
G.D. Magoulas Department of Electrical and Computer Engineering, University of Patras, GR-261.10, Patras, Greece. Email: [email protected]
S. Manchanda Dept. of Chemical and Process Engineering, University of Newcastle upon Tyne, Newcastle upon Tyne, NE1 7RU, UK. Email: [email protected]
Glenn Marion Department of Statistics and Modelling Science, Livingstone Tower, 26 Richmond Street, Glasgow G11XH, UK. Email: [email protected]
J. C. Mason School of Computing and Mathematics, University of Huddersfield, Queensgate, Huddersfield HD1 3DH, UK. Email: [email protected]
Ronny Meir Electrical Engineering Department, Technion, Haifa 32000, Israel. Email: [email protected]
H. N. Mhaskar Department of Mathematics, California State University, Los Angeles, California 90032, USA. Email: [email protected]
Katsuhiro Moizumi Thayer School of Engineering, 8000 Cummings Hall, Dartmouth College, Hanover, NH 03755 USA.
Contributors
Christophe Molina Cambridge University Engineering Department, Trumpington Street, Cambridge CB2 1PZ, UK. Email: [email protected]
Reinhard Moratz AG Angewandte Informatik, Universitat Bielefeld, Postfach 100131, D-33501 Bielefeld, Germany.
Yves Moreau Katholieke Universiteit Leuven, Dept. of Electrical Engineering, ESAT-SISTA Kardinaal Mercierlaan 94, B-3001 Leuven (Heverlee), Belgium. Email: [email protected]
B. Mulgrew Dept. of Electrical Eng., The University of Edinburgh, UK.
Ian T Nabney Neural Computing Research Group, Aston University, Birmingham, B4 7ET, UK. Email: [email protected]
Mahesan Niranjan Cambridge University Engineering Department, Trumpington Street, Cambridge CB2 1PZ, UK. Email: [email protected]
Toru Ohira Sony Computer Science Laboratory 3-14-13 Higashi-gotanda, Shinagawa, Tokyo 141, Japan. Email: [email protected]
Rob Potharst Department of Computer Science, Erasmus University Rotterdam, The Netherlands. Email: [email protected]
Marco Protasi Dipartimento di Matematica, Universita. di Roma, "Tor Vergata" Via della Ricerca Scientifica, 00133 Roma, Italy. Email: [email protected]
Adam Pruegel-Bennett NORDITA Blegdamsvej 17, DK-2100 Copenhagen 0, Denmark.
Cazhaow S. Qazaz Neural Computing Research Group, Dept. of Computer Science and Applied Mathematics,
XVlI
Aston University, Birmingham B4 7ET, UK.
M. Quoy Universitiit Bielefeld, BiBos, Postfach 100131, 33501 Bielefeld, Germany.
Magnus Rattray Department of Computer Science, University of Manchester, Manchester, M13 9PL, UK.
Peter Rieper FB Mathematik, Universitaet Hamburg, Bundesstr. 55, D-20146 Hamburg, Germany.
G. Rodriguez Department of Mathematics, University of Cagliari, viale Merello 92,09123 Cagliari, Italy.
Richard Rohwer Prediction Co., 320 Aztec Street, SuiteB, Santa Fe, NM87501, USA.
Barbara Rosario Dipartimento di Fisica & INFN, Via Valerio 2, 34127, Trieste, Italy. Email: [email protected]
XVlll
David Saad Dept. of Computer Science and Applied Mathematics, University of Aston, Birmingham B4 7ET, UK. Email: [email protected]
M. Samuelides Centre d'Etudes et de Recherches de Toulouse, 2 avenue Edouard Belin, BP 4025, 31055 Toulouse Cedex, France. Email: [email protected].
S. Seatzu Department of Mathematics, University of Cagliari, viale Merello 92, 09123 Cagliari, Italy.
Jonathan L. Shapiro Department of Computer Science, University of Manchester, Manchester, M13 9PL, UK.
Sergey A. Shumsky P.N . Lebedev Physics Institute, Leninsky pr.53, Moscow, Russia. Email: [email protected]
A. Shustorovieh Business Imaging Systems, Eastman Kodak Company, Rochester, USA. Email: [email protected]
J. Smid Department of Mathematics, Morgan State University, Baltimore, MD 21239 USA. Email: [email protected]
Sara A. Solla CONNECT, The Niels Bohr Institute, Blegdamsvej 17,Copenhagen 2100, Denmark.
Peter Sollieh Department of Physics, University of Edinburgh, Edinburgh EH9 3JZ, UK. Email: [email protected]
David MeG. Squire School of Computing,
CONTRIBUTORS
Curtin University of Technology, Perth, Western Australia. Email: [email protected]
Shin Suzuki Information Science Research Laboratory, NTT Basic Research Laboratories, 3-1 Morinosato-Wakamiya, Atsugi-Shi, Kanagawa Pref., 243-01 Japan. Email: [email protected]
G. Tambouratzis Dept. of Mathematics, Agricultural Univ.of Athens, Iera Odos 75, Athens 118 55, Greece. Email: [email protected]
D. Tambouratzis Dept. of Mathematics, Agricultural Univ.of Athens, Iera Odos 75, Athens 118 55, Greece.
T. Tambouratzis Institute of Nuclear Technology - Radiation Protection, NRCPS "Demokritos", Aghia Paraskevi 153 10, Greece.
Gregory L. Tarr Phillips Laboratory, Kirtland Air Force Base Albuquerque, New Mexico, USA. Email: [email protected]
J. G. Taylor Department of Mathematics and Centre for Neural Networks, King's College, London WC2R ILS, UK.
Rua-Huan R. Tsaih Department of Management Information Systems, National Chengchi University, Taipei, Taiwan, ROC. Email: [email protected]
Contributors
Joos Vandewalle Katholieke U niversiteit Leu yen, Dept. of Electrical Engineering, ESAT-SISTA Kardinaal Mercierlaan 94, B-3001 Leuven (Heverlee), Belgium.
P. Volf UTIA The Czech Academy of Sciences, Prague 8, CZ 180 00 Czech Republic. Email: [email protected]
M.N. Vrahatis Department of Mathematics, University of Patras, GR-261.10 Patras, Greece. Email: [email protected]
K. Warwick Cybernetics Department, University of Reading, Whiteknights, Reading RG6 6A Y, UK. Email: [email protected]
Ansgar H. L. West Neural Computing Research Group, Aston University, Aston Triangle, Birmingham B4 7ET, UK. Email: [email protected]
Christopher K. I. Williams Neural Computing Research Group, Department of Computer Science and Applied Mathematics, Aston University, Birmingham B4 7ET, UK. Email: [email protected]
Li-Qun Xu Intelligent Systems Research Group, BT Research Laboratories Martlesham Heath, Ipswich, IP5 7RE, UK. Email: [email protected]
Howard Hua Yang Lab for Information Representation, FRP, RIKEN, 2-1 Hirosawa, Wako-Shi, Saitama 351-01, Japan. Email: [email protected]
XIX
Rafal Zbikowski Department of Mechanical Engineering, James Watt Building, University of Glasgow, Glasgow G12 8QQ, Scotland, UK.
Assaf J. Zeevi Electrical Engineering Department, Technion, Haifa 32000, Israel.
Huaiyu Zhu Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM87501, USA. Email: [email protected]
R. Zimmer Computer Science Department, Brunei University, Uxbridge, Middx. UB8 3PH, UK. Email: [email protected]. uk
xx
Dedication This volume is dedicated to Professor Patrick Parks who died in 1995. Patrick was famous both for his work in nonlinear stability and automatic control and for his more recent contributions to neural networks-especially on learning procedures for CMAC systems. Patrick was known to many as a good friend and colleague, and as a gentleman, and he is greatly missed.
PREFACE
This volume of research papers comprises the proceedings of the first International Conference on Mathematics of Neural Networks and Applications (MANNA), which was held at Lady Margaret Hall, Oxford from July 3rd to 7th, 1995 and attended by 116 people. The meeting was strongly supported and, in addition to a stimulating academic programme, it featured a delightful venue, excellent food and accommodation, a full social programme and fine weather - all of which made for a very enjoyable week. This was the first meeting with this title and it was run under the auspices of the Universities of Huddersfield and Brighton, with sponsorship from the US Air Force (European Office of Aerospace Research and Development) and the London Mathematical Society. This enabled a very interesting and wide-ranging conference programme to be offered. We sincerely thank all these organisations, USAF-EOARD, LMS, and Universities of Huddersfield and Brighton for their invaluable support. The conference organisers were John Mason (Huddersfield) and Steve Ellacott (Brighton), supported by a programme committee consisting of Nigel Allinson (UMIST), Norman Biggs (London School of Economics), Chris Bishop (Aston), David Lowe (Aston), Patrick Parks (Oxford), John Taylor (King's College, London) and Kevin Warwick (Reading). The local organiser from Huddersfield was Ros Hawkins, who took responsibility for much of the administration with great efficiency and energy. The Lady Margaret Hall organisation was led by their bursar, Jeanette Griffiths, who ensured that the week was very smoothly run. It was very sad that Professor Patrick Parks died shortly before the conference. He made important contributions to the field and was to have given an invited talk at the meeting. Leading the academic programme at the meeting were nine invited speakers. Nigel Allinson (UMIST), Shun-ichi Amari (Tokyo), Norman Biggs (LSE), George Cybenko (Dartmouth), Frederico Girosi (MIT), Stephen Grossberg (Boston), Morris Hirsch (Berkeley), Helge Ritter (Bielefeld) and John Taylor (King's College, London). The supporting programme was substantial; out of about 110 who submitted abstracts, 78 delegates were offered and accepted opportunities to contribute papers. An abundance of relevant topics and areas were therefore covered, which was indeed one of the primary objectives. The main aim of the conference and of this volume was to bring together researchers and their work in the many areas in which mathematics impinges on and contributes to neural networks, including a number of key applications areas, in order to expose current research and stimulate new ideas. We believe that this aim was achieved. In particular the meeting attracted significant contributions in such mathematical aspects as statistics and probability, statistical mechanics, dynamics, mathematical biology and neural sciences, approximation theory and numerical analysis: alge-
XXi
XXll MATHEMATICS OF NEURAL NETWORKS
bra, geometry and combinatorics, and control theory. It also covered a considerable range of neural network topics in such areas as learning and training, neural network classifiers, memory based networks, self organising maps and unsupervised learning, Hopfield networks, radial basis function networks, and the general area of neural network modelling and theory. Finally, applications of neural networks were considered in such topics as chemistry, speech recognition, automatic control, nonlinear programming, medicine, image processing, finance, time series, and dynamics. The final collection of papers in this volume consists of 6 invited papers and over 60 contributed papers, selected from the papers presented at the conference following a refereeing procedure of both the talks and the final papers. We seriously considered dividing the material into subject areas, but in the end decided that this would be arbitrary and difficult - since so many papers addressed more than one key area or issue. We cannot conclude without mentioning some social aspects of the conference. The reception in the atmospheric Old Library at LMH was accompanied by music from a fine woodwind duo, Margaret and Richard Thorne, who had first met one of the organisers during a sabbatical visit to Canberra, Australia! The conference dinner was memorable for preliminary drinks in the lovely setting of the Fellows Garden, excellent food, and an inspirational after-dinner speech by Professor John Taylor. Finally the participants found some time to investigate the local area, including a group excursion to Blenheim Palace. We must finish by giving further broad expressions of thanks to the many staff at Universities of Huddersfield and Brighton, US Air Force (EOARD), London Mathematical Society, and Lady Margaret Hall who helped make the conference possible. We also thank the publishers for their co-operation and support in the publication of the proceedings. Finally we must thank all the authors who contributed papers without whom this volume could not have existed.