pulse density recurrent neural network systems … density recurrent neural network systems with...

Pulse Density Recurrent Neural Network Systemswith Learning Capability Using FPGA

YUTAKA MAEDA, YOSHINORI FUKUDA AND TAKASHI MATSUOKADepartment of Electrical and Electronic Engineering

Kansai University3-3-35 Yamate-cho Suita 564-8680

[email protected]

Abstract: - In this paper, we present FPGA recurrent neural network systems with learning capability using thesimultaneous perturbation learning rule. In the neural network systems, outputs and internal values arerepresented by pulse train. That is, analog recurrent neural networks with pulse frequency representation areconsidered. The pulse density representation and the simultaneous perturbation enable the systems withlearning capability to easily implement as a hardware system. As typical examples of the recurrent neuralnetworks, Hopfield neural network and the bidirectional associative memory are considered. Details of thesystems and the circuit design are described. Analog and digital examples for these Hopfield neural networkand the bidirectional associative memory are also shown to confirm a viability of the system configuration andthe learning capability.

Key-Words: Hardware implementation, Pulse density, Hopfield neural network, Bidirectional associativememory, Learning, Simultaneous perturbation, FPGA

1 IntroductionNeural network is interesting research target alsofrom practical applications. Then implementation,especially, hardware implementation is crucialproblem in connection with learning scheme.Hardware implementation of neural networkspromotes many practical applications of neuralnetworks[1,2].

Back-propagation is successful learning scheme.However, this does not match to the hardwareimplementation. This learning scheme requirescomplicated circuit design and wiring problem sothat the circuit size becomes large. On the otherhand, the simultaneous perturbation optimizationmethod is suitable learning scheme of neuralnetworks for hardware realization[3]. Themechanism of learning is so simple that the circuitdesign is simple and easy. Since we can economizecircuit size, we can realize larger scale of neuralnetworks on the same size of a chip.

Next, let us think about representation scheme ofthe neural networks. We have some options such asanalog representation or digital representation[4].The choice of the representation is also importantmatter. Generally, analog approach requires smallercircuit size but is sensitive to noise. Moreover, thecircuit design for analog system is labored task. Onthe other hand, digital system is easy to design and

invulnerable to noise. EDA technique can stronglyassist the design.

If we adopt pulse representation, we can handleanalog quantity based on digital circuit design. Onthe whole, pulse systems compensate these demeritsof the analog and digital systems. Pulserepresentation of neural networks has manybeneficial properties.

If we combine the pulse representation and thesimultaneous perturbation method, we can easilydesign a hardware neural network system withlearning capability. The network can handleanalogue problems as well.

From this point of view, in this paper, wepresents pulse density recurrent neuralnetworks(RNNs) with learning ability using thesimultaneous perturbation optimization method. Therecurrent neural network FPGA systems learndigital and analog problems. Details of the systemsincluding learning mechanism are described. Someresults for analog and digital problems are alsoshown.

2 Pulse Density Representation andSimultaneous Perturbation LearningRule2.1 Pulse Density Representation

WSEAS TRANSACTIONS on CIRCUITS AND SYSTEMS Yutaka Maeda, Yoshinori Fukuda and Takashi Matsuoka

ISSN: 1109-2734 321 Issue 5, Volume 7, May 2008

The pulse density representation is sometimes usedfor neural systems. It is also called different namessuch as stochastic logic[5,6], pulse-mode neuron[7],bit-stream neuron[8], frequency-based network[9]and so on[10]. We also know spiking neuralnetworks, which realize central pattern generator,and their hardware implementation[11].

Totally, the pulse density type of neural networkshas attractive properties. We can summarize themerits of pulse density neural networks as follows; Biological analogy Invulnerability to noisy environment Analogue system realized by digital

system Ease of system design assisted by the

EDA technique Economizing circuit areaThe analogy to biological neural systems is

intriguing. A part of our nerve systems areemploying the pulse density scheme. Especially, itis well-known that nerve impulses are transmittedfrom retina to visual cortex. These systems areinvulnerable to noisy conditions. Limited noises donot result in serious malfunction of the overallsystem. Moreover, we can handle quantized analogquantities, based on the digital technology that isrelatively effortless to design and implement.Recent EDA technique can assist trouble-free designof overall system. In addition, pulse systemsimplifies arithmetic circuit. For example,multipliers are realized by the AND operation. Thisresults in economization of the circuit design.

The learning capability is the most importantfactor of artificial neural network systems. Evenwhen we consider hardware implementation ofneural network systems, realizing the learningability is essential. In other words, it is crucial toadopt suitable learning scheme for hardwareimplementation.

Eguchi et al. reported a pulse density neuralsystem[12]. They used the back-propagation methodas a learning rule. However, the learning schemerequires complicated circuitry, since the learningrule uses derivatives of an error function to updateweights. This results in larger area on chip ordifficulty in circuit design. Particularly, it seemsimpossible to realize large scale of neural networksystem.

On the other hand, the simultaneous perturbationlearning scheme[3] is an option. Because of itssimplicity, the learning rule is suitable for hardwareimplementation[3,13,14]. The learning scheme isapplicable to recurrent neural networks as well[15].

Moreover, if we adopt pulse density representation,we can take advantage of the simultaneousperturbation effectively[16].

The combination of the pulse density system andthe simultaneous perturbation learning rule results inan easy configuration which can be implemented inthe hardware neural network system. In ourhardware implementation, pulse density is used torepresent the weight values, outputs and inputs.

2.2 Analog recurrent neural networkCompared with the feed forward neural networks,recurrent neural networks(RNNs) have inherent andinteresting properties such as dynamics. As a typicalexample of RNN, let us think about Hopfield neuralnetwork(HNN)[17]. HNNs are used to store patternsor to solve combinatorial optimization problems likethe traveling salesman problem. For these problems,the weights in the network are typically determinedby patterns to be memorized based on Hebbianlearning rule[18] or an energy function based on theproblem. If our patterns are analogue or if we cannotfind a proper energy function, it will be impossibleto apply these techniques to find the optimal weightvalues of the network.

Hebbian learning is widely used for the HNNtype of neural networks including the bidirectionalassociative memories. However, this learning rulecan cope with only binary values; +1 and -1 forbipolar representation, +1 and 0 for unipolar one.

Since the HNNs can handle analog quantities assame as the other ordinary neural networks, it isimportant to contrive a new learning scheme andsuitable representation of analog quantities.

The simultaneous perturbation learning schemeis applicable to the analog problems[13]. Moreover,when we combined this with the pulse densityrepresentation of analog quantities, the simultaneousperturbation learning scheme increases in value.

2.3 Learning via simultaneous perturbationWhen we use a RNN for a specific purpose, weneed to determine the proper values of the weightsin the RNN. That is, the so-called learning of RNNsis necessary.

In many applications of RNNs, we know theideal output for the network. Using this information,we can evaluate how well the network performs.Such an evaluation function gives us a clue foroptimizing the weights of the network. Of course,like Hebbian learning, we can determine the weightvalue through off-line learning. However, on-linelearning scheme for RNNs is interesting.



Thus we consider a recursive learning schemefor RNNs. Then back-propagation throughtime(BPTT) is typical example. In order to use theBPTT, the error quantity must propagate throughtime from a stable state to an initial state. Thisprocess is so complicated. It seems difficult to usesuch a method directly, because it takes a long timeto compute the modifying quantities correspondingto all weights. At the same time, it seems practicallydifficult to realize the learning mechanism as ahardware system. In addition, BPTT is notapplicable to the situation that RNNs learn anoscillatory outputs or trajectory solution.

On the other hand, the simultaneous perturbationscheme is proposed[19] and it is shown that thelearning scheme is suitable for the learning of NNsand their hardware implementation[3,13-16]. Thesimultaneous perturbation optimization methodrequires only values of an evaluation function asmentioned. If we know the evaluation of a stablestate, we can obtain the modifying quantities of allweights of the network without complicated errorpropagation through time.

The simultaneous perturbation learning rule forrecurrent neural networks is described as follows;

1t t t t w w w s (1)

J t c t J t

t tc

w s ww s (2)

Where, w(t) denotes the weight vector of a networkat the t-th iteration. α is a positive constant, c is themagnitude of the perturbation. Δw(t) represents acommon quantity to obtain the modifying vector forall the weights. s(t) denotes a sign vector whoseelement si (t) is 1 or -1 with zero mean. The sign ofsi (t) are randomly determined. That is, E(si (t)) = 0,moreover, si (t1) and sj (t2) are independent withrespect to different components i and j, and differenttime t1 and t2. Where, E denotes the expectation.J(w) denotes an error or an evaluation function, forexample, defined by outputs of neurons in a stablestate and a pattern to be embedded.

We can summarize the advantages of thesimultaneous perturbation learning rule for NNs asfollows; Applicability to RNNs Error back propagation through time is not

necessary It is simple Applicability to analogue problems Applicability to oscillatory solutions or

trajectory learning

An energy function is not necessary

3 FPGA Implementation3.1 Design BackgroundThere are some ways of realizing hardware RNNswith learning capability. For example, C.Lehmannet al. reported a VLSI system for a HNN with on-chip learning [20]. Also in our previous research,the FPGA implementation based on digital circuitdesign technology was used to realize the HNN[13].On the other hand, we are considering a FPGAimplementation of a pulse density RNN with arecursive learning capability using the simultaneousperturbation method. Neurons in the network haveconnected each other.

We adopted VHDL (VHSIC HardwareDescription Language) in basic circuit design forFPGA. The design result by VHDL is configured on

MU200－SX60 board (Mitsubishi Electric Micro-

Computer Application Software Co., Ltd.) withEP1S60F1020C6 (Altera) (see Fig.1). This FPGAcontains 57,120 LEs with 5,215,104 bit usermemory. The number of LE prescribes the scale ofthe neural network such as number of neurons.

Fig.1 FPGA board MU200－SX60

Visual Elite (Summit) is used for the basic deign.Synplify Pro(Synplicity) carried out the logical

synthesis for the VHDL. Finally, QuartusⅡ(Altera)

is used for wiring. The result is send to the FPGA

through BYTE-BLASTER-Ⅱ.

In this work we adopt pulse densityrepresentation with sign bit. Maximum number ofpulses is 255. Addition to this, sign bit shows thatthe signal is positive or negative. As a result, we canhandle values in the range [-1(=-255/255)+1(=+255/255)]. As a result, resolution of the valuesis 1/255.

3.2 Configuration



Fig.2 shows overall configuration of the RNNsystem. The system consists of three units of theRNN unit, the weight modification unit and thelearning unit.

The RNN unit realizes ordinary operation of theRNN to obtain a stable state based on certainweights. The weight modification unit updates allweights used in the RNN unit. The learning unitgenerates modifying quantities for all the weightsusing the simultaneous perturbation learning rule.

Fig.2 System configuration.

3.3 Neural network unitPulse density scheme enable the RNN simple, sinceweighted sum of neuron operations are realized byAND and OR operations shown in Fig.3(a). ANDoperations can realize multiplication of inputs andthe corresponding weights. Next OR operation isused for addition of the previous results. The RNNunit consists of the neurons.

We designed the values of weights and outputsby pulse train with sign signals. Therefore,multiplication is replaced by Fig.3(b). Then resultsof the circuit are connected up/down counter. If theresult is positive, we find pulses in the lower output.The line is connected to up counter. If the result isnegative, we find pulse train in the upper line. Theline is connected to down counter. As a whole, thiscircuit realizes a multiplication of pulses with signsignal. We can implement multiplication andsumming up by the simple circuit configuration.Similar idea is used in many works handling pulse-based operation[6,9].

Input-output characteristic or activation functionis realized by saturation of pulse train. Even ifmultiplication of weight value and input is too large,the summing result is limited by the maximumnumber of pulses in a certain specified period.Between the upper and lower limitation, the result is

linearly produced. As a result, a piecewise linearfunction with a certain saturation is realized as theinput-output characteristic of a neuron.

Fig.4

Actually, cothat some overesult, we ocharacteristicactual additiohorizontal axisa and b. Thenumber as ahorizontal axisaxis. Howeverfor the linear255.

AND

OR

w1

w2

wn

x1

x2

xn

(b)

(a) U

wisign

sign

pulse

pulsexi

Neural netunit

Weightmodification

unit

Learningunit

Weight

Initial

inputs

Modifying

quantity Teachingsignals

Perturbation

Outputs (States)


ISSN: 1109-2734 324

Fig.3 Pulse opera

Input-output char

llisions of some prlapped pulses abtain a rounde

as shown in Fig.n result in pulsdenotes a numbervertical axis sho

n output. Therefis theoretical re

, in Fig.4, the actutheoretical one, an

Signed pulse oper

nsigned pulse op

Issue 5, Vo

tion.

acteristic.

ulses are happen, sore neglected. As ad curve as the4. Fig.4 shows ane expression. Theof pulses as inputs

ws observed pulseore, twice of thesult for the verticalal result is roundedd has saturation at

ation

eration

xiwi

(if positive)

xiwi

(if negative)

lume 7, May 2008

Without any specific operation, we can realizesigmoid-like property using pulse expression. Inmany analog hardware neural network systems,there are many reports for realizing the sigmoidfunction and this is one of the points to correctlyrealize the function of overall neural system. Fromthis point of view, pulse expression is beneficial.

Combining this circuit, we can implement theoperation of RNN. Repeating the operation of thecircuit, the system obtains a stable state of thenetwork. This procedure is relatively easy.

With pulse representation, the unit is so simpleand consumes smaller circuit size than ordinarybinary representation as a RNN system. When wehave stable outputs, these outputs signals are sent tothe learning unit to realize the recursivemodification of the weights.

3.4 Leaning unitThe learning unit achieves the so-called learningprocess using the simultaneous perturbation andsends the basic modifying quantity to the weightmodification unit, which is common to all weights.The block diagram is shown in Fig.5.

One of the features of this learning rule is that itrequires only operations of the RNN. When theRNN arrive at a stable state, the state, namely outputis probed by the learning unit. Based on the outputand its corresponding desired output, the learningunit generates modifying quantities for all theweights via the simultaneous perturbation.

Fig.5 Learning unit.

Error defined by the final states of the RNN and

their ideal ones is as follows in this work. Thisabsolute error makes the system simple, because asimple counter can realize this calculation.

j jj

J t O t T t w (3)

Where, Oj and Tj are final output of RNN andtheir teaching signal, respectively. j is index forneuron number in the network.

In the learning process, the following quantity iscommon for all weights.

c

tJtctJt

)()()()(

wsww

(4)

To implement the above quantity easily we setα/c as 1, 1/2, 1/4, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256or 1/512. This process is realized by bit shift.Therefore, the operation becomes very easy andsimplifies the circuit design.

3.5 Weight modification unitFig.6 shows flowchart of the weight modificationunit. This unit controls the weight values of RNN.In this system, perturbed operation of the RNN andordinary operation are required to generate basicquantity for weights modification.

Fig.6 Flowchart of the weight modification unit.

3.6 Other technical factorsIt is generally difficult for hardware system toproduce random numbers. To generate randomnumber, the linear feedback shift register is used.

All system requires control unit. This RNNsystem also requires clock signal and many timingsignals. This system contains the circuit for thispurpose. The clock frequency is 19.5MHz for actual



operation of the FPGA system.Interface for LED display and ten-key, or output

for measurement is not essential from academiapoint of view but important for practical experiment.This function was also designed in this system.

4 HNN SystemHNN is a typical example of recurrent NNsproposed by Hopfield. HNN realizes auto-association in which incomplete or corrupted initialpattern is recovered. Output of HNN is determinedas follows;

1t t O f WO (5)

Where O is state or output vector for neurons ofHNN. f is a output-input characteristic such as thesigmoid function. t denotes iteration. For a certainweight matrix W whose diagonal elements are zero,repeating the calculation of Eq.(5) gives a stablestate. This is basic operation of HNN.

Now, we fabricate the HNN system using FPGA.We consider examples for HNN with 25 neurons.The HNN learns the following two examples. Theperturbation c is 7/255 and the learning coefficient αis (7/255)×(1/26). This setting is empiricallydetermined through preliminary experiments.

4.1 Character problemThe first example is to memorize a character shownin the following figure of capital A. Numberswritten in Fig.7 shows neuron number.

Fig.7 Character A.

Black pixels represent value of 1(=255/255), thatis, corresponding neuron has to produce 255 pulsesfor a certain interval and white pixels are -1(=-255/255), that is, corresponding neuron has toproduce 255 pulses with negative sign signal. Thisis basically a binary problem.

Configuration result is shown in Table 1. As inthe table, 45% of logic elements are used. Somepins are used to observe operation of the system.

Table 1 Configuration result for EP1S60F1020C6.

Fig.8 Change of error in FPGA system.

Fig.9 Recalled patterns.

Fig.8 shows change of the error defined by Eq.(4)in learning process. As learning proceeds, the error



decreases. After about 4000 times learning, thesystem perfectly learns the character of A. Since thesimultaneous perturbation learning rule is based ona stochastic gradient method, the learning curvedoes not monotonously decrease.

Fig.9 shows change of recalled patterns. In thisfigure, black pixel and white one mean that outputsof the corresponding neuron are positive andnegative respectively. If the output of the HNN isunstable, the pixel is depicted in grey.

From Fig.8 and Fig.9, we can see that the systemlearns the pattern A. After about 4000 learning, thesystem recalls the pattern perfectly from a corruptedinitial state.

4.2 Sinusoidal patternNext we consider an analog problem. The HNNsystem must learn sinusoidal wave in the range [-1(=-255/255) +1(=255/255)].

Fig.10 Teaching pattern for each neurons.


The HNN contains 25 neurons which memorizeanalog values shown in Fig.10 respectively. For

example, the first neuron has to output no pulsebecause the desired output is 0, the seventh neuronhas to produce 255 pulses because the output is 1and so on.

Change of the error function in learning processis depicted in Fig.11. Error decreases as iterationproceeds. This figure shows that the system learnsthe sinusoidal pattern as iteration proceeds.

5 BAM SystemBidirectional associative memory proposed byKosko in 1988 is an extension of Hopfield neuralnetwork. The network realizes the hetero-associativememory in which recalled patterns are differentfrom triggering patterns as HNN can realize auto-association.

BAMs consist of two layers. Based on certainweight values and initial states of neurons in the Alayer, inputs of neurons in the B layer are calculated.And input-output characteristic determines outputsof neurons in the B layer. Next these outputs of theB layer and weights become inputs of neurons in theA layer. Then outputs of the A layer determined.That is, the following calculation is carried outrepeatedly.

B AO f WO (6)

A BO f WO (7)

Where OA and OB are state or output vector ofneurons in the A layer and the B layer, respectively.f is a output-input characteristic. For a certainweight matrix W, repeating the operations give astable state of neurons in the network. Then we canobtain certain pair of output pattern of the A layerand the B layer. This is basic operation of BAM.Evaluation function is defined as follows;

Fig.12 BAM unit.

layer

layer

A Aj j

j A

B Bj j

j B

J t O t T t

O t T t

w

(8)



Where, OAj and OA

j denote the i-th elements of OA

and OA respectively. T means their correspondingteaching outputs.

Now, we fabricate the BAM system with 3neurons in the two layers respectively. The BAMlearns the following two examples. The perturbationand the learning coefficient are determined shown inthe following tables through preliminaryexperiments. The BAM unit is shown in Fig.12.

5.1 Binary patternThe first example for BAM is to memorize a pair ofthe following simple pattern shown in Fig.13. Alayer and B layer must learn three black and threewhite pattern. Black and white pixels mean thatoutputs of the pixels denoted 1(=255/255) or -1(=0/255) respectively in stable state.Parameters inthe learning and learning values as the teachingpattern are shown in Table 2.

Fig.13 Teaching pattern for BAM.

Table 2 Parameters and learning values.

Parameters

Perturbation c 0.0156 (4/255)

Learning coefficient α 0.0078 (4/2551/2)

Learning pattern for the layer A

Learning value for neuron 1 1(=255 / 255)Learning value for neuron 2 1(=255 / 255)Learning value for neuron 3 1(=255 / 255)

Learning pattern for the layer B

Learning value for neuron 1 -1(=255 / 255)Learning value for neuron 2 -1(=255 / 255)Learning value for neuron 3 -1(=255 / 255)

Change of the error is shown in Fig.14. Afterabout 110 learning, error decreases suddenly. About200 learning is enough to obtain sufficient recall forthe desired pattern.

Fig.15 shows recall results under learning process.In these results, grey pixels mean that the output ofthe neuron is not stable. After 200 learning, stable

outputs are equal to the desired ones shown inFig.15(d).



5.2 Analogue patternNow, we consider an analogue problem in whichtarget outputs are analogue. When we handleanalogue problems, we cannot use Hebbian learningto determine weight values of the network. Thenrecursive learning scheme is important. Even for theanalogue problems, we can apply the simultaneousperturbation learning scheme as same as digital orbinary problems.

We handle the analogue quantities shown inTable 3. Six neurons have to learn analogue value

such as 0.59(150/255) or 0.78(200/255) and soon. Used parameters are also written in the table.



Table 3 Learning pattern

Parameters

Perturbation c 0.0549 (14/255)

Learning coefficient α 0.0137(14/2551/4)

Learning pattern for the layer A

Learning value for neuron 1 150 / 255Learning value for neuron 2 200 / 255Learning value for neuron 3 250 / 255

Learning pattern for the layer B

Learning value for neuron 1 -150 / 255Learning value for neuron 2 -200 / 255Learning value for neuron 3 -250 / 255

Change of the error is shown in Fig.16. Afterabout 50 learning, error decreases suddenly. About200 learning, the BAM could recall the desiredpattern. The BAM system learns the analogueproblem as well.


6 ConclusionIn this paper, we described FPGA RNN systemswith learning capability based on pulse expression.HNN and BAM were handled as actual systems.The simultaneous perturbation method is adopted asthe learning scheme. We could effectively realize auseful hardware learning mechanism. Moreover, weconfirm a feasibility of recurrent neural networksusing the simultaneous perturbation learning method.

This system could handle both analog and digitalproblems as well.

AcknowledgementThis work is financially supported by MEXTKAKENHI (No.19500198) of Japan, AcademicFrontier Project, and the Intelligence systemtechnology and kansei information processingresearch group, ORDIST, Kansai University.

References:[1] M. Zheng, M. Tarbouchi, D. Bouchard, J.

Dunfield, FPGA Implementation of a NeuralNetwork Control System for a Helicopter,WSEAS TRANSACTIONS on SYSTEMS,Vol. 5, 2006, pp.1748-1751.

[2] C. Lee, C. Lin, FPGA Implementation of aFuzzy Cellular Neural Network for ImageProcessing, WSEAS TRANSACTIONS onCIRCUITS and SYSTEMS, Vol. 6, 2007,pp.414-419.

[3] Y. Maeda, H. Hirano and Y. Kanata : Alearning rule of neural networks viasimultaneous perturbation and its hardwareimplementation, Neural Networks, Vol. 8, No.2, 1995, pp. 251–259.

[4] H. S. Ng and K. P. Lam, Analog and digitalFPGA implementation of BRIN foroptimization problems, IEEE Trans. NeuralNetworks, Vol. 14, 2003, pp.1413-1425.

[5] Y. Kondo and Y. Sawada, Functional abilitiesof a stochastic logic neural network, IEEETrans. Neural Networks, Vol. 3, 1993, pp. 434–443.

[6] S. Sato, K. Nemoto, S. Akimoto, M. Kinjo, andK. Nakajima, Implementation of a newneurochip using stochastic logic, IEEE Trans.Neural Networks, Vol. 14, 2003, pp.1122-1127.

[7] H. Hikawa, A new digital pulse-mode neuronwith adjustable activation function, IEEE Trans.Neural Networks, Vol. 14, 2003, pp. 236-242.

[8] D. Braendler, T. Hendtlass, and P. O’Donoghue,Deterministic bit-stream digital neurons, IEEETrans. Neural Networks, Vol. 13, 2002,pp.1514–1525.

[9] H. Hikawa, Frequency-based multilayer neuralnetwork with on-chip learning and enhancedneuron characteristics, IEEE Trans. NeuralNetworks, vol. 10, 1999, pp.545-553.

[10] M. Martincigh and A. Abramo, A newarchitecture for digital stochastic pulse-modeneurons based on the voting circuit, IEEETrans. Neural Networks, Vol. 16, 2005,pp.1685-1693.

[11] L. Bako, S. Brassai, Spiking Neural NetworksBuilt in FPGAs: Fully Parallel Implementations,WSEAS TRANSACTIONS on CIRCUITS andSYSTEMS, Vol. 5, 2006, pp.346-353.



[12] H. Eguchi, T. Furuta, H. Horiguchi and S.Oteki, Neuron model utilizing pulse densitywith learning circuits, IEICE Technical Report,vol.90, 1990, pp.63-70. (in Japanese)

[13] Y. Maeda and M. Wakamura : Simultaneousperturbation learning rule for recurrent neuralnetworks and its FPGA implementation, IEEETrans. Neural Networks, Vol. 16, No. 6, 2005,pp.1664-1672.

[14] Y. Maeda, A. Nakazawa, and Y. Kanata,Hardware implementation of a pulse densityneural network using simultaneous perturbationlearning rule, Analog Integrated Circuits andSignal Processing, Vol. 18, No. 2, 1999, pp. 1–10.

[15] Y. Maeda and M. Wakamura : Bidirectionalassociative memory with learning capabilityusing simultaneous perturbation,Neurocomputing, Vol. 69, 2005, pp.182-197.

[16] Y. Maeda and T. Tada : FPGA implementationof a pulse density neural network with learningability using simultaneous perturbation, IEEETrans. Neural Networks, Vol. 14, No. 3, 2003,pp.688-695.

[17] J. J. Hopfield, Neurons with graded responsehave collective computational properties likethose of two-state neuron, in Proc. Nat. Acad.Sci., Vol. 81, 1984, pp. 3088–3092.

[18] B. Kosko, Neural Networks and Fuzzy Systems.Englewood Cliffs, NJ: Prentice-Hall, 1992.

[19] J. C. Spall, Multivariable stochasticapproximation using a simultaneousperturbation gradient approximation, IEEETrans. Autom. Control, Vol.37, 1992, pp. 332–341.

[20] C. Lehmann, M. Viredaz, and F. Blayo, Ageneric systolic array building block forneural networks with on-chip learning, IEEETrans. Neural Networks, Vol. 4, 1993, pp.400– 407.



pulse density recurrent neural network systems … density recurrent neural network systems with...

Documents