efﬁcient implementation of the backpropagation algorithm ...lfranco/a40-ortega+jerez++2015.pdf ·...

1

Efficient implementation of the Backpropagationalgorithm in FPGAs and microcontrollersFrancisco Ortega-Zamorano, José M. Jerez, Daniel Urda, Rafael M. Luque-Baena, and

Leonardo Franco Senior Member, IEEE.

Abstract—The well known backpropagation learning algo-rithm is implemented in a FPGA board and a microcontroller,focusing in obtaining efficient implementations in terms ofresource usage and computational speed. The algorithm wasimplemented in both cases using a training/validation/testingscheme in order to avoid overfitting problems. For the case of theFPGA implementation, a new neuron representation that reducesdrastically the resource usage was introduced by combining theinput and first hidden layer units in a single module. Further, atime-division multiplexing scheme was implemented for carryingout product computations taking advantage of the built-in DSPcores. In both implementations, the floating point data typerepresentation normally used in a PC has been changed to amore efficient one based on a fixed point scheme, reducing systemmemory variable usage and leading to an increase in computationspeed. The results show that the modifications proposed produceda clear increase in computation speed in comparison to thestandard PC-based implementation, demonstrating the usefulnessof the intrinsic parallelism of FPGAs in neurocomputational tasksand the suitability of both implementations of the algorithm forits application to real world problems.

Keywords: Hardware implementation, FPGA, Microcon-trollers, Supervised learning, embedded systems.

I. INTRODUCTION

THE backpropagation algorithm (BP) is the most usedlearning procedure for training multilayer neural net-

works architectures. Even if the algorithm was originallyproposed by Werbos in 1974 [1], it was not until 1986that it become popularized through the work of Rumelhartet al. [2]. The BP algorithm is a gradient descent basedmethod that minimizes the error between target and actualnetwork outputs, computing the derivatives of the error in anefficient way [3], [4], [5]. As a gradient descent algorithm thesearch for a solution can get stuck in a local minima but inpractice the algorithm is quite efficient, and as so it has beenapplied to a wide range of areas from pattern recognition [6],medical diagnosis [7], stock market prediction [8], etc. Real-time applications require extra computational resources [9],involving in some cases also energy consumption restrictions,and thus the use of embedded (dedicated) systems [10] or lowpower consumption devices are needed, as in those cases a PCmight not be the most adequate device for executing neuralnetwork models.

The authors are within the Departamento de Lenguajes y Cien-cias de la Computación, Universidad de Málaga, Campus de TeatinosS/N, 29071, Málaga, SPAIN. Corresponding author: L. Franco, e-mail:[email protected], Tel.: +34-952-133304, Fax: +34-952-131397. Copyright(c) 2014 IEEE. Personal use of this material is permitted. However, permissionto use this material for any other purposes must be obtained from the IEEEby sending a request to [email protected]

Field Programmable Gate Arrays (FPGA) are reconfigurablehardware devices that can be reprogrammed to implementdifferent combinational and sequential logic created with theaim of prototyping digital circuits, as they offer flexibilityand speed. In recent years, the advance in technology havepermitted to construct FPGAs with considerable large amountsof processing power and memory storage, and as so theyhave been applied in several domains (telecommunications,robotics, pattern recognition tasks, infrastructure monitoring,etc.) [11], [12], [13]. In particular FPGAs seem quite suitablefor neural network implementations as they are intrinsicallyparallel devices as is the processing of information in neuralnetwork models. Several studies have analyzed the implemen-tation of neural networks models in FPGAs [14], [15], [16],[17], [18], [19], but it is worth noting the difference betweenoff and on chip implementations. In off-chip learning imple-mentations [20], [21] the training of the neural network modelis usually performed externally in a personal computer (PC),and only the synaptic weights are transmitted to the FPGA thatacts as a hardware accelerator, while on-chip learning imple-mentations includes both training and execution phases of thealgorithm [22], [23], [18]. Existing specific implementationsof the artificial neural network backpropagation algorithm inFPGA boards include the works of [24], [25], [26], noting thatdespite recent advances on the computational power of theseboards, still the size of the neural architectures that can beimplemented is quite limited. FPGA boards are predominantlyprogrammed using hardware description languages such asVHDL (VHSIC Hardware Description Language) or Verilogand programming them is usually very time consuming.

Apart from FPGAs, other devices very much used in neuralnetwork applications are microcontrollers, which are smalland low-cost computers built on a single integrated circuitcontaining a processor core, memory, and programmable in-put/output peripherals built for dealing with specific tasks.These devices are commonly used in sensor nodes (widelyused in wireless sensor network (WSN) [27]) usually underenvironmental changing conditions, because they are an eco-nomic, small and flexible solutions to interpret signals fromvarious sensors and take a decision according to the inputsreceived [28], [29], [30], [31], [32], [19]. An advantage ofmicrocontrollers is that they can be easily programmed usingstandard programming languages such as C, C++, Java, etc.,while their main limitations are memory size and computingspeed.

In the present work, we have implemented the backprop-agation algorithm in a VIRTEX-5 XC5VLX110T FPGA and

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.The final version of record is available at http://dx.doi.org/10.1109/TNNLS.2015.2460991

Copyright (c) 2015 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing [email protected].

2

an Arduino Due microcontroller. Our aim was two fold: first,to obtain efficient implementations on both types of devicesthat permit its practical application in real life problems andsecond to compare the efficiency between them and to astandard PC based implementation. The organization of thepresent work is as follows: next section includes details aboutthe BP algorithm. The FPGA implementation is describedin Section III, which contains four parts: the first threesubsections describe each one of the three blocks used for thealgorithm implementation, while the fourth subsection dealswith specific implementation details. Section IV contains themicrocontroller implementation of the algorithm followed bythe Results section that presents results of both implemen-tations on a set of benchmark functions, together with adetailed analysis of the computational costs involved (numberof cycles and execution times) and of the general functioningof the algorithm. The work finishes with the discussion andconclusions.

II. THE BACKPROPAGATION ALGORITHM

The backpropagation algorithm is a supervised learningmethod for training multilayer artificial neural networks, andeven if the algorithm is very well known, we summarizein this section the main equations in relationship to theimplementation of the backpropagation algorithm, as they areimportant in order to understand the current work.

Let’s consider a neural network architecture comprisingseveral hidden layers. If we consider the neurons belonging toa hidden or output layer, the activation of these units, denotedby yi, can be written as:

yi = g

L∑j=1

wij · sj

= g(h) , (1)

where wij are the synaptic weights between neuron i inthe current layer and the neurons of the previous layer withactivation sj . In the previous equation, we have introduced has the synaptic potential of a neuron. g is a sigmoid activationfunction given by:

g(x) =1

1− e−βx(2)

The objective of the BP supervised learning algorithm is tominimize the difference between given outputs (targets) for aset of input data and the output of the network. This errordepends on the values of the synaptic weights, and so theseshould be adjusted in order to minimize the error. The errorfunction computed for all output neurons can be defined as:

E =1

2

p∑k=1

M∑i=1

(zi(k)− yi(k))2, (3)

where the first sum is on the p patterns of the data set andthe second sum is on the M output neurons. zi(k) is thetarget value for output neuron i for pattern k, and yi(k) is thecorresponding response output of the network. By using themethod of gradient descent, the BP attempts to minimize thiserror in an iterative process by updating the synaptic weights

upon the presentation of a given pattern. The synaptic weightsbetween two last layers of neurons are updated as:

∆wij(k) = −η∂E

∂wij(k)= η[zi(k)− yi(k)]g

′i(hi)sj(k), (4)

where η is the learning rate that has to be set in advance (aparameter of the algorithm), g′ is the derivative of the sigmoidfunction and h is the synaptic potential previously defined,while the rest of the weights are modified according to similarequations by the introduction of a set of values called the“deltas” (δ), that propagate the error form the last layer intothe inner ones.

Training and validation processes: The training procedureis executed a certain number of times (epochs) using thetraining patterns. In one epoch, the training patterns are allpresented once in random ordering, adjusting the synapticweights in an on-line manner. A well known and severeproblem affecting all predictive algorithms is the problem ofoverfitting, caused by an overspecialization of the trainingprocedure on the training set of patterns [33]. In order toalleviate this effect, one straightforward alternative is to splitthe set of available training patterns in training, validationand test sets. The training set will then be used to adjustthe synaptic weights according to Eq. 4, while the validationset is used to control overfitting effects, storing in memorythe values of the synaptic weights that have so far led to thelowest validation error, so when the training procedure ends,the algorithm returns the stored set of weights. The test set isused to estimate the performance of the algorithm in unseendata patterns.

III. FPGA IMPLEMENTATION OF THE BP ALGORITHM

FPGAs [34] are reprogrammable silicon circuits, usingprebuilt logic blocks and modifiable routing resources that canbe configured to implement custom hardware. Besides the factthat FPGAs can be completely reconfigured allowing to changeits behavior almost instantaneously by loading a new circuitryconfiguration, they can be also used as hardware accelerators,in particular for neural based applications given their intrinsicparallel computational capabilities. FPGAs are usually pro-grammed using a hardware description language (VHDL). Forthe current implementation we used the Virtex-5 OpenSPARCEvaluation Platform (ML509) that includes a Xilinx Virtex-5 XC5VLX110T FPGA. The board was programmed usingthe “Xilinx ISE Design Suite 12.4” environment within the“ISim M.81d” simulator. Fig. 1 shows a picture of a Virtex-5OpenSPARC board, and Table I shows some characteristics ofthe Virtex-5 XC5VLX110T FPGA, indicating its main logicresources. The table indicates for the mentioned board thenumber of slice registers, Look-Up Tables (LUTs), BondedInput-Output Banks (IOB), Block RAM (BR) and DSP48cores. Given that every computation in the FPGA has tobe defined from first principles, they usually contain DSPsfor helping to perform certain operations. A Digital SignalProcessor, or DSP, is a specialized microprocessor that has anarchitecture optimized for the fast operational needs of digitalsignal processing. A Digital Signal Processor (DSP) process



3

Fig. 1. Picture of a Virtex-5 OpenSPARC Platform used for the implemen-tation of the C-Mantec algorithm.

N1a

Inputs (1, ..., NI)

Output 1

Output NO

Pattern

Validation

Process

Data setting set

S_Train

S_Error Error

Activate

Validation Pattern

Pattern block Control block

Architecture block

Inp_Hid_Neu

Hid_Neu

Out_Neu

Activate

Training Pattern

Activation

signals

Process

Nio

Ni1

NI

1N21N11

N2b

Fig. 2. Scheme of the FPGA design where control, pattern and architectureblocks are shown together with the information exchanged between them.

data in real time, making it ideal for applications that can nottolerate delays [35], [36].

TABLE IMAIN SPECIFICATIONS OF THE VIRTEX-5 XC5VLX110T FPGA RELATED

TO ITS AVAILABLE SLICE LOGIC.

Device Slice Slice Bonded BR DSP48Registers LUTs IOBsVirtex-5 69,120 69,120 34 148 64XC5VLX110T

The FPGA implementation of the BP algorithm was carriedout using three blocks: control, pattern and architecture blocks.The control block organizes the whole information process bysending and processing the information from the architectureand pattern blocks. The pattern block manages the exchangeof information between the PC and FPGA for reading theset of patterns to be stored in blocks RAM, and also is usedto send a given pattern to the architecture block, that it willbe in charge of the training process. Circuit computationshave been programmed using fixed point arithmetic which isthe standard way to work with FPGA boards. Floating pointoperations can be codified in a FPGA but they tend to beinefficient in comparison to fixed point representation [25]. We

describe below the organization of each one of the three blocksin separate subsections, followed by a fourth subsection thatcomments on specific implementations details. Fig. 2 showsa diagram of the FPGA design, where the three blocks usedare shown together with the information that is exchangedbetween them.

A. Pattern block

The pattern block manages the data exchange between thePC and the FPGA board through the serial communicationRS-232 port of the device. This port has been used becauseit can be easily implemented in VHDL and ported to otherarchitectures.

To start the process, the user sends the set of parameters ofthe algorithm and the training patterns. The set of parametersspecifies the number of training (#Train) and validationpatterns (#V alid), the number of neurons in each layer(#Ni), the number of epochs (#Epoch), and the learning ratevalue η. The training and validation data sets are stored in thedistributed-RAM block of the FPGA, storing the training set inthe first positions and the validation in the following ones. Twobytes (1 byte = 8 bits) of memory are used for representingeach attribute and each class of a pattern, and thus the totaloccupied memory of the data set is defined by the equation:

#bytes = 2 · (NI +NO) · (#V alid+#Train), (5)

where NI is the number of inputs (attributes) and NO is thenumber of output classes.

During the execution of the algorithm the pattern blockmight receive two different signals from the control block inorder to send a random training or validation pattern. To avoidtraining several times a given pattern, the memory position ofthe last sent pattern is switched with the one correspondingto the final eligible memory position, while the number ofeligible memory positions is reduced by 1. This action isrepeated until the eligible memory is null, finishing the epochat this moment and starting a following one. The same processis applied for both training and validation sets independently.

B. Control block

The control block organizes the whole information flowprocess within the FPGA board by sending and processingthe information from the architecture and pattern blocks. Thestructure of this block is organized into two main processes:i) First, the main function of this block is to control twoactivation signals that indicate whether a training or a val-idation pattern should be sent to the architecture block. Inorder to perform this action the control block receives a signalvalue from the pattern block that indicates the total number oftraining (#train) and validation (#val) patterns set for the wholelearning procedure. ii) The secondary process of this block isto execute the validation process, included with the aim ofavoiding overfitting effects. In essence, this process computesan error value using the validation set of patterns to store thesynaptic weight values that have led to the smallest validationerror thus far as the training of the network proceeds. The



4

No

Yes

No

Yes

No

End

Begin

Count1

= #Train?

Activate Training Pattern

Count1++

Activate Validate Pattern

Count2++

Count3

= #Epoch?

Count2

= #Valid?Yes

YesE_Epoch

<

E_Min?

E_Min=E_Epoch

Store Network

Sent

stored Network

No

Yes

Yes

No

No

Count1=0, Count2=0,

Count3=0, E_Epoch=0

Count1=0

Count2=0

Count3++

E_Epoch=0

E_Epoch = E_Epoch + Error

Calculate Error

S_Error

= ON?

S_Train

= ON?

Validation

process

Fig. 3. Flowchart of the FPGA control block operations. The region insidethe dashed line corresponds to the validation procedure implemented.

implementation of the whole validation process in the FPGAis detailed in section III-B.

When the computations starts, the set of patterns are loadedinto the pattern block that sends a signal to the control blockin order to start the execution of the algorithm. Fig. 3 shows aflowchart of the control block operations. At the beginningof the process, a set of counters related to the number oftraining patterns, number of validation patterns and number ofepochs are initialized to zero. If the number of actual trainingpatterns has not reached the value # Train (set by the user),the training procedure starts by sending a signal to the patternblock indicating that a random chosen training pattern shouldbe sent to the architecture block. The architecture block willthen train the network, sending back a signal (S_Train) tothe control block when the training of this pattern finishes,increasing the trained pattern counter Count1. When thisvalue gets equal to the total number of training patterns, thenthe validation process start (this step is described in detailbelow). After the validation process, an epoch counter is usedfor checking whether the whole training-validation procedureshould continue or not, as the previous steps are repeated untilthe maximum number of epochs (# Epoch) is reached.

Validation process: The validation process, included toprevent overfitting problems, is executed after finishing atraining epoch. This process requires the storing of the lowestvalidation error obtained so far (as the training procedureadvances) together with the synaptic weights that led to thiserror. When this procedure is activated at the end of a trainingepoch, it computes the mean square error (MSE) for thevalidation set, and if this value is lower than the stored one,then it is saved together with all synaptic weights in a blockRAM using a FIFO procedure. As the neurons included in thenetwork architecture are indexed, the control block demandssequentially the set of synaptic weights associated to eachneuron so they can be stored in a single FIFO RAM whilepreventing memory collision problems. The flowchart of the

TABLE IINUMBER OF EMPLOYED RESOURCES FOR THE REALIZATION OF EACH

TYPE OF POSSIBLE NEURON.

ResourceResource for type of Neuron

Input InpHid Hidden OutputLUTs 524 1126 923 502

Register 254 396 391 255DSP48 1 1 1 1

Block RAM 1 1 1 1

validation process is included in Fig. 3.

C. Architecture block

The architecture block is in charge of the physical imple-mentation of the neural network architecture. The number oflayers and the maximum number of neurons in each layerhas to be predefined by the user before the execution of thealgorithm.

Previous works [37], [38], [23] use three different types ofneurons, corresponding to input, hidden and output layer neu-rons, as they all have different functionalities. Nevertheless, wedecide to use a different approach in order to optimize furtherthe FPGA resources, and thus the proposed implementationeliminate input layer neurons as they are included togetherwith the first hidden layer neurons in a new module that wename input-hidden neurons. The definition of this new typeof module is possible, mainly because the input layer neuronsdo not process the information as they simple act as input tothe network. The implementation of the neurons consists of agroup of LUTs (LookUp Table) with a specific functionalityof the backpropagation algorithm. The input-hidden neuronsmanage the input data, the synaptic weights between the inputand first hidden layer, the output computation of the firsthidden neuron and also the synaptic weights connecting to theoutput or to a further hidden layer. The hidden layer modules(in case they are included) compute the neuron activation andstore the values of the synaptic weights connecting to furtherneurons. Finally, the output modules evaluate the value of theoutput units in order to compute the error of the presentedpattern. A scheme of a two hidden layer neural network isshown in Fig. 2 where the three different types of modulesare indicated. The table II shows the number of employedresources by each type of neuron with a word size of 32 bits,16 for the integer part (N1) and 16 to the decimal part (N2).The election of the word size is described in section III-D1

To specify a given architecture the number of active neuronsin each layer should be sent to the FPGA as part of thesetting data set. However the system are composed of a deter-mined number of neurons and layers, so that the architectureof the network and the maximum number of neurons arepredetermined and delimited by the resources of the device.The novel layer (the first layer blocks), that is employed inthis implementation, reduces the required resources for anyarchitecture. Table III shows board resources needed by theproposed method (Prop.) for different size neural architecturesin comparison to a conventional implementation (Conv.) , andalso to the published results in Ref. [38] (Gomp.).



5

TABLE IIINUMBER OF EMPLOYED RESOURCES FOR THE REALIZATION OF EACH

TYPE OF POSSIBLE NEURON.

Architecture TypeResource

LUTs Regis. BR DSP

10− 3− 1Conv. 11044 6676 79 15Gomp. 8043 2243 – 70Prop. 6413 4151 69 5

10− 6− 3− 2Conv. 17084 9277 86 22Gomp. 20021 5342 – 169Prop. 13062 6767 76 12

10− 50− 1Conv. 54425 25053 126 62Prop. 59335 22763 116 52

30− 30− 10−2 Conv. 56177 26478 137 73Prop. 46547 19008 107 43

50− 10− 10− 5Conv. 49703 24503 140 76Prop. 25533 11853 90 26

60− 15− 10− 5Conv. 59558 28998 155 91Prop. 33163 13833 95 31

Reduction mean 25.8% 35.2% 23.5% 50.1%

D. Implementation details

We describe below details related to the choice of synapticweights precision, for carrying out the implementation ofproducts, and about the computation of the sigmoid functionused as the transfer function of the neurons.

1) Synaptic weights precision: The representation of thesynaptic weights can be chosen according to the availableresources, taking into account that requiring higher accuracymay need a larger representation, leading also to an increasein the number of LUTs per neuron (consequently reducingthe maximum number of available neurons), and a decrease inthe maximum operation frequency of the board. On the otherhand, synaptic weights accuracy cannot be much reduced, asa proper operation of the backpropagation algorithm requiresa certain level of precision [39], [25]. A synaptic weight isrepresented by a bit array with integer and fractional parts oflength N1 and N2. N1 determines the minimum and maximumvalues that can be represented −2(N1−1) to 2(N1−1) while N2

defines the accuracy 2(−N2). The number of bits needed torepresent all possible discrete values within a certain rangeof positive values depends on the difference between themaximum and minimum of the interval, and can be obtainedfrom the following equation:

#bits = log2((1 +max(wij))/(min(wij))). (6)

Table IV shows the number of LUTs needed to representeach type of neuron modules as a function of the numberof bits used for representing the synaptic weights. N1 andN2 indicate the integer and fractional parts of the synapticweight representation. The last column shows the maximumwhole system operation frequency allowed for the chosenrepresentation.

TABLE IVNUMBER OF LUTS PER EACH TYPE OF POSSIBLE NEURON MODULE

ACCORDING TO THE NUMBER OF BITS USED FOR REPRESENTING THESYNAPTIC WEIGHTS.

N1 N2LUTs per type of Neuron fmax

Input InpHid Hidden Output (MHz)8 8 347 723 592 343 191.28 12 390 871 671 409 186.78 16 433 1028 763 448 183.712 12 430 913 740 425 183.712 16 476 1031 859 475 180.616 16 524 1126 923 502 177.3

2) Product implementation: The execution of the backprop-agation algorithm requires the computation of several products,mainly between neuron activations and synaptic weights values(see Eqs. 1 - 4), and so an efficient implementation of thisoperation is crucial in order to optimize board resources(affecting the number of LUTs per neuron required andthe operation frequency of the FPGA). Multipliers can beimplemented by shifters and adders, following the approachpresented in [40] or by available specific DSP cores in theFPGA. The number of required LUTs for the first type ofimplementation is proportional to the bit size of the input data,as for example, for two vectors with Na and Nb bits lengthrespectively, the product requires Na×Nb LUTs while for thesecond type of implementation, one DSP for each neuron isneeded (clearly this puts a limitation in the maximum numberof neurons in the system). We decided to use the DSP basedstrategy as the board frequency operation can be up to fourtimes faster, as we measured the operation of the board withoutusing the DSPs.

For an efficient use of the DSP resources, we implemented atime-division multiplexing scheme, using only one multiplierblock per neuron and thus performing sequentially the com-putation of several products. This time-division multiplexingscheme is shown in Fig. 4. The neuron module comprises afirst block (we named it neuron block) that includes severalLUTs, registers and one BR, while a second block consistsjust of a DSP. Both blocks are synchronous but the DSP usesa frequency two times larger than the one used by the neuronblock, so that a product operation could be completed in oneoperation cycle of the FPGA.

3) Implementation of the Sigmoid function: The operationof the neurons involves the computation of sigmoid functionsfor obtaining the output value of the neurons. As we areusing a fixed point representation (as it is more efficient thanthe floating point one), then the computation of the sigmoidfunction needs the use of an approximation. A look-up tablecontaining equispaced values of the function has been createdto obtain certain number of output values. Neverthless, ashigh precision values are needed for the correct execution ofthe algorithm, the computation of the function approximationwas further complemented by a linear interpolation procedureusing the two adjacent tabulated values (lower and larger)with respect to the input. Storing table values requires largeamounts of memory, and as one table per neuron is needed, thisnumber should be optimized. 64 tabulated values were used, asthis ensures the obtention of absolute errors lower than 10−3.These values start from −8 to 8 with 0.25 increasing steps



6

Neuron module

Nx

Ny

Nx+Ny

CLK_b ( 2f MHz)

S_Error

Error

S_Train

Pattern

Data setting set

CLK_a ( f MHz)

Neuron block

- LUTs

- Registers

- BR

DSP48a

b

c=a*b

Fig. 4. Scheme of an implemented neuron that uses a time-division multi-plexed strategy to execute several multiplications. Neuron and DSP blocksare synchronous but the DSP uses a frequency two times larger than the usedby the neuron block, so that a product operation could be completed in oneoperation cycle of the FPGA

−10 −5 0 5 10−1

−0.5

0

0.5

1x 10

−3 Absolute error

−10 −5 0 5 10−0.02

−0.01

0

0.01

Relative error

Fig. 5. The computation of the exponential function and its approximationbased on a lookup table plus a linear interpolation scheme (Top graph).Absolute (Middle graph ) and relative errors (Bottom graph) committed inthe approximation of the function (See text for more details).

(values lower than −8 and higher than 8 were set to 0 and 1respectively). Fig. 5 (Top) plots real, tabulated and interpolatedvalues for the sigmoid function, with the inset plot showingan enlargement of a portion of the curve. Fig. 5 (Middle andBottom graphs) shows absolute and relative errors involvedin the computation of the sigmoid function in the range from−10 to 10.

Fig. 6. Picture of an Arduino DUE board used for the implementation of theC-Mantec algorithm.

IV. MICROCONTROLLER (µC) IMPLEMENTATION

We have further implemented the backpropagation algo-rithm in an Arduino DUE microcontroller. We describe belowin several subsections all the details of the implementationprocess, highlighting the results of a comparison carried outbetween using a fixed point representation or a floating pointone.

A. The Arduino board

Arduino is a single-board microcontroller designed to makethe process of using electronics in multidisciplinary projectsmore accessible [41]. The hardware consists of a simple opensource board designed around an 32-bit Atmel ARM coremicrocontroller, and the software includes a standard program-ming language compiler that runs in a standard PC and a bootloader for loading the compiled code on the microcontroller.Arduino is a descendant of the open-source Wiring platformand is programmed using a Wiring-based language (syntax andlibraries), similar to C++ with some slight simplifications andmodifications, and a processing-based integrated developmentenvironment.

The Arduino DUE is based on the SAM3X8E ARM Cortex-M3 CPU [42], and it has 54 digital input/output pins (of which12 can be used as PWM outputs), 12 analog inputs, 4 UARTs(hardware serial ports), a 84 MHz clock, an USB OTG capableconnection, 2 DAC (digital to analog), and a reset and erasebuttons. The SAM3X has 512 KB (2 blocks of 256 KB) offlash memory for storing code, it also comes with a preburnedbootloader that is stored in a dedicated ROM memory. Theavailable SRAM amounts to 96 KB in two contiguous banksof 64 and 32 KB. A picture of the Arduino DUE board isshown in Fig. 6.

B. Learning and execution phases of the algorithm

The implementation of the backpropagation neural networklearning model comprises two phases: the learning phasewhere the synaptic weights of the chosen architecture areadjusted according to the set of patterns presented to thenetwork, and the execution phase in which the microcontrolleroutputs a signal in response to sensed input data according tothe model previously adjusted.

The learning phase has been divided into two different pro-cesses: loading of input patterns and neural network training.



7

Data can be loaded into the microcontroller memory on-lineby I/O pins or by a serial communication USB port (this lastoption was the used one in the current implementation forsimplicity reasons), but in both cases, the patterns have to bestored into the memory board because the learning processworks in cycles in which patterns are repeatedly used. Themicrocontroller memory has been further divided in two parts,one of 64 KB for the loading process, which is used for storingthe inputs patterns, and a second part comprising 32 KB ofmemory to be used for system variables. This second part ofmemory is used for storing the value of the synaptic weights,and all other variables of the algorithm, like the activationvalue of the neurons, the deltas, the learning rate, etc. Theneural network training phase consists of the backpropagationalgorithm itself that was implemented with a validation phaseto avoid overfitting effects. Once the training phase finishesthe synaptic are stored in the memory block of 64 KB.

The execution phase is programmed to be carried out fromexternal data, as usually the microprocessor will be used asan independent sensor. In this mode, a pattern would be readfrom a sensor connected to one of the ports of the board, andthe previously trained model will be executed to obtain theneural network output.

C. Pattern storage

The number of bits used for representing each of theinputs of a pattern has to be decided in advance of theimplementation. In the present case, 8 bits have been usedto represent each variable, taking into account that theseinput values have to be previously normalized between 0 and255. Using this representation for the patterns, the maximumnumber of samples that can be stored in a 64KB memory isgiven by the following equation:

NP · (NI +NO) ≤ 65536 , (7)

where NP is the number of patterns, NI is the number ofinput variables (the dimension of the patterns) and NO is thenumber of outputs of the patterns that determines the numberof output of the neural architecture.

D. Data type representation

The microcontrollers are devices with limited computingpower so in order to speed up the learning process, we decidedto utilize a fixed point data representation. We note thatfloating point is the usual data type representation used in thiskind of device but this representation is not always the mostefficiency. This paradigm shift involves important changes inthe way the BP algorithm is programmed but in return offersa faster learning process and a smaller size representationof variables. The following list give details of the type ofrepresentation used for variables related to the implementationof the neurons:

- deltas (δ): 2 bytes integer.- Synaptic weights(w): 2 bytes integer.- Outputs (y): 2 bytes integer.

The previous choice for the representation of the neuralnetwork related variables affects the maximum network size

that can be utilized. The total number of neurons (NN ) in thewhole architecture can be expressed as the sum of the numberof neurons in each layer (NN = NN1 + NN2 + ...), so themaximum number of neurons used in each layer should verifythe following constraint:

2 · (NN1 +NN2 + ...) +

+2 · (NI ·NN1 +NN1 ·NN2 +NN2 ·NN3 + ...) +

+2 · (NN1 +NN2 + ...) 6 32768, (8)

where the first term relates to the variable storage space for theδs, the second term account for the synaptic weights betweenall the layers of neurons, and the last term is related to theoutput value of the neurons in each layer. (NN1 represents thenumber of neurons in the first layer, NN2 of the second layerand so on, while NI is used for the number of inputs).

From the 2 bytes used for representing the variables ofthe system, 10 bits were used for the decimal part, and theremaining 6 for the integer part, and thus the value of thesystem variables ranges between 32 and −32. A special casewas the representation used for the variable computing thesummation of the synaptic potential, because in order to avoidsaturation effects a 4 bytes representation was used.

E. Computation of the Sigmoid function

The computation of the sigmoid function can be imple-mented using the specific ALU (arithmetic and logic unit) forresolving the exponential function. The previous computationinvolves two different variable conversions, the first related tothe input values of the sigmoid (casting from integer to a float-ing point representation), and the second conversion is doneto the output value in a reverse casting. The computationalcost of implementing the previous method is high, with anapproximate time of operation of 62µs, and thus an alternativemethod based on a lookup table plus linear interpolationof adjacent values, similar to the one used in the FPGAimplementation, was chosen. The method is explained in detailin section III-D3, and in this case the computation timeemployed is reduced to 2µs (97% reduction in comparisonto the first mentioned method).

F. Fixed point vs floating point representation comparison

Figs. 7 Top, middle and bottom show the number of timesthat the implementation based on integer data type is faster incomparison to the floating point representation as a functionof the number of neurons in the different layers of the neuralarchitecture. An architecture with only one hidden layer hasbeen used to compute the values represented in the figure,where NN represents the number of neurons, NI the numberof inputs and NO is used for the number of outputs. The Fig.7 top shows the comparison for variable values of NN andNI (keeping fixed NO equal to 1), Fig. 7 middle is computedfor different values of NN and NO with NI equals to 10,and finally Fig. 7 bottom represents the values obtained as afunction of NI and NO for NN equals to 5.



8

2 4 6 8 10 12 14 16 18 208

9

10

11

12

13

14

15

16

17

18

Number of Neurons

Nu

mb

er

of

tim

es

Number of times that integer is faster than !oating point representation

NO

=1

2 Inputs

5 Inputs

10 Inputs

20 Inputs

40 Inputs

2 4 6 8 10 12 14 16 18 208

9

10

11

12

13

14

15

16

17

18

Number of Neurons

Nu

mb

er

of

tim

es


NI=10

1 Output

2 Outputs

3 Outputs

5 Outputs

10 Outputs

5 10 15 20 25 30 35 40 45 508

9

10

11

12

13

14

15

16

17

18

Number of inputs

Nu

mb

er

of

tim

es


NN

=5

1 Output

5 Output

10 Output

15 Output

20 Output

Fig. 7. Number of times that the fixed point representation used in themicrocontroller is faster than the floating point one for different number ofneurons in the layers of a one hidden layer architecture (see text for moredetails).

V. RESULTS

We analyze in this section several aspects in relationshipto the two implementations of the backpropagation algorithmcarried out in a FPGA board and in an Arduino DUE mi-crocontroller, considering also a third implementation of thealgorithm in personal computer (PC) for comparison purposes.The PC implementation of the algorithm has been executedunder Matlab code and run in an Intel(R) core (TM) i5-3330CPU @3.00GHZ with 16 GB of RAM memory. All threeimplementations of the algorithm follow the same operationsteps and the only evident differences between them arethe random number generator used for the initialization ofthe synaptic weights, the type of data representation usedin each case, and the computation of the sigmoid function.

100

102

104

106

FPGA

Microcontroller

PC−Matlab

Type of implementation

Number of clock cycles

Number of clock cycles for learning one pattern

← 64·NO

+22·NO

·NN

+

+20·NN

·(NO

+NI)+N

N+N

O

←

80·NN

·(NI+N

O)+N

N+N

O

10+2·NO

+NI

11+NI+N

O+N

N

calculate output

learning

Fig. 8. Number of clock cycles involved in the learning process of a singlepattern according to the type of implementation used for the case of a 4−5−3neural network architecture.

The FPGA implementation uses a LFSR random numbergeneration routine, while the microcontroller and the Matlabcode use the built-in random and randn functions respectively.

Fig. 8 shows the estimated number of clock cycles that eachimplementation employs for the learning of an input patternfor a single hidden layer neural network architecture witha 4-5-3 structure. Each bar in the graph is further dividedin two parts: computation of the output of the network inresponse to the input pattern (clear part of the bar), and cyclesinvolved in the modification of the synaptic weights includingthe backward phase of the BP algorithm (dark part of thebar) (Note the logarithmic scale of the graph). The number ofclock cycles in the FPGA implementation has been measuredstraightforwardly using the ISIM simulator. The estimation ofthe number of cycles for the microcontroller has been doneby computing the time that takes the computation of eachinstruction of the algorithm, multiplying this value by the clockfrequency of the microcontroller (80 MHZ) and then summingover all the instructions involved in the algorithm. The numberof cycles used in the PC-Matlab implementation cannot becomputed directly and has been estimated by measuring thetotal computation time and multiplying this value by the CPUfrecuency (3 GHZ), even if a strict evaluation of the numberof cycles should involve taking into account other factors likeinstructions of the operative system, libraries, etc. The methodused in the FPGA and microcontroller implementations per-mits to estimate the number of operation cycles as a functionof the number of neurons in one hidden layer architecture(NI , NN , and NO) and is represented inside the bars in Fig.8. It is worth noting that even if in principle microcontrollerand computer codes used are quite similar, there are differ-ences regarding the implementations as the data representationused is different (fixed and floating point respectively), andthe computation of the sigmoid function values is done ina different way (tabulated values plus interpolation for themicrocontroller vs ALU in the case of a computer). To testthe correct implementation of the BP algorithm in the FPGAand microcontroller devices, we tested the training, validationand test errors on a set of benchmark problems from the UCI



9

database [43] frequently used in the literature. Table V showsthe accuracy (generalization ability) values obtained for thethree implementations of the algorithm for eleven benchmarkproblems. The first three columns indicate the data set name,number of inputs and outputs respectively, while the last threecolumns shows the accuracy obtained by using neural networkarchitectures with 5 neurons in the single hidden layer, as thenumber of inputs and outputs is determined by the problemsthemselves. We have not optimized the neural architecturefor each problem as our aim is to demonstrate the correctimplementation of the algorithm and not to obtain optimalvalues of prediction accuracy. For carrying out the simulationsa training, validation and test sets splitting was used in a 50-20-30 % scheme; in which the validation set was used to find thenumber of epochs for evaluating the test error, the maximumnumber of epochs was set to 1000, and the learning rate wasequal to 0.2.

We further computed execution times for the whole learningprocess (training and validation) for the same set of benchmarkfunctions mentioned above, and the results are shown in Fig.9 for the FPGA, microcontroller and PC versions of thebackpropagation algorithm (note the logarithmic scale usedin the Y-axis of the figure). We also perform an analysis tosee how FPGA and microcontroller performances behaves incomparison to the PC implementation as the complexity ofthe functions grow. We have used the execution time neededin the PC implementation timePC as an estimation for thecomplexity of the benchmark functions, to obtain that thenumber of times that the FPGA implementation is faster thanthe PC (#timesFPGA) grows as #timesFPGA = 94 +2 · timePC (Pearson correlation coefficient equals to 0.753),while the analysis for the microcontroller shows not significantperformance increase: #timesµC = 1.6 + 0.0083 · timePC

(Pearson correlation coefficient equals to 0.25).

TABLE VACCURACY

Function #I #O PC FPGA µCDiabetes 8 2 78.31 79.35 79.13Cancer 9 2 95.63 95.73 95.60

Statlog (Heart) 13 2 78.52 78.27 78.26Climate 18 2 93.27 94.14 94.51

Ionosphere 34 2 88.21 87.57 87.14HeartC 35 2 78.80 80.11 80.22Iris 4 3 92.22 92.77 90.89

Balance Scale 4 3 87.93 87.82 87.61Seeds 7 3 97.62 96.51 96.66Wine 13 3 88.89 87.04 86.67Glass 10 6 93.85 91.54 92.31

Average 88.48 88.26 88.09

Fig. 10 shows the Root Mean Square Error (RMSE) ob-tained for training and validation processes when the BPalgorithm is implemented in the FPGA, µC and PC as afunction of the number of epochs for the Iris data set using a4-5-3 architecture (similar results were observed for all datasets). It can be seen that the training error always decreases astraining advances being lower for the PC implementation thanfor other two, indicating that a more precise representation (32bit floating point) helps to adjust the synaptic weights during

10−1

100

101

Iris WineSeeds

GlassStatlog (Heart)

Balance Scale

HeartCIonosphere

CancerDiabetes

Climate

tim

e(s

)

Execution times for a set of benchmark functions

FPGA

Micro

PC

Fig. 9. Execution times (in Seconds) for the whole learning process (trainingand validation) for the set of benchmark functions used for verifying thecorrect implementation of the algorithm in the FPGA and Microcontrollerusing neural network architectures with a single hidden layer (see text moredetails).

0 200 400 600 800 10000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Number of epochs

Ro

ot

Me

an

Sq

ua

re E

rro

r

Root mean square error for validation and training processes

Training in PC

Validation in PC

Training in FPGA

Validation in FPGA

Training in Microcontroller

Validation in Microcontroller

Fig. 10. Root mean square error for the training and validation process whenlearning the Iris data set using a 4-5-3 architecture for the three differentimplementations used.

training. However for the validation RMSE, the three curvesare quite similar noting also that the values increase at certainpoint of the process (approximately at 200 epochs), indicatingoverfitting effects and justifying the use of a validation set.As in general the interest on the application of supervisedneural networks to practical problems is related to prediction,validation and test errors are the important features and thusthe results confirm that the representation used for the FPGAand µC (16 bits fixed point) is adequate.

VI. DISCUSSION AND CONCLUSIONS

The backpropagation algorithm has been successfully im-plemented in a FPGA board and in an Arduino microcon-troller, in a learning paradigm that includes a validationscheme in order to prevent overfitting effects. The implemen-tation of the algorithm in the FPGA board involved severalchallenges as hardware programming is a totally different



10

paradigm approach in comparison to standard software pro-gramming, and as such, we have first introduced a new neuronrepresentation that permits to increase the efficiency of thetraditional implementation, obtaining an average reduction of25.8% in the total number of LUTs needed to implementdifferent architectures, as shown in Table III. Further improve-ments are related to a time-division multiplexing scheme forcarrying out product computations taking advantage of theFPGA DSP built-in cores, and a lookup table plus linearinterpolation scheme for computing the transfer function ofthe neurons (the Sigmoid function). At the time of a realimplementation, the limitation in terms of the size of theneural network architectures that can be simulated would comefrom the specific FPGA board used, and this analysis canbe done from the results shown in table II. In our case inwhich we are using a Xilinx Virtex V XC5VLX110T boardthe main limitation comes from the number of available DSPcores, as the mentioned board includes 64 DSP cores, andthus this factor limits the maximum number of neurons inthe architecture to 63, as an extra core is needed in thevalidation process. If we compare these new results withprevious published works, for example those appearing in [38],we see a significant efficiency increase that will permit theutilization of much larger neural architectures. Neverthelessthe efficiency increase for a particular case would depend onthe values of the combination of neural network architectureparameters and resources of the FPGA board used.

Considering the microcontroller implementation, the stan-dard floating point data type representation has been changedto a more efficient one based on a fixed point scheme, reducingsystem memory variable usage and an increase in computationspeed, obtaining that the Fixed point representation is approx-imately 10 times faster than the floating point one (see Fig.7).

The results shown in Fig. 10 indicate that the representationused for the FPGA and microcontroller was adequate asvalidation and test errors were similar to the PC imple-mentation of the algorithm. The use of a lower precisionrepresentation affects the learning error (it is lower for thePC implementation) but it is not relevant regarding predictionaccuracy. We hypotethize that this fact might be related to theeffect observed when noise is added both to input and synapticweight values [44], that instead of having negative effects, itcan help to improve generalization by preventing overfittingeffects.

An estimation of the computation time (Fig. 9 involvedin relationship with the three implementations of the BPalgorithm (FPGA, µC and PC) shows the potential advantagesof using a FPGA board as a hardware accelerator device forneurocomputing applications, obtaining a speed up of hundredof times with an improvement increasing with the complexityof the problem, highlighting the intrinsic parallelism of thedevice.

As an overall conclusion, the present work shows thepotential advantages of using FPGA boards as hardware ac-celerator devices for neurocomputing applications giving theirintrinsic parallel capabilities, while in relationship to the useof neural networks in microcontrollers we highlight the "on-

chip" characteristic of the presented implementation that willpermit its use in remote sensors using a stand-alone operationmode.

VII. ACKNOWLEDGEMENTS

The authors acknowledge support from Junta de Andalucathrough grants P10-TIC-5770 and P08-TIC-04026, and fromCICYT (Spain) through grant TIN2010-16556 (all includingFEDER funds).

REFERENCES

[1] P. J. Werbos, “Beyond regression: New tools for prediction and analysisin the behavioral sciences,” Ph.D. dissertation, Harvard University, 1974.

[2] D. Rumelhart, G. Hinton, and R. Williams, “Learning representationsby back-propagating errors,” Nature, vol. 323, no. 6088, pp. 533–536,1986.

[3] K. Mehrotra, C. K. Mohan, and S. Ranka, Elements of Artificial NeuralNetworks. Cambridge, MA, USA: MIT Press, 1997.

[4] S. Haykin, Neural Networks: A Comprehensive Foundation, 2nd ed.Upper Saddle River, NJ, USA: Prentice Hall PTR, 1998.

[5] R. D. Reed and R. J. Marks, Neural Smithing: Supervised Learning inFeedforward Artificial Neural Networks. Cambridge, MA, USA: MITPress, 1998.

[6] J. Li, K. Ouazzane, H. Kazemian, and M. Afzal, “Neural networkapproaches for noisy language modeling,” IEEE Transactions on NeuralNetworks and Learning Systems, vol. 24, no. 11, pp. 1773–1784, Nov2013.

[7] K. Misra, S. Chattopadhyay, and D. Kanhar, “A hybrid expert tool forthe diagnosis of depression.” Journal of medical imaging and healthinformatics., vol. 3, no. 1, pp. 42–47, 2013.

[8] H. Mo and J. Wang, “Volatility degree forecasting of stock marketby stochastic time strength neural network.” Mathematical Problems inEngineering., vol. 2013, pp. 1–11, 2013.

[9] G.-B. Huang and C.-K. Siew, “Real-time learning capability of neuralnetworks,” IEEE Transactions on Neural Networks, vol. 17, no. 4, pp.863–878, July 2006.

[10] S.-M. Baek and J.-W. Park, “Hessian matrix estimation in hybrid systemsbased on an embedded ffnn,” IEEE Transactions on Neural Networks,vol. 21, no. 10, pp. 1533–1542, Oct 2010.

[11] E. Monmasson, L. Idkhajine, M. Cirstea, I. Bahri, A. Tisan, andM. Naouar, “Fpgas in industrial control applications,” IEEE Transactionson Industrial Informatics, vol. 7, no. 2, pp. 224–243, May 2011.

[12] D. Bacon, R. Rabbah, and S. Shukla, “Fpga programming for themasses,” Queue, vol. 11, pp. 40–52, 2013.

[13] P. Conmy and I. Bate, “Component-based safety analysis of fpgas,” IEEETransactions on Industrial Informatics, vol. 6, no. 2, pp. 195–205, May2010.

[14] D. Le Ly and P. Chow, “High-performance reconfigurable hardwarearchitecture for restricted boltzmann machines,” IEEE Transactions onNeural Networks, vol. 21, no. 11, pp. 1780–1792, Nov 2010.

[15] Q. N. Le and J.-W. Jeon, “Neural-network-based low-speed-dampingcontroller for stepper motor with an fpga,” IEEE Transactions onIndustrial Electronics, vol. 57, no. 9, pp. 3167–3180, Sept 2010.

[16] W. Mansour, R. Ayoubi, H. Ziade, R. Velazco, and W. E. Falouh, “Anoptimal implementation on fpga of a hopfield neural network,” Advancesin Artificial Neural Systems, vol. 2011, pp. 1–9, 2011.

[17] L.-W. Kim, S. Asaad, and R. Linsker, “A fully pipelined fpga ar-chitecture of a factored restricted boltzmann machine artificial neuralnetwork,” ACM Trans. Reconfigurable Technol. Syst., vol. 7, no. 1, pp.5–23, Feb. 2014.

[18] F. Ortega-Zamorano, J. Jerez, and L. Franco, “Fpga implementation ofthe c-mantec neural network constructive algorithm,” IEEE Transactionson Industrial Informatics, vol. 10, no. 2, pp. 1154–1161, May 2014.

[19] F. Ortega-Zamorano, J. M. Jerez, J. L. Subirats, I. Molina, and L. Franco,“Smart sensor/actuator node reprogramming in changing environmentsusing a neural network model,” Engineering Applications of ArtificialIntelligence, vol. 30, no. 0, pp. 179 – 188, 2014.

[20] S. Himavathi, D. Anitha, and A. Muthuramalingam, “Feedforward neuralnetwork implementation in fpga using layer multiplexing for effectiveresource utilization,” IEEE Transactions on Neural Networks, vol. 18,no. 3, pp. 880–888, May 2007.



11

[21] T. Orlowska-Kowalska and M. Kaminski, “Fpga implementation of themultilayer neural network for the speed estimation of the two-mass drivesystem,” IEEE Transactions on Industrial Informatics, vol. 7, no. 3, pp.436–445, Aug 2011.

[22] A. Dinu, M. Cirstea, and S. Cirstea, “Direct neural-network hardware-implementation algorithm,” IEEE Transactions on Industrial Electronics,vol. 57, no. 5, pp. 1845–1848, May 2010.

[23] J. Shawash and D. Selviah, “Real-time nonlinear parameter estimationusing the levenberg-marquardt algorithm on field programmable gatearrays,” IEEE Transactions on Industrial Electronics, vol. 60, no. 1, pp.170–176, Jan 2013.

[24] A. Omondi and J. Rajapakse, FPGA Implementations of Neural Net-works. Secaucus, NJ, USA: Springer-Verlag New York, Inc., 2006.

[25] A. Savich, M. Moussa, and S. Areibi, “The impact of arithmeticrepresentation on implementing mlp-bp on fpgas: A study,” IEEETransactions on Neural Networks, vol. 18, no. 1, pp. 240–252, Jan 2007.

[26] A. Gomperts, A. Ukil, and F. Zurfluh, “Implementation of neuralnetwork on parameterized fpga.” in AAAI Spring Symposium: EmbeddedReasoning. Stanford University, CA, USA, 2010, pp. 45–51.

[27] J. Yick, B. Mukherjee, and D. Ghosal, “Wireless sensor network survey,”Comput. Netw., vol. 52, no. 12, pp. 2292–2330, Aug. 2008.

[28] M. Sayed-Mouchaweh and E. Lughofer, Learning in non-stationaryenvironments: Methods and Applications. Springer, New York, 2012.

[29] D. Urda, E. Canete, J. L. Subirats, L. Franco, L. Llopis, and J. M.Jerez, “Energy-efficient reprogramming in wsn using constructive neuralnetworks,” International Journal of Innovative, Computing, Informationand Control, vol. 8, pp. 7561–7578, 2012.

[30] E. E. Canete, J. Chen, R.Luque, and B. Rubio, “Neuralsens: A neuralnetwork based framework to allow dynamic adaptation in wireless sensorand actor networks.” J. Network and Computer Applications, vol. 35,no. 1, pp. 382–393, 2012.

[31] S. Mahmoud, A. Lotfi, and C. Langensiepen, “Behavioural patternidentification and prediction in intelligent environments,” Appl. SoftComput., vol. 13, no. 4, pp. 1813–1822, Apr. 2013.

[32] M. Rassam, A. Zainal, and M. Maarof, “An adaptive and efficientdimension reduction model for multivariate wireless sensor networksapplications,” Appl. Soft Comput., vol. 13, no. 4, pp. 1978–1996, Apr.2013.

[33] D. M. Hawkins, “The problem of overfitting,” Journal of ChemicalInformation and Computer Sciences, vol. 44, no. 1, pp. 1–12, 2004.

[34] S. Kilts, Advanced FPGA Design: Architecture, Implementation, andOptimization. Wiley-IEEE Press, 2007.

[35] U. Meyer-Baese, Digital Signal Processing with Field ProgrammableGate Arrays, 3rd ed. Springer-Verlag New York, NJ, USA, 2007.

[36] Virtex-6 FPGA DSP48E1 Slice USer Guide (v1.3), Xilinx inc., 2011.[37] C.-J. Lin and C.-Y. Lee, “Implementation of a neuro-fuzzy network with

on-chip learning and its applications,” Expert Syst. Appl., vol. 38, no. 1,pp. 673–681, Jan. 2011.

[38] A. Gomperts, A. Ukil, and F. Zurfluh, “Development and implementationof parameterized fpga-based general purpose neural networks for onlineapplications,” IEEE Transactions on Industrial Informatics, vol. 7, no. 1,pp. 78–89, Feb 2011.

[39] M. Moussa, S. Areibi, and K. Nichols, On the Arithmetic Precisionfor Implementing Back-Propagation Networks on FPGA: A Case Study,A. R. Omondi and J. C. Rajapakse, Eds. Springer US, 2006.

[40] P. P. Chu, RTL Hardware Design Using VHDL: Coding for Efficiency,Portability, and Scalability. John Wiley & Sons, 2006.

[41] J. Oxer and H. Blemings, Practical Arduino: Cool Projects for OpenSource Hardware. Berkeley, CA, USA: Apress, 2009.

[42] Atmel, “Datasheet atmel sam3x8e arm cortex-m3 cpu,”http://www.atmel.com/Images/doc11057.pdf.

[43] U. of California Irvine, “Machine learning repository,”http://archive.ics.uci.edu/ml/.

[44] G. An, “The effects of adding noise during backpropagation trainingon a generalization performance,” Neural Comput., vol. 8, no. 3, pp.643–674, Apr. 1996.

Francsico Ortega-Zamorano (born 1980) receivedhis M.Sc. and Ph.D. degree in computer sciencefrom the University of of Málaga, Spain, in 2013and 2015, respectively. He joined the Department ofComputer Languages and Computer Science, Uni-versity of Málaga in 2011, where he is currently aPostdoctoral Researcher. His current research inter-ests include FPGA hardware implementation, neuro-computational real-time systems, smart sensor net-works and deep learning.

José M. Jerez received the Ph.D. degree in computerscience in 2003 from the University of Málaga,Málaga, Spain. He is currently an Associate Profes-sor with the University of Málaga. His research in-terests lie in the areas of computational intelligence,image analysis, and bioinformatics. In particular,he is developing prediction software for biomedicalproblems using artificial intelligence techniques incollaboration with the Málaga University hospital.He has participated in more than 15 research projectsfunded by international and national boards and he

belongs to several reviewing scientific committees.

Daniel Urda Muñoz was born in Málaga, Spainin 1982. He studied Computer Sciences at the Uni-versity of Málaga obtaining his degree in February2008. After completing a Master degree in SoftwareEngineering and Artificial Intelligence in June 2010,he began his PhD studies. Finally, he finished hisdoctorate in February 2015 at the University ofMálaga, mainly focused on developing predictivemodels based on genetic profiles to identify robustgenetic signatures with high prediction capabilitiesfor applications of Personalized Medicine. Currently,

he is working as a Marie Curie postdoctoral researcher at Pharmatics Ltd. inMachine Learning for Personalized Medicine.

Rafael M. Luque-Baena received the M.S. andPh.D. degrees in Computer Engineering from theUniversity of Málaga, Spain in 2007 and 2012,respectively. He moved to Mérida, Spain in 2013,and is currently a lecturer at the Department ofComputer Engineering in the Centro Universitariode Mérida, University of Extremadura. He alsokeeps pursuing research activities in collaborationwith other Universities. His current research interestsinclude visual surveillance, image/ video processing,neural networks and pattern recognition.

Leonardo Franco (M’06 SM’13) received theM.Sc. and Ph.D. degrees in 1995 and 2000, re-spectively, both from the National University ofCórdoba, Argentina. He was then a postdoctoralresearcher at SISSA, Italy and Oxford University,UK. Since 2005, he has been with the ComputerScience Department at Malaga University Spainwhere he is currently an Associate Professor. Hisresearch interests include neural networks, com-putational intelligence, biomedical applications andcomputational neuroscience. He has authored more

than 60 publications in journals and international conferences proceedings.



efﬁcient implementation of the backpropagation algorithm ...lfranco/a40-ortega+jerez++2015.pdf ·...

Documents