A Novel Nonlinear RBF Neural Network Ensemble Model forFinancial Time Series Forecasting

Donglin Wang and Yajie Li

Abstract— In this paper, a novel nonlinear Radial BasisFunction Neural Network (RBF-NN) ensemble model based on𝜈-Support Vector Machine (SVM) regression is presented forfinancial time series forecasting. In the process of ensemblemodeling, the first stage the initial data set is divided intodifferent training sets by used Bagging and Boosting technology.In the second stage, these training sets are input to the differentindividual RBF–NN models, and then various single RBF–NNpredictors are produced based on diversity principle. In thethird stage, the Partial Least Square (PLS) technology is usedto choosing the appropriate number of neural network ensemblemembers. In the final stage, 𝜈-Support Vector Machine (SVM)regression is used for ensemble of the RBF–NN to predictionpurpose. For testing purposes, this paper compare the new en-semble model’s performance with some existing neural networkensemble approaches in terms of two financial time series:S & P 500 and Nikkei 225. Experimental results reveal thatthe predictions using the proposed approach are consistentlybetter than those obtained using the other methods presentedin this study in terms of the same measurements. Those resultsshow that the proposed nonlinear ensemble technique providesa promising alternative to financial time series prediction.

I. INTRODUCTION

F INANCIAL time series modeling and forecasting isregarded as a rather challenging task because financial

time series are inherently noisy and nonstationary [1][2].Due to the high degrees of irregularity, dynamic mannerand nonlinearity, it is extremely difficult to capture theirregularity and nonlinearity hidden in financial time series bytraditional linear models such as multiple regression, expo-nential smoothing, autoregressive integrated moving average,etc. In recent years, Artificial Neural Network (ANN), whichemulate the parallel distributed processing of the humannervous system, have been successfully used for modelingfinancial time series [3][4][5]. Due to their powerful capa-bility and functionality, ANN provide an alternative approachfor many engineering problems that are difficult to solve byconventional approaches. An ANN with moderate numberof hidden layer(s) is capable of approximating any smoothfunction to any desired degree of accuracy without anyassumptions of traditional statistical approaches required. Inaddition, ANN are computationally robust, having the abilityto learn and generalize from examples to produce meaningfulsolutions to problems even when the input data contain errorsor are incomplete [6][7].

The application of an ANN, involves a complicated de-velopment process. As ANN approaches want of a rigorous

Donglin Wang and Yajie Li are with the Department of Mathematics, Bei-jing Vocational College of Electronic Science, No.1jia, 100024, ShaoyaojuStreet, Chaoyao District, Beijing (email: wangshuxue@sina.cn).

theoretical support, effects of applications strongly dependupon operators experience. Even for some simple problems,different structures of neural networks (e.g., different numberof hidden layers, different hidden nodes and different initialconditions) result in different patterns of network gener-alization. In the practical application, the results of manyexperiments have shown that the generalization of singleneural network is not unique. That is, ANN results are notstable [8].

In order to overcome the main limitations of ANN, re-cently a novel ensemble forecasting model, i.e. artificial neu-ral network ensemble (ANNE), has been developed [9][10].Because of combining multiple neural networks learned fromthe same training samples, ANNE can remarkably enhancethe forecasting ability and outperform any individual neuralnetwork. It is an effective approach to the development of ahigh performance forecasting system [11].

The important motivation of ensemble learning method isintegrated different neural network models based on the fun-damental assumption that one cannot identify the true processexactly. Meantime, some linear ensemble learning methodson the standard feed-forward neural network models [12][13]such as back-propagation neural networks (BPNN), are alsopresented. Different from the previous work, this paperproposes a novel nonlinear Radial Basis Function NeuralNetwork (RBF-NN) ensemble forecasting method in terms of𝜈-Support Vector Machine Regression (𝜈-SVMR) principle.

The rest of this paper is organized as follows: SectionII describes the building process of the nonlinear RBF–NNensemble forecasting model in detail. For further illustration,two real financial time series are used for testing in SectionIII. Finally, some concluding remarks are drawn in SectionIV.

II. THE BUILDING PROCESS OF THE NONLINEAR

ENSEMBLE MODEL

In this section, a triple-phase nonlinear RBF–NN ensemblemodel is proposed for financial time series forecasting. Firstof all, many individual RBF–NN predictors are generatedin terms of diversification. Then an appropriate numberof RBF–NN predictors are selected from the considerablenumber of candidate predictors by the Partial Least Square(PLS) technology. Finally, selected RBF–NN predictors arecombined into an aggregated neural predictor in terms of𝜈-Support Vector Machine Regression.

86

Third International Workshop on Advanced Computational Intelligence August 25-27, 2010 - Suzhou, Jiangsu, China

978-1-4244-6337-4/10/$26.00 @2010 IEEE

A. RBF Neural Network

The financial data is a complex nonlinear systems sothat it is difficult to model by using linear regressionmethodologies. Dissimilar to the regression, neural networksare nonlinear and their parameters are determined by somelearning techniques and searching algorithms such as errorback propagation and steep gradient algorithm. Radial basisfunction was introduced into the neural network literatureby Broomhead and Lowe [14]. The RBF–NN model is anetwork with local neurons which was motivated by thepresence of many local response neurons in human brain.Other motivation came from numerical mathematics. On thecontrary to the other type of NN used for nonlinear regressionlike back propagation feed forward networks, the RBF–NN learns quickly and has a more compact topology. TheGaussian RB–NN is found suitable not only in generalizinga global mapping but also in refining local features [15].

The RBF neural network is generally composed of threelayers: input layer, hidden layer and output layer. The maindifference from that of MLP is the absence of hidden-layer weights. The hidden neurons outputs are not calculatedby using the weighted-sum mechanism/sigmoid activation;rather each hidden neurons output 𝑑𝑖 is obtained by thecluster of the input 𝑋 to an m-dimensional parameter vector𝐶𝑖 associated with the ith hidden unit. The response outputof ith hidden neurons is calculated by

𝑑𝑖 = 𝑒𝑥𝑝{− 1

2𝜎2𝑖∥𝑋 − 𝐶𝑖∥2} (1)

where 𝐶𝑖 and 𝜎𝑖 is centers and width of RBF in ith hiddenneurons, respectively. The distance is calculated by theEuclidean norm

∥𝑋 − 𝐶𝑖∥ =

√√√⎷𝑘∑

𝑗=1

(𝑥𝑖 − 𝑐𝑖)2 (2)

Given an input vector 𝑋 , the output of the RBF network isgiven by

𝑡𝑘 =

𝑚∑𝑗=1

𝜔𝑖𝑑𝑖 + 𝜔0 (3)

where 𝜔0 is the bias, {𝜔𝑖, 𝑖 = 1, 2, ⋅ ⋅ ⋅ ,𝑚} are the weightparameter, 𝑚 is the number of nodes in the hidden layers ofthe RBF neural network. The sum of square errors (SSE) isgiven by

𝑆𝑆𝐸 =

𝑁∑𝑘=1

[𝑡𝑘 − 𝑦𝑘]2 (4)

where 𝑁 is the number of training cases, 𝑦𝑘 is desired outputand 𝑡𝑘 is the actual output.

The training procedure of the RBF networks is a complexprocess, this procedure requires the training of all parametersincluding the centers of the hidden layer units (𝑐𝑖, 𝑖 =1, 2, ⋅ ⋅ ⋅ ,𝑚), the widths (𝜎𝑖) of the corresponding Gaussianfunctions, and the weights (𝜔𝑖, 𝑖 = 0, 1, ⋅ ⋅ ⋅ ,𝑚) between thehidden layer and output layer. In this paper, the the orthog-onal least squares algorithm (OLS) is used to training RBF

based on the minimizing of SSE. The more detailed aboutalgorithm is described by the related literature [16] [17].

B. Generating individual RBF-NN predictors

With the work about bias–variance trade–off of Breti-man [18], an ensemble model regression model consistingof diverse models with much disagreement is more likely tohave a good generalization [19]. Therefore, how to generatediverse models is a crucial factor. For RBF–NN model,several methods have been investigated for the generation ofensemble members making different errors. Such methodsbasically depended on varying the parameters of RBF–NNor utilizing different training sets. In this paper, there arethree methods for generating diverse models.

(1) Using different RBF–NN architecture, such as thedifferent number of nodes in hidden layer, diverse RBF–NNwith much disagreement can be created.

(2) Utilizing different the parameters of RBF–NN, such asdifferent cluster center 𝑐 of the RBF–NN, : through varyingthe cluster center 𝑐 of the RBF–NN, different cluster radius𝜎 of the RBF–NN, different RBF–NN can be produced.

(3) Using different training data: by re-sampling andpreprocessing data, different training sets can be obtained.Typical methods include bagging [20], cross-validation (CV)[21], and boosting [22]. With these different training datasets,diverse RBF–NN models can be produced.

C. Selecting appropriate ensemble members

After training, each individual neural predictor has gener-ated its own result. However, if there are a great number ofindividual members, we need to select a subset of represen-tatives in order to improve ensemble efficiency. In this paper,the Partial Least Square (PLS) regression technique [23] isadopted to select appropriate ensemble members. Interestedreaders can be referred to [23] for more details.

D. 𝜈-Support vector regression

SVM was a significant result of machine learning researchin recent years, which has been introduced by Cortes andVapnik in 1995 [24]. Compared with the traditional neuralnetwork, the support vector machine not only simple instructure but also all sorts of technical performances arebetter than neural network obviously, which were testified bylots of experiments [25]. It was developed on the foundationof small samples statistical learning theory that proposedby Vapnik etc., and its algorithm is based on the structuralrisk minimization principle [26], which minimizes the upperbound of the generalization error which is bounded by boththe sum of the training error and a confidence interval termwhich depends on the Vapnik–Chervonekis dimension.

With the introduction of Vapnik’s 𝜀-insensitive loss func-tion, the SVM can be extended to regression problem, say,support vector regression (SVR), which is constructed byminimizing Vapnik’s 𝜀-insensitive loss function residuals be-tween the outputs of SVR and the target values and the modelcomplexity, and SVR is applied to various fields like optimalcontrol, time–series prediction, and image segmentation.

87

If the proper hyperparameters are picked up, SVR willgain good generalization performance and vice versa, so itis important to select right model. Instead of selecting anappropriate 𝜀 Schokopf et al [27] [28]. proposed a variant,called 𝜈-support vector regression, which introduces a newparameter 𝜈 which can control the number of support vectorsand training errors without defining 𝜀 a prior. To be moreprecise, they proved that 𝜈 is an upper bound on the fractionof margin errors and lower bound of the fraction of supportvectors [29].

Given a data set {𝑥𝑖, 𝑦𝑖, 𝑖 = 1, 2, ⋅ ⋅ ⋅ , 𝑁} randomlygenerated from an unknown function, where 𝑥𝑖 ∈ ℜ𝑛 is aninput variable and 𝑦𝑖 is the corresponding target value, Theregression problem of 𝜈-SVR can be described as follows:⎧⎨⎩

min 𝑔𝑝 = − 12w

Tw + 𝐶(𝜈𝜀+ 1𝑁

𝑁∑𝑖=1

(𝜉𝑖 + 𝜉∗𝑖 ))

𝑠.𝑡. 𝑑𝑖 − (wT𝜑(𝑥𝑖) + 𝑏) ≤ 𝜀+ 𝜉∗𝑖(wT𝜑(𝑥𝑖) + 𝑏)− 𝑑𝑖 ≤ 𝜀+ 𝜉𝑖𝜉𝑖, 𝜉

∗𝑖 ≥ 0, 𝑖 = 1, 2, ⋅ ⋅ ⋅ , 𝑁, 𝜀 ≥ 0.

(5)

where 0 ≤ 𝜈 ≤ 1, 𝐶 is the regulator, and training data 𝑥𝑖are mapped into a high (even infinite) dimensional featurespace by the mapping function 𝜑(⋅). This primal optimizationproblem is a linearly constrained quadratic programmingproblem, which can be solved by introducing Lagrangianmultipliers and applying Karush-Kuhn-Tucker (KKT) condi-tions to solve its dual problem. Therefore, we can get theregression function formula as follows:

𝑓(𝑥) =∑

𝑥𝑖∈𝑆𝑉

(��∗𝑖 − ��𝑖)𝐾(𝑥𝑖, 𝑥𝑗) + �� (6)

Where 𝐾(𝑥𝑖, 𝑥𝑗) = 𝜙𝑖 ⋅ 𝜙𝑗 is kernel function, it is generallytaken as 𝐾(𝑥𝑖, 𝑥𝑗) = exp(− ∥ 𝑥𝑖 − 𝑥𝑗 ∥2 /2𝜎2, whichis to meet the conditions of any symmetric kernel functioncorresponds to the feature space of the dot product, 𝛼∗𝑖 , 𝛼𝑖 arethe Lagrangian multipliers associated with the constraints.

E. The Establishment of Combination Forecasting Model

To summarize, the proposed nonlinear RBF neural networkensemble forecasting model consists of four main stages.Generally speaking, in the first stage, the initial data setis divided into different training sets by used Bagging andBoosting technology. In the second stage, these trainingsets are input to the different individual RBF–NN models,and then various single RBF–NN predictors are producedbased on diversity principle. In the third stage, PLS modelis used to select choose the appropriate number of neuralnetwork ensemble members. In the four stage, 𝜈-SVR isused to aggregate the selected ensemble members. In sucha way final ensemble forecasting results can be obtained.For verification and testing purposes, four typical foreignexchange rates are used as testing targets for empiricalanalysis. The basic flow diagram can be shown in Fig.1.

III. EMPIRICAL ANALYSIS

In this section, two main two main financial time seriesare used to test the proposed nonlinear ensemble forecasting

Training SetTR1

RBF1

Bagging and BoostingTechnology

Training SetTR2

Training Set TRm

PLS SelectEnsemble Members

v-SVR NonlinearRegression Ensemble

Ensemble ResultOutput

Training SetTR1,TR2,...TRm

Input Data

. . . . . .

RBF2

RBFm

Fig. 1. A Flow Diagram of The Proposed Ensemble Forecasting Model.

model. First of all, we describe the data and evaluationcriteria used in this study and then report the experimentalresults.

A. Data description and evaluation criteria

The stock market is a complex system, which is alsoaffected by political, economic and social factors. Besidesthat, people are not sure of which variables have a significantimpact on the system for the lack of understanding of itsinternal operating mechanism. At present, technical–standardof the stock market are often used as neural network inputvariables to predict the stock market and they have achievedcertain effects. The model proposed by this paper chooses sixvariables as the neural network input based on reference [30].The description of six variables are presented in Table 1.

TABLE I

SIX VARIABLES AND THEIR FORMULAS

Variables indicators Formulas

Prices(𝑥𝑡) 𝑥𝑡, (𝑡 = 1, 2, ⋅ ⋅ ⋅ , 𝑛)Moving Average(𝑀𝐴5)(𝑥1) 1

5

∑𝑡𝑖=𝑡−5+1 𝑥𝑖

Moving Variance(𝑀𝑉5)(𝑥2) 15

∑𝑡𝑖=𝑡−5+1(𝑥𝑖 − 𝑥𝑡)

Moving Variance Ratio(𝑀𝑉𝑅5)(𝑥3) 𝑀𝑉 2𝑡 /𝑀𝑉 2

𝑡−5

Disparity5(D5)(𝑥4) 𝑥𝑡/𝑀𝐴5

Disparity10(D10)(𝑥5) 𝑥𝑡/𝑀𝐴10

Price Oscillator(PO)(𝑥6) (𝑀𝐴5 −𝑀𝐴10)/𝑀𝐴5

The data set used for our experiment consists of two timeseries data: the S&P 500 index series, and the Nikkei 225index series. The data used in this study are obtained fromDatastream (http://www.datastream. com). The entire data setcovers the period from January 1, 2008 to December 31,2009. This paper established the predictive models based on421 daily data from January 1, 2008 to August 31, 2009as the training data sets, and test model based on 87 dailydata from September 1, 2009 to December 31, 2009 as thetesting data set, which are used to evaluate the good or badperformance of predictions.

In order to measure effectiveness of the proposed method,two typical indicators, normalized mean squared error(NMSE) and directional statistics (𝐷𝑠𝑡𝑎𝑡) were used in thispaper. Given 𝑁 pairs of the actual values (𝑦𝑡) and predicted

88

values (𝑦𝑡), the NMSE can be defined as

𝑁𝑀𝑆𝐸 =

𝑛∑𝑖=1

(𝑦𝑖 − 𝑦𝑖)2𝑛∑

𝑖=1

(𝑦𝑖 − 𝑦𝑖)2(7)

where 𝑦𝑖 is the average of actual value.Directional change statistics (𝐷𝑠𝑡𝑎𝑡) can be expressed as

𝐷𝑠𝑡𝑎𝑡 =1

𝑛

𝑁∑𝑡=1

𝐼𝑡 ⋅ 100% (8)

where 𝐼𝑡 = 1 if (𝑦𝑡+1 − 𝑦𝑡)(𝑦𝑡+1 − 𝑦𝑡) ≥ 0, and 𝑎𝑡 = 0otherwise.

In order to investigate the effect of the proposed model, thesimple averaging ensemble, the mean squared error (MSE)based regression ensemble and variance-based weighted en-semble are established. Those are fitted the 421 samplesand forecasted the 87 samples by the those models, thecomparison results are used to test the effect of predictivemodels.

B. Analysis of the Results

In this study, four main ensemble methods are imple-mented on four exchange rates data-sets for comparison. Thestandard RBF neural networks with Gaussian–type activationfunctions in hidden layer were trained for each training set,then tested as an ensemble for each method for the testing set.Each network was trained using the neural network toolboxprovided by Matlab software package.

TABLE II

THE NMSE COMPARISON WITH DIFFERENT MODELS FOR DIFFERENT

TIME SERIES ABOUT TRAINING SAMPLES

Ensemble S&P500 Nikkei 225

Method NMSE Rank NMSE Rank

Single RBF-NN 0.2389 5 0.1406 5Simple averaging 0.2274 4 0.0793 3MSE regression 0.2110 3 0.0932 4

Variance-based weight 0.1064 2 0.0757 2𝜈-SVMR 0.0670 1 0.0530 1

TABLE III

THE 𝐷𝑠𝑡𝑎𝑡 COMPARISON WITH DIFFERENT MODELS FOR DIFFERENT

TIME SERIES ABOUT TRAINING SAMPLES

Ensemble S&P500 Nikkei 225

Method 𝐷𝑠𝑡𝑎𝑡(%) Rank 𝐷𝑠𝑡𝑎𝑡(%) Rank

Single RBF-NN 89.68 2 82.75 5Simple averaging 79.65 5 87.54 4MSE regression 88.45 4 89.90 3

Variance-based weight 89.53 3 94.27 2𝜈-SVMR 93.75 1 95.93 1

Tables 2 and 3 show the fitting results of 421 trainingsamples for different models. In the two tables, a clear

comparison of various methods for the two financial timeseries is given via NMSE and 𝐷𝑠𝑡𝑎𝑡. From Tables 2 and3, the results show that 𝜈-SVM ensemble model better thanthose of the single RBF neural network model and otherensemble forecasting models for the two main financial timeseries in fitting.

The more important factor to measure performance of amethod is to check its forecasting ability of testing samplesin order. Tables 4 and 5 show the forecasting results of 87testing samples for different models about different measureindex. Generally speaking, the forecasting results obtainedfrom the two tables also indicate that the prediction perfor-mance of the proposed 𝜈-SVM ensemble forecasting modelis better than those of the single RBF–NN model and otherensemble forecasting models for the two main financial timeseries in forecasting.

TABLE IV

THE NMSE COMPARISON WITH DIFFERENT MODELS FOR DIFFERENT

TIME SERIES ABOUT TESTING SAMPLES

Ensemble S&P500 Nikkei 225

Method NMSE Rank NMSE Rank

Single RBF-NN 1.8527 4 1.9073 5Simple averaging 1.9708 5 1.6362 3MSE regression 1.5616 3 1.5873 2

Variance-based weight 0.3018 2 1.6967 4𝜈-SVMR 0.0230 1 0.0331 1

TABLE V

THE 𝐷𝑠𝑡𝑎𝑡 COMPARISON WITH DIFFERENT MODELS FOR DIFFERENT

TIME SERIES ABOUT TESTING SAMPLES

Ensemble S&P500 Nikkei 225

Method 𝐷𝑠𝑡𝑎𝑡(%) Rank 𝐷𝑠𝑡𝑎𝑡(%) Rank

Single RBF-NN 69.75 5 63.10 5Simple averaging 74.21 3 66.71 3MSE regression 70.89 4 76.09 2

Variance-based weight 78.98 2 65.28 4𝜈-SVMR 83.65 1 80.93 1

In detail, the NMSE of the our proposed 𝜈-SVMR ensem-ble model reaches 0.02303 in the testing samples for S &P500 time series, however the NMSE of the Single RBF-NNmodel is 1.8527; the NMSE of the Simple average model is1.9708; the NMSE of the MSE regression model is 1.5616and the NMSE of the Variance-based weight model is 1.5616;These results show the NMSE of the 𝜈-SVMR ensemblemodel is less than other model, which has obvious advantagesover other models for S & P500 time series forecasting.

Similarly, for 𝐷𝑠𝑡𝑎𝑡 efficiency index, the proposed 𝜈-SVMR ensemble model has higher valuer in the testingsamples for S & P500 time series forecasting. From Table5, the 𝐷𝑠𝑡𝑎𝑡 for the 𝜈-SVMR ensemble model reaches only83.65%, while for the Simple average model, the 𝐷𝑠𝑡𝑎𝑡 is69.75%; the 𝐷𝑠𝑡𝑎𝑡 of the MSE regression model is only

89

74.21%; the 𝐷𝑠𝑡𝑎𝑡 of the MSE regression model is 70.89%;the 𝐷𝑠𝑡𝑎𝑡 of the Variance-based weight model is 78.89%.Those show that the 𝜈-SVMR ensemble model is closeto their financial series data for S & P500 time seriesforecasting.

From Tables 4 and 5, the results show the ensemble modelperforms the best for Nikkei 225 time series data forecasting.we can conclude that

(1) in all the ensemble methods the 𝜈-SVMR based ensem-ble model performs the best, followed by the neural networkbased ensemble method and other three linear ensemblemethod from a general view and;

(2)the nonlinear ensemble methods including RBF–NN–based and 𝜈-SVMR-based method outperform all the linearensemble methods, indicating that the nonlinear ensemblemethods are more suitable for financial time series forecast-ing than the linear ensemble approaches due to high volatilityof the financial time series.

IV. CONCLUSIONS

Stock market faces a complex external environment ofrapid change. Because of the factors of uncertainty in fore-casting increasing, forecasting from single prediction modelcan not achieve the ideal effects from purely linear or nonlin-ear models due to the non–extensive information sources andnon–sensitivity to the model set. In this study, we proposea novel nonlinear RBF neural network ensemble predictorbased on 𝜈-SVMR for financial time series forecasting. Ex-amples of calculation shows that the method can significantlyimprove the system’s predictive ability, prediction accuracy,and with a high prediction accuracy of the rising and fallingtrend of the stock market. Empirical results obtained revealthat the proposed nonlinear combination technique is a verypromising approach to financial time series forecasting.

REFERENCES

[1] E. H. Francis and L. J. Chao, “ Modified support vector machine infinaancial time series forecasting,” Neurocomputing, vol. 48, pp. 847-861, 2002.

[2] L. J. Cao and E. H. Francis, “Support vector machine with adaptiveparameters in financial time series forecasting,” IEEE Transactions onNeural Networks, vol. 14, no. 6, pp. 15063-1518, 2003.

[3] K. A. Smith and N. D. Gupat Jatinder, “Neural Networks in business:techniques and applications for the operations researcher,” Computers& Operations Research, vol. 27, pp. 1023-1044, 2000.

[4] G. P. Zhang and V. L. Berardi, “ Time series forecasting with neural net-work ensembles: an application for exchange rate prediction,” Journalof the Operational Research Society, vol. 52, pp. 652-664, 2001.

[5] K. K. Lai, L. Yu, W. Huang and S. Y. Wang, “A novel support vectormachine metamodel for business risk identification,” In: Q. Yang andG. Webb, (eds.) PRICAI 2006. Lecture Note in Computer Science, vol.4099, pp. 980-984, Springer, Heidelberg, 2006.

[6] L. Yu, S. Y. Wang and K. K. Lai, “A novel nonlinear ensembleforecasting model incorporating GLAR and ANN for foreign exchangerates,” Computers & Operations Research, vol. 32, pp. 2523-2541,2005.

[7] A. Jain and A. M. Kumar, “Hybird neural network models for hy-drologic time series foreasting,” Applied Soft Computing, vol. 7, pp.585-592, 2007.

[8] Jiansheng Wu, “A novel nonparametric regression ensemble for rainfallforecasting using particle swarm optimization technique coupled withartificial neural network,” Lecture Note in Computer Science, vol. 5553,no. 3, pp. 49-58, Springer-Verlag Berlin Heidelberg, 2009.

[9] P, Sollich and A. Krogh, “Learning with Ensembles: How Over-fittingcan be useful,” In: Touretzky D, Mozer M, Hasselmo,M.(eds). Advancesin Neural Information Processing Systems 8, Cambridge, MA: MITPress, pp. 190-196, 1996.

[10] J. Mao, “A case study on bagging boosting and basic ensemblesof neural networks for OCR,” In Processing of International JointConference on Neural Networks-98, Anchorage, AK, vol. 3, pp. 1828-1833, 1998.

[11] Z. Zhou, J. Wu and W. Tang, “Ensembling neural networks: Manycould be better than all,” Artificial Intelligence, vol.137, no.1-2, pp.239-263, 2002.

[12] J. Carney and P. Cunningham,, “Tuning diversity in bagged ensem-bles,” International Journal of Neural Systems, vol. 10, pp. 267-280,2000.

[13] B. E. Rosen, “Ensemble learning using decorrelated neural networks,”Connection Science, vol. 8, pp. 373-384, 1996.

[14] D. S. Broomhead and D. Lowe, “Multivariable functional interpolationand adaptive networks,” Complex Systems, vol. 2, pp. 321-355, 1988.

[15] P. K. Dash, A. K. Pradhan and G. Panda,, “Application of minimalradial basis function neural network to distance protection,” IEEETransactions on Power Delivery, vol. 16, no. 1, pp. 68-74, 2001.

[16] Z. Moravej, D. N. Vishwakarma and S. P. Singh, “Application ofradial basis function neural network for differential relaying of a powertransformer,” Computers and Electrical Engineering, vol. 29, pp. 421-434, 2003.

[17] A.M.S. Barreto, H. J. C. Barbosa and N. F. F. Ebecken, “GOLS–Genetic orthogonal least squares algorithm for training RBF networks,”Neurocomputing, vol. 69, pp. 2041-2064, 2006.

[18] L. Breiman, “Combining Predictors,” Proceedings of Combining Artifi-cial Neural Nets-Ensemble and Modular Multi-net Systems, the SpringerPress, Berlin, vol. 1, pp. 31-50, 1999.

[19] J. A. Benediktsson, J. R. Sveinsson, O. K. Ersoy and P. H. Swain,“Parallel Consensual Neural Neural Networks,” IEEE Transactions onNeural Networks, vol. 8, pp. 54-64, 1997.

[20] L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, pp. 123-140, 1996.

[21] A. Krogh and J. Vedelsby, “Neural network ensembles, cross valida-tion, and active learning,” G. Tesauro, D. Touretzky and T. Leen (Eds.),Advances in Neural Information Processing Systems, vol. 7, pp. 231-238, MIT Press, Cambridge, MA, 1995.

[22] R. E. Schapire, “The strength of weak learnability,” Machine Learning,vol. 5, pp. 197-227,1990.

[23] D. M. Pirouz. “An overview of partial least square,” Technical report,The Paul Merage School of Business, University of California, Irvine,2006

[24] C. Cortes and V. Vapnik, “Support vector networks,” Machine Learn-ing, vol.20, pp.273-295, 1995.

[25] B. Holkopf, K. Sung and C. J. C. Burges, “ Comparing support vectormachine with Gaussian kernel to radial basis function classifiers,” IEEETranslated Signal Processing, vol.45, pp.2758-2765, 2001.

[26] P. L. Bartlett and J. S. Taylor, “Generalization performance on supportvector machines and pattern classifiers,” B. Sholkopf, C. Burges, andA. Smola Eds., Advances in Kernel Methods-Support Vector Learn-ing,Cambridge, MA: MIT Press, 1999.

[27] B. Schokopf and A. J. Smola, “Generalization performance on supportvector machines and pattern classifiers,” B. Sholkopf, C. Burges, andA. Smola Eds., Advances in Kernel Methods-Support Vector Learn-ing,Cambridge, MA: MIT Press, 1999.

[28] C. C. Chang and C. J. Taylor, Learning with kernels,Cambridge, MA:MIT Press, 2002.

[29] P. L. Bartlett and J. S. Lin, “Training 𝜈-support vector regression:theory and algorithms,” Neural Computation, vol.14, 1959-1977, 2002.

[30] K. J. Kim, “Financial time series forecasting using support vectormachines,” Neurocomputing, vol. 55, pp. 307-319, 2003.

90