measurement of fitness function efficiency using data envelopment analysis

1

3

4

5

6

7 Q1

89

1011

1 3

1415161718192021222324

2 5

43

44

45

46

47 Q3

48

49

50

51

52

53

54

55

56

57

58

59

Q2Q1

Expert Systems with Applications xxx (2014) xxx–xxx

ESWA 9363 No. of Pages 14, Model 5G

17 June 2014

Q1

Contents lists available at ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier .com/locate /eswa

Measurement of Fitness Function efficiency using Data EnvelopmentAnalysis

http://dx.doi.org/10.1016/j.eswa.2014.06.0010957-4174/� 2014 Published by Elsevier Ltd.

⇑ Corresponding author. Tel.: +55 8133206490.E-mail addresses: [email protected] (D.A. Silva), gabbybel@hotmail.

com (G.I. Alves), [email protected] (P.S.G. de Mattos Neto), [email protected] (T.A.E. Ferreira).

URL: http://www.ppgia.ufrpe.br/tiago (T.A.E. Ferreira).

Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Function efficiency using Data Envelopment Analysis. Expert Systems witcations (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

David A. Silva a, Gabriela I. Alves a, Paulo S.G. de Mattos Neto b, Tiago A.E. Ferreira a,⇑a Department of Statistics and Informatics, Federal Rural University of Pernambuco, Recife, Pernambuco, Brazilb Department of Computing, University of Pernambuco, Garanhuns, Pernambuco, Brazil

262728293031323334

a r t i c l e i n f o

Keywords:Efficiency measureFitness FunctionData Envelopment AnalysisTime series forecastingArtificial Neural NetworksEvolutionary StrategyHybrid Intelligent SystemsOptimization

35363738394041

a b s t r a c t

Over the last years, Evolutionary Algorithms (EAs) have been proposed aiming to find the best configu-ration of the Artificial Neural Networks (ANN) parameters. Among several parameters of an EA thatcan influence the quality of the found solution, the choice of the Fitness Function is the most importantfor its effectiveness and efficiency, given that different Fitness Functions have distinct fitness landscapes.In other words, the Fitness Function guides the evolutionary process of the candidate solutions accordingwith a given criterion of the performance. However, there is not an universal criterion to identify the bestperformance measure. Thus, what is the Fitness Function more efficient among a set of several possibleoptions? This paper presents a methodology based on Data Envelopment Analysis (DEA) to find the moreefficient Fitness Function among candidates. The DEA is used to determine the best combination ofstatistical measures to build the more efficient Fitness Function for a EA. The case study employed hereconsists of a hybrid system composed by Evolutionary Strategy and ANN applied to solve the time seriesforecasting problem. The data analyzed are composed by financial, agribusiness and natural phenomena.The results show that establishment of the Fitness Function is a crucial point in the EA design, being a keyfactor to obtain the best solution for a limited number of EA’s iteration.

� 2014 Published by Elsevier Ltd.

42

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

1. Introduction

In literature, Artificial Neural Networks (ANN) has been widelyused in different tasks, such as: function approximation (Petkovic,Cojbašic, & Lukic, 2013), classification (Fernandes, Cavalcanti, &Ren, 2013), pattern recognition (Ma, Chan, Saha, & Ekanayake,2013), time series forecasting (Ferreira, Vasconcelos, & Adeodato,2008; da S. Gomes & Ludermir, 2013), among them. Regardless ofthe application, the most important step after the choice of theANN type, is the adjustment of the its parameters, as the networktopology, number of layers, number of neurons per layer, activa-tion function, etc. The determination of the optimal values of theseparameters generally is a huge task. In this sense, EvolutionaryAlgorithms (EAs) (Eiben & Smith, 2003) has been used to adjustANN parameters (Donate, Li, Sánchez, & de Miguel, 2013; Ferreiraet al., 2008; Rodrigues, de Mattos Neto, & Ferreira, 2009, Rodrigueset al., 2010). The EAs are heuristic optimization algorithms,

77

78

79

80

81

82

inspired by biological evolution, used commonly to find optimalconfiguration of a specific system. The combination EA with ANNis very popular in the literature (Belfore & Arkadan, 1997;Bhuiyan, 2009; Ferreira et al., 2008; Gonzalez, Donate, Cortez,Sanchez, & de Miguel, 2012; Grzesiak, Meganck, Sobolewski, &Ufnalski, 2007; Guo, Kang, Liu, Sun, & Mei, 2007; Liao, 2012;Lima, Cannon, & Hsieh, 2012; Tomczak, 2011; Sotiroudis, Goudos,Gotsis, Siakavara, & Sahalos, 2013), where this combination is com-monly called of intelligent hybrid systems.

In general, for any EA (Eiben & Smith, 2003), trial solutions arerepresented by individuals of a population, where each one ofthese individuals has a chance of being selected to generate thenext offspring. This procedure is repeated until an optimal solutionis found. In each iteration, the recombination and mutation opera-tors form the basis to create new offspring aiming to preserve thediversity in the population.

Despite the importance of these operators, the selection andsurvival of each individual at each generation is guided by FitnessFunction. This Fitness Function is used to measure the quality ofthe individuals. Different Fitness Functions can lead to differentsolutions, whereas each Fitness Function have its own fitness land-scape (Kitts, Edvinsson, & Beding, 2001; Merz, 2004), that exertsstrong influence in the effectiveness of the evolutionary search.

h Appli-

http://dx.doi.org/10.1016/j.eswa.2014.06.001

mailto:[email protected]






http://www.ppgia.ufrpe.br/tiago


http://www.sciencedirect.com/science/journal/09574174

http://www.elsevier.com/locate/eswa


83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139140142142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

2 D.A. SilvaQ1 et al. / Expert Systems with Applications xxx (2014) xxx–xxx


17 June 2014

Q1

For a intelligent hybrid system (here, a EA combined with ANN),the fitness landscape will not depend of the Evolutionary Algo-rithm used to evolve the ANN. The fitness landscape will dependof the ANN’s structure and the data of the problem studied. How-ever, in general, the performance of the EA will be dependent of thefitness landscape.

In literature, some works investigated the Fitness Functions(Chen, Yang, Dong, & Abraham, 2005; Fan, Fox, Pathak, & Wu,2004; Ferreira et al., 2008; Pai & Hong, 2005; Rodrigues et al.,2009; Rodrigues, Silva, de Mattos Neto, & Ferreira, 2010), as refer-ences to understand the performance of the hybrid systems(EA + ANN), where different fitness functions guide the EA to dif-ferent solution. However, how is possible to determine the FitnessFunction most efficient to solve a given problem? Wang et al.(2011) proposed a systematic approach of constructing fitnessfunction, where the multi-objective Fitness Function is built fromthe combination of the simple fitness functions, but with theWang’s work is not possible measure the relative efficiency of dif-ferent fitness functions.

In this paper, a methodology based on Data Envelopment Anal-ysis (DEA) (Charnes et al., 1994) is proposed to analyze the FitnessFunction efficiency. The DEA is a non-parametric methodology thatconstructs an efficiency frontier with the best units. Its applica-tions involve many topics as banking (Luo, Bi, & Liang, 2012;Shyu & Chiang, 2012), mining (Touloo, Sohrabi, & Nalchigar,2009), industry (Sarkis & Cordeiro, 2012), neural networks(Desheng, 2009; Makui & Noushabadi, 2012). The motivation forusing this approach in this work is due few studies involvingDEA to analyze the efficiency of the Fitness Function, especiallyapplied to time series forecasting problem. The junction of thisstudy is interesting for both researchers investigating the DEAand for research in the area of Evolutionary Algorithms and timeseries. In this work, the DEA is used to find the Fitness Functionthat increases the performance of the EA. To check the efficiencyof the best fitness functions, graphs comparing the predicted valueto the original data are used.

Therefore, this article proposes a methodology to guide the pro-ject of an expert system based on Evolutionary Algorithms to solvereal world problems, in particular to solve the time series forecastingproblem. For Evolutionary Algorithms the Fitness Function is a fun-damental point to be defined. The Fitness Function will guide theEvolutionary Algorithm to reach a good (maybe, an optimal) solu-tion. Thus, the correct choice of the Fitness Function is fundamentalto guarantee the good performance of the Evolutionary Algorithm.

In the next Section is introduced the time series forecastingproblem that is the case study addressed in this paper. In Section 3is presented the DEA concepts, the determination of the DEA modeland the definition of its variables. In Section 4 is presented themethodology to be used in this article. The experimental resultsare presented in Section 5 and the conclusions and future worksare presented in Section 6.

194

195

196197

199199

200201

203203

204

2. Time series forecasting problem

A time series is a sequence of observations chronologicallyordered about a given phenomenon. This series can be composedby discrete or continuous data. In general practical terms, an obser-vation of a phenomenon results in a discrete sampling of data. Adiscrete time series is represented as a set

Zt ¼ fzt 2 R j t ¼ 1;2;3; . . . ;Ng ð1Þ

where t is an chronological index, commonly the time, and N thetotal number of observations.

The study of the time series behavior has evolved considerablyover the years. The use of more advanced computational resources

Please cite this article in press as: Silva, D. A., et al. Measurement of Fitness Funccations (2014), http://dx.doi.org/10.1016/j.eswa.2014.06.001

combined with statistical techniques resulted in more accurateprediction, leading to lower cost in time to generate the forecast-ing. However, despite all these features, the ability to perform agood prediction will depend basically on the methodology usedand the complexity of the phenomenon studied by the researcher.Thus, reducing the prediction error as much as possible for bestresults have been faced with the problem studied.

The choice of model or technique to forecast, among many fac-tors, depends on the level of precision that is required, desiredforecast horizon, type of data used and the cost to produce theforecasts (Abraham & Ledolter, 2009). Alternatives approaches tostatistical models (Box & Jenkins, 1994) for time series analysisand forecasting problem have been developed based on ArtificialIntelligence techniques, like ANN (Areekul, Senjyu, Toyama, &Yona, 2010; Ferreira et al., 2008; Haykin, 1998; Mandal, Senjyu,Urasaki, Funabashi, & Srivastava, 2007; Yan, 2012) and EA(Amjady & Keynia, 2009; Eiben & Smith, 2003; Hinojosa & Hoese,2010). Among those techniques, the hybrid systems based on thecombination of ANNs with EAs have been used with success reach-ing relevant results (Donate et al., 2013; da S. Gomes & Ludermir,2013; Ferreira et al., 2008; Rodrigues et al., 2009; Rodrigueset al., 2010; Stepnicka, Cortez, Donate, & Stepnicka, 2013).

3. Data Envelopment Analysis

Data Envelopment Analysis (DEA) (Charnes et al., 1994) is amethodology originally used in the area of operational research,where the interest is to compare different and independent units(firms, departments, etc.) in relation to their productive efficiency.In DEA, these units are represented by a variable called DMUs(Decision Make Unit). Each DMU is composed of a number ofinputs required for a given amount of products. The main idea iscompare these units to obtain the best combination of inputsand outputs aiming to better production efficiency.

The DEA methodology was based on the concepts of relativeefficiency proposed by Farrell in 1957 (Farrell, 1957). AccordingFarrell, a company is considered efficient when it is able to producea large quantity of products given a mix of resources. The ineffi-ciency would be obtained when this company failed to get themost of your products from a number of resources.

The formal concept about the DEA methodology was introducedby Charnes, Cooper and Rhodes in 1978 (Charnes et al., 1978),where to each DMU the DEA determines the maximum ratiobetween inputs and outputs, weighted by real factors determinedby the model.

3.1. DEA models

3.1.1. The CCR modelThe CCR model (Charnes, Cooper, Rhodes) was the first pro-

posed DEA model (Charnes et al., 1978) and assumes constantreturns to scale. The solution of this model is given by the linearprogramming problem (multiplier form) below (Cooper, Seiford,& Tone, 2007),

max wk ¼Xs

j¼1

ujyjk ð2Þ

subject to

Xr

i¼1

v ixik ¼ 1 ð3Þ

Xs

j¼1

ujyjk �Xr

i¼1

v ixik � 0 ð4Þ

where

tion efficiency using Data Envelopment Analysis. Expert Systems with Appli-


205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221222224224

225226

228228

229

230

231

232

233

234

235

236

237

238

239

240241

243243

244

245

246

247

248

249

250251

253253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268269

271271

272273

275275

276

277

278

279

280281283283

284285

287287

288

289

290

291

292

293

294

D.A. SilvaQ1 et al. / Expert Systems with Applications xxx (2014) xxx–xxx 3


17 June 2014

Q1

� i represents the inputs, where r is the maximum number ofinputs (i ¼ 1; . . . ; r)� j represents the outputs, where s is the maximum number of

outputs (j ¼ 1; . . . ; s)� wk – relative efficiency for kth DMU.� xik; yjk – amount for input i, output j respectively for kth DMU.� v i;uj – weights (P 0) for input i, output j respectively.

The DMU is considered efficient, if the optimal objective valuew�1 is equal to 1 in the Section 3.4 and there exists at least one opti-mal solution v�;u� in Eqs. (3) and (4), with v� > 0;u� > 0. Otherwise,the DMU is considered inefficient (Cooper et al., 2007). For inefficientunits, the DEA model identifies a set of efficient DMUs, called the setof reference or peer group, taken as benchmarks for the projection ofinefficient units onto efficient frontier. The identification of this setof reference is better viewed through the dual model (envelopmentform) shown below

min zk ð5Þ

subject to

Xn

k¼1

kkxik 6 zkxjk ð6Þ

Xn

k¼1

kkyjk P yjk ð7Þ

kk P 0;8k

where kk 2 R is the intensity of unit k. The optimal objective valuez� in Eq. (5) is equivalent to the optimal value w� in Eq. (2) due tothe duality relation of these models (Cooper et al., 2007). Whilethe weights (v�; u�) are important variables to the multipliers formin Eqs. (3) and (4), the optimal solution (k�; z�) has its relativeimportance to the form enveloped in (6) and (7). It is possible toobtain, from the values of lambda, the points (xo; yo) for which theinefficient units are projected onto the efficient frontier (definedby all efficient units). In this process of projection, inefficient unitsdo not quite fit the surface causing slacks which are related toinputs excesses (s�) or outputs shortfalls (sþ). These slacks aredefined in the equations below

s� ¼ zoxjo �Xn

k¼1

kkxik ð8Þ

sþ ¼Xn

k¼1

kkyjk � yjo ð9Þ

s� 2 Ri; sþ 2 Rr

The ideal goal is to obtain the optimal objective value z� ¼ 1 and allslacks equal to zero (s�� ¼ 0; sþ� ¼ 0) for a unit to be considered effi-cient, known as Pareto Efficiency (Cooper et al., 2007). Otherwise,the unit is considered inefficient (Cooper et al., 2007; Thanassoulis,2001). From Eqs. (8) and (9), it is possible to obtain the coordinatesof the points that represents the direction for the improvement ofinefficient units. The coordinate points (xo; yo) is obtained by

xo ¼ z�xo � s�� ð10Þ

yo ¼ yo þ sþ� ð11Þ

Eqs. (10) and (11) suggest that the efficiency for a given unit can beimproved if the values of the inputs are reduced radially by the opti-mal objective value (z�) and the excesses of inputs (s�� ) are elimi-nated. In the same way, the efficiency can be attained if theoutput values are augmented by the output shortfalls (sþ� )

1 The superscript � is used to denote the optimal value of a variable (Thanassoulis,2001).


(Cooper et al., 2007). Since the inefficient unit is below of the effi-ciency frontier, the comparison between the targets and evaluatedunits allows to obtain the percentage of improvement potentialfor reducing inputs and increasing products for the evaluate unitto become efficient (Coll & Blasco, 2006).

3.1.2. The BCC modelThe BCC model (Banker, Charnes, & Cooper, 1984) assumes var-

iable returns of scale (VRS), where the variations of inputs and out-puts are not proportional. The efficiency of the kth DMU hk isdefined by (multipliers form) (Cooper et al., 2007),

max hk ¼Xs

j¼1

ujyjk þ jk ð12Þ

subject to

Xr

i

v ixik ¼ 1 ð13Þ

Xs

j

ujyjk �Xr

i

v ixik þ jk � 0 ð14Þ

v i;uj P 0; k 2 R

where the variable j indicates the return of scale.Due to the relation between the CCR model and BCC model, the

considerations made in the previous subsection are also empha-sized for the BCC model. The dual model (form enveloped) is givenby

min bk ð15Þ

subject to

Xn

k¼1

kkxik 6 bkxjk ð16Þ

Xn

k¼1

kkyjk P yjk ð17Þ

Xn

k¼1

kk ¼ 1 ð18Þ

kk P 0;8k

where Eq. (18) is a restriction of convexity related by the variable j.The value taken by this restriction indicates whether the firm(DMU) is operating in an area of decreasing (< 1), constant (¼ 1)or increasing (> 1) returns to scale. If the constraint is equal to 1,the BCC model is known as the CCR model.

Fig. 1 shows the possible operating regions to the productionpossibility set. The DMUs P under the efficiency frontier is consid-

Fig. 1. Efficiency frontier in a DEA model.



295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310311

313313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384385387387

388

389

390

391

392393

395395

396

397

398

399400

402402

403

404

405

406

407

408

409

410

411412

414414

415

416

417



17 June 2014

Q1

ered inefficient. The way that this unit is projected onto efficientfrontier determines the orientation of the model. If ‘‘input ori-ented’’, the DMU is projected onto the efficient frontier, reducingthe input and maintaining constant the output (projection of P toB or C). If ‘‘output oriented’’, the input is constant and the outputis variable (projection of P to O or F).

Under the assumption of variable returns to scale input ori-ented, the fraction xp=xb (see Fig. 1) is the pure technical efficiency(PTE) of ‘‘P’’. The unit ‘‘E’’ (xe; ye) has the largest average productiv-ity within the production possibility set and represents an aggre-gate technically and scale efficient (know as overall efficiency(OE)) unit for the input/output mix (X;Y) (Boussofiane, Dyson, &Thanassoulis, 1991). The measure of the overall efficiency (OE) of‘‘P’’ in comparison to unit ‘‘C’’ is the ratio DC=DP. This measurecan be decomposed into pure technical efficiency (ratio DB=DP)and scale efficiency (SE) (ratio DC=DB) by the relation

OE ¼ ðPTEÞðSEÞ ¼ DBDP

DCDB

ð19Þ

This relation indicates that the efficiency of a model with returnsconstant to scale (overall efficiency) can be decomposed into amodel with variable returns to scale (pure technical efficiency)and its scale efficiency. If scale efficiency (SE) is equal to one, theefficient unit following the BCC model has characteristics of theCCR model and operates in the most productive scale size (Cooperet al., 2007). The scale inefficiency occurs when SE < 1 and it iscaused by the distance between DC and DB (see Fig. 1). So, whenSE < 1 is necessary to verify whether this was due to unit operatingunder increasing returns to scale or decreasing returns to scale. Inorder to determine this, one must calculate an additional efficiencymeasure (DRS) that follows the decreasing return to scale and thendetermine the following relation (Cooper et al., 2007; Fare,Grosskopf, & Lovell, 1994)

1. If OE ¼ PTE, returns to scale is constant.2. If OE < PTE and OE ¼ DRS, scale inefficiency is due to increasing

returns to scale.3. If OE < PTE and OE < DRS, scale inefficiency is due to decreasing

returns to scale.

The determination of slacks and benchmarks for the improve-ment of inefficient units follows the same model presented inEqs. (8)–(11). An unit will be efficient if it obtains optimal objectivevalue b� ¼ 1 and all slacks equal to zero (s�� ¼ 0; sþ� ¼ 0) (Cooperet al., 2007; Thanassoulis, 2001), following the same path as theCCR model.

3.2. Definition and Selection of DMUs

A group of DMUs is considered homogeneous if the analyzedunits perform the same objectives under the same market condi-tions and the inputs and outputs that characterize the performanceof all units of the group are the same differing only in intensity ormagnitude (Golany & Roll, 1989).

Typically, in an Evolutionary Algorithm, the selection process ofcandidate solution (individual selection) is guided by the FitnessFunction. This Fitness Function ranks the individuals in the popu-lation of the Evolutionary Algorithm, where the best individual(best solution) will have the bigger value of Fitness Function andthe worse individual will have the smaller value of fitness.

Therefore, the Fitness Function is a key element in the project ofEvolutionary Algorithm, where how the Fitness Function ranks theindividual will influence directly in the algorithm performance.Thus, the correct choice of this Fitness Function will influencethe performance of evolution to guide individuals to an optimalsolution. Therefore, the Fitness Function will be considered in this


paper as a variable of decision in the analysis of efficiencies in DEA,i.e., each Fitness Functions analyzed will be a DMU in the DEAmethodology.

3.3. Determination of inputs and outputs factors

The input and output factors are chosen based on the impor-tance of these variables to the performance of DMUs in the DEAmodel (Kittelsen, 1993). This step is very important because theresults of the model are highly influenced by the input and outputchoice. The procedures for selection of these factors can be madethrough the critical judgment of the researcher or by statisticalanalysis (Kittelsen, 1993).

In this work, the time series forecasting problem employ as nat-ural measure of performance the prediction error (Rodrigues et al.,2009). The performance measures are based on forecast error, butthere is not consensus in the research area about which error mea-sure provides the best results. Some works in the literature usingperformance measures based on different types of statistical errors(Armstrong & Collopy, 1992; Rodrigues et al., 2009; Rodrigueset al., 2010) have shown that the prediction accuracy reached bya evolutionary predictive model, when each performance measureis applied, is dependent on the Time Series (and its features). Theseworks also shown that if the performance measure uses more thanone statistical error simultaneously then there is a couplingbetween the statistical errors (Rodrigues et al., 2009; Rodrigueset al., 2010).

The forecast error et is formulated as the difference between thecurrent value of the series (Zt) and the predicted value (Ot)

et ¼ Zt � Otð Þ ð20Þ

From Eq. (20) is possible define several performance measures.

3.3.1. MSE (Mean Square Error)The mean square error (MSE) is the performance measure most

commonly used in the literature for time series forecasting. Itsequation is given by

MSE ¼ 1N

XN

t¼1

etð Þ2 ð21Þ

where N indicates the number of points in the time series.

3.3.2. MAPE (Mean Absolute Percent Error)The mean absolute percentage error (MAPE) is expressed by the

following formula:

MAPE ¼ 1N

XN

t¼1

et

Zt

�� ð22Þ

where Zt is the current observation of the time series at time t and Nthe number of points.

3.3.3. U of Theil statisticsThis metric is based on the U-statistic developed by Theil et al.

(1966). The U-statistic is an accuracy measure that emphasizes therelevancy base for comparison, where this comparison is com-monly done with a naive forecasting method. Here, the MSE errorof the predictive model is compared with the MSE error of a ran-dom walk model like. This metric is done by the equation:

THEIL ¼PN

t¼1 Zj � Oj� �2

PNt¼1 Zj � Zjþ1� �2 ð23Þ

If THEIL ¼ 1, the predictive model has a performance equals to arandom walk. If THEIL > 1, a performance of the predictive modelis worse than a random walk. And if THEIL < 1, the predictive model



418

419

420

421

422

423

424

425

426427

429429

430

431

432

433

434

435

436

437

438

439440

442442

443444

446446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465466468468

469

470

471

472

473

474

475476

478478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540



17 June 2014

Q1

has a performance better than a random walk model like. In theideal case, this metric will go to zero (THEIL ¼ 0).

3.3.4. ARV (Average Relative Variance)The performance measure Average Relative Variance (ARV)

(Nowlan & Hinton, 1992; Shadbolt & Taylor, 2002), like the U-sta-tistic, is also a comparative metric. In the ARV metrics the MSEerror is compared with the quadratic deviation of the predictionwith respect to the mean of the time series.

The ARV performance measure is given by

ARV ¼PN

t¼1 Oj � Zj� �2

PNt¼1 Zj � Z� �2 ð24Þ

where Z represents the mean of the time series data. If ARV ¼ 1, themodel is equal to predict the mean of the time series. If ARV > 1, thepredictive model is worse than predict the mean of the time series,and if ARV < 1, the model is better than predict the mean of the set.In the ideal case, this metric will go to zero (ARV ¼ 0).

3.3.5. POCID (Prediction of Change in Direction)The Prediction of Change in Direction (Rodrigues et al., 2009), or

POCID for short, is a metric that measure the local trend of the pre-dictive model.

The POCID measure is given by the equation:

POCID ¼ 100PN

t¼1Dt

Nð25Þ

where

Dj ¼1; if Zj � Zj�1

� �Oj � Oj�1� �

> 00; otherwise

(

For the perfect predictive model, the POCID is equal to 100%.

3.4. Definition of the DEA model

Before proceeding with the application of the DEA model, it isnecessary to decide what kind of model will be used for each timeseries: the CCR model or the BCC model. One factor used to differ-entiate them is the return of scale, as described in Section 3.1.2.Simar and Wilson (2002) believes that the wrong choice of this fac-tor may have economic implications because if some DMU doesnot exhibit constant returns to scale then some unit of productionmay have obtained larger or smaller weights. Also, if a particularmodel is assumed to be ‘‘constant returns to scale’’ when in factit is ‘‘variable returns to scale’’, this mistake can cause loss of sta-tistical efficiency. In an attempt to avoid this mistake, Simar andWilson (1998, 2000) proposed a bootstrap procedure to test thehypothesis of scale return. Bogetoft and Otto (2010) explain thisbootstrap procedure in their book clearly.

Consider the set of combinations of inputs and outputs, wherethe input can produce the output. This combination is called aset of technology T and represented by

T ¼ x; yð Þ j x can produce yf g ð26Þ

where x is the input and y is the output.The interest is to test whether the technology T exhibits con-

stant return to scale (null hypothesis) against the alternativehypothesis of variable returns to scale. The statistic test for thishypothesis is formulated based on the concept of efficiency inscale, where for a given set of K firms observations has the follow-ing test statistic S

S ¼PK

k¼1EkCRSPK

k¼1EkVRS

ð27Þ


where ECRS and EVRS are respectively the efficiencies of scale of amodel with constant returns to scale and variable returns to scale.

The null hypothesis is rejected if S < ca, where ca is the criticalvalue (here a ¼ 0:05). Since the distribution of S under the nullhypothesis is not known it is impossible to calculate ca directly.Alternatively, the bootstrap method (Simar & Wilson, 1998,2000) is used to simulate the empirical distribution of S underthe null hypothesis and thus obtain the results of the testing ofhypotheses (Banker, 1996; Bogetoft & Otto, 2010; Simar &Wilson, 1998; Simar & Wilson, 2002). The results of the hypothesistest for return scales using the bootstrap method is presented inthe next section.

4. Applied methodology and experimental setup

A set of three relevant time series was used to evaluate the rel-ative efficiency of different Fitness Functions proposed in thiswork. The first time series is a stock market index based on thecommon stock prices of 500 leading companies publicly tradedAmerican Companies, named S&P500 Index (Standard & Poor500) (S.P.S. Index, 2011). Here, the S&P500 series consists of 369monthly observations from January 1970 to August 2003. This timeseries has a inherent random behavior of financial market and apronounced trend.

The second time series analyzed is the sunspot series (derLinden, 2011). This time series has a non linear and quasi-periodicbehavior without a trend. The sunspot population quickly rise andmore slowly fall on irregular cycle around of 11 years.

Finally, the third time series boarded here is the monthly milkproduction in the United States (N.A.S. Service(Nass)), consistingof 168 points collected monthly between January 1962 andDecember 1975. This series combine two behaviors: seasonalityand trend.

Before of the simulations, all series employed were normalizedbetween ½0;1� and divided into three parts: training set (50% of thedata), validation set (25% of the data) and test set (25% of thedata). In experiments, a Multi-layer Perceptron (MLP) ArtificialNeural Network (ANN) was used with fixed architecture 3–5–1,i.e. 3 nodes in input layer, 5 nodes in hidden layer and 1 neuronin output layer (forecasting horizon of one step ahead).

The training of the ANNs is driven by an Evolutionary Algo-rithm, where here was applied an Evolutionary Strategy (Beyer &Schwefel, 2002; Rechenberg, 1978; Schwefel, 1981) (ES) to evolvethe ANN weights to minimize the prediction errors. The ES imple-mented in this paper uses the Sum Strategy (lþK) with l ¼ 1 andK ¼ 1, where in each iteration an individual (solution) generates anoffspring (another solution) through a mutation following a Nor-mal Distribution and both solutions compete to generate the pop-ulation of the next generation. Here, an individual, or a solution, isan ANN and the routine to adjust the ANN weights follows thesteps bellow:

1. The initial population of ANNs is randomly generated using aNormal Distribution in the range [0;1];

2. The fitness of the individual (ANN) is calculated using the eval-uation measures (Fitness Function);

3. An offspring is generated through a mutation using a NormalDistribution with mean equals to zero and standard deviationequal to r;

4. The offspring is evaluated. If the offspring’s fitness is at least asgood as the parent one, it becomes the parent in the nextgeneration. Otherwise, the offspring is disregarded.

For all series the same parameters were used. Computationally,the individual is a vector composed by ANN weights plus the



541

542

543544

546546

547

548

549

550

551552

554554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

586

587

588

589 Q4

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

Table 1Fitness Functions.

Fitness Functions

f1 11þARV

f11 POCID1þARV

f2 11þMSE

f12 POCID1þMAPE

f3 11þMAPE

f13 POCID1þMSE

f4 11þTHEIL

f14 POCID1þTHEIL

f5 11þMSEþARV

f15 POCID1þMSEþARV

f6 11þMSEþMAPE

f16 POCID1þMSEþMAPE

f7 11þMSEþTHEIL

f17 POCID1þMSEþTHEIl

f8 11þARVþMAPE

f18 POCID1þARVþMAPE

f9 11þARVþTHEIL

f19 POCID1þARVþTHEIL

f10 11þMAPEþTHEIL

f20 POCID1þMAPEþTHEIL

Table 2Hypothesis Testing using the bootstrap method to choose the DEA model (a ¼ 0:05).

Series S ca

S&P500 0.868312 0.715752Sunspot 0.936844 0.955787Milk 0.783449 0.869256

0 100 200 3000

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index

Fig. 2. S&P500 Index.

0 0.2 0.4 0.6 0.8 10

50

100

150

200

250

300

350

400

Efficiency

Frequency

Fig. 3. Histogram for the distribution of efficiencies of the Fitness Functions(S&P500 series).



17 June 2014

Q1

mutations steps ri, where i ¼ 1; . . . ;wtotal (wtotal is the number ofANN weights, where here wtotal ¼ 57). Therefore, the individual jis represented by a chromosome given by the vector:

indiv idualj ¼ wðjÞ1 ;wðjÞ2 ; . . . ;wðjÞwtotal

;rðjÞ1 ;rðjÞ2 . . . ;rðjÞwtotal

D Eð28Þ

where j ¼ 1;2; . . . ; ðlþKÞ. Here, the size of the population islþK ¼ 2.

The mutation mechanism and the criterion of r coevolution(Eiben & Smith, 2003) used were the non-correlated mutation,given by equations:

r0i ¼ ries0Nð0;1ÞþsNð0;1Þ ð29Þw0i ¼ wi þ r0iNð0;1Þ ð30Þ

where Nð0;1Þ is a Normal Distribution with mean equals to zeroand standard deviation equals to one. The initial value of r is ran-domly generated by Uniform Distribution in the range ½0;1�. The

learning rates are given by s ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi2ffiffiffiffiffiffiffiffiffiffiffiwtotalpp� ��1

¼ffiffiffiffiffiffiffiffiffiffiffiffiffi2ffiffiffiffiffiffi57pp� ��1

e s0 ¼ ðffiffiffiffiffiffiffiffiffiffiffiffiffiffi2wtotalp

Þ�1 ¼ ðffiffiffiffiffiffiffiffiffi114p

Þ�1following the heuristics rules

reported by Bäck (1996) and Eiben and Smith (2003). Here, aboundary rule is also applied to prevent standard deviation r veryclose to zero, where if ri < �0 then ri ¼ �0 (where �0 ¼ 10�3).

The ES will stop if the maximum number of ES iterations isreached (where was used the maximum number of iteration equalsto 106) or if a fraction of the maximum number of iterations (thisfraction is 10%) is reached without an improvement in the fitness.The ES procedure combined with MLP is demonstrated in the Algo-rithm 1.

Algorithm 1. ES procedure

Initialize a 0 // Iterations NumberGenerate PðaÞ // Initial Populationoffspring Mutation (PðaÞ);Evaluate f (offspring)// f (.) is Fitness Functionwhile stop criterion not satisfied do

if f ðoffspringÞP f ðPðaÞÞ thenPðaþ 1Þ ¼ offspring;

elsePðaþ 1Þ ¼ PðaÞ;

end ifa ¼ aþ 1;offspring Mutation (PðaÞ);

end while

609

610

611

612

After the end of the simulations, the Fitness Functions value ofthe best individuals of the Evolutionary Strategy will be used as aDMUs in DEA (as seen in Section 3.2). The efficiencies are calcu-lated and the generated predictions are analyzed, correlating theprediction accuracy with the Fitness Function efficiency.


For the analysis, 20 Fitness Function were built (Table 1), andfor each one of them was executed 30 simulations. Since for eachsimulation there is an elected individual (the best individual) withdifferent characteristics, 20 groups of 30 DMUs were created.Therefore, a total of 600 DMUs per time series were generated.

The input and output variables were defined (as cited in Section3.3) based on measures of forecast error, calculated after the elec-tion of the best individuals for each simulation with a Fitness Func-tion. In general, in the DEA, the variables which should beminimized are considered inputs and one that which should bemaximized are considered outputs. Here, the variables are:

� Inputs: ARV, MAPE, MSE, THEIL� Outputs: POCID

4.1. Hypothesis test

As described in Section 3.4, the bootstrap method was used inthis work to determine the best DEA model for each time series.The results of the statistical test, where the hypothesis of constantreturns to scale and the alternative hypothesis of return variablesof scale are compared using the bootstrap method (Bogetoft &Otto, 2010). Table 2 shows the results of the statistical test using



613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

Table 3Metrics for efficient and less efficient Fitness Functions for the S&P500 series and the throughput with respect to best individual.

Order DMUs OE ðz�Þ k� Observed Values Slacks Projected Values

f7rep1 MSE ARV MAPE THEIL POCID s��

mse s��

arv s��

mape s��

theil sþ�

pocidMSE ARV MAPE THEIL POCID

1 f7rep1 1.0000 1.0000 0.0002 0.0169 0.0139 2.1185 0.5111 0.0000 0.0000 0.0000 0.0000 0.0000 0.0002 0.0169 0.0139 2.1185 0.51112 f9rep10 0.8824 1.1739 0.0003 0.0303 0.0185 3.7019 0.6000 8.9E-5 0.0068 0.0000 0.7797 0.0000 0.0002 0.0198 0.0163 2.4870 0.60003 f9rep1 0.7973 1.1086 0.0004 0.0359 0.0193 4.3920 0.5666 0.0001 0.0098 0.0000 1.1532 0.0000 0.0002 0.0187 0.0154 2.3488 0.56664 f4rep24 0.6782 1.0000 0.0004 0.0381 0.0205 4.6331 0.5111 0.0001 0.0089 0.0000 1.0240 0.0000 0.0002 0.0169 0.0139 2.1185 0.51115 f14rep20 0.6403 1.0652 0.0006 0.0479 0.0231 6.0052 0.5444 0.0001 0.0126 0.0000 1.5888 0.0000 0.0002 0.0180 0.0148 2.2567 0.5444

596 f13rep29 0.0114 0.8260 0.6910 53.4892 0.9997 6600.61 0.4222 0.0077 0.6009 0.0000 74.1286 0.0000 0.0001 0.0139 0.0114 1.7501 0.4222597 f13rep24 0.0112 0.7826 0.6615 51.2021 0.9694 6313.91 0.4000 0.0072 0.5618 0.0000 69.2555 0.0000 0.0001 0.0132 0.0108 1.6580 0.4000598 f13rep21 0.0105 0.7608 0.6913 53.5108 0.9999 6603.29 0.3888 0.0071 0.5535 0.0000 68.2902 0.0000 0.0001 0.0128 0.0105 1.6119 0.3888599 f3rep24 0.0099 0.6956 0.6673 51.6543 0.9750 6370.78 0.3555 0.0064 0.5009 0.0000 61.7642 0.0000 0.0001 0.0117 0.0096 1.4737 0.3555600 f2rep12 0.0094 0.6739 0.6812 52.7292 0.9893 6505.33 0.3444 0.0063 0.4882 0.0000 60.2202 0.0000 0.0001 0.0114 0.0093 1.4277 0.3444

Table 4Potential improvement for inefficient units in series S&P500.

Order DMUs Improvement Potential (%)

MSE ARV MAPE THEIL POCID

1 f7rep1 0.00 0.00 0.00 0.00 0.002 f9rep10 �34.42 �34.42 �11.76 �32.82 0.003 f9rep1 �47.75 �47.75 �20.26 �46.52 0.004 f4rep24 �55.56 �55.56 �32.17 �54.27 0.005 f14rep20 �62.37 �62.37 �35.96 �62.42 0.00



17 June 2014

Q1

the bootstrap method to choose the appropriate DEA model for thethree times series investigated here, the sunspot series, the S&P500index series and the milk series. The values of the statistic S and thecritical value ca for the three series are shown. The null hypothesisis rejected for the sunspot and milk series, because the value of thestatistic S is smaller than the critical value. Thus, the variablereturns to scale is adopted for the sunspot series and milk. Forthe S&P500 series (Table 2, S > ca) is adopted the model with con-stant return to scale.

656

657

658

659

660

661

662

663

664

665

666

667

668

669

596 f13rep29 �99.97 �99.97 �98.85 �99.97 0.00597 f13rep24 �99.97 �99.97 �98.88 �99.97 0.00598 f13rep21 �99.98 �99.98 �98.94 �99.98 0.00599 f3rep24 �99.98 �99.98 �99.01 �99.98 0.00600 f2rep12 �99.98 �99.98 �99.05 �99.98 0.00

Table 5Weights and percentage contribution input/output for the efficiency (multiplierform)—S&P500 series.

Order DMUs v�mse v�arv v�mape v�theil u�pocid

1 f7rep1 0.000 0.000 71.876(100%) 0.000 1.957(100%)2 f9rep10 0.000 0.000 54.029(100%) 0.000 1.471(100%)3 f9rep1 0.000 0.000 51.693(100%) 0.000 1.407(100%)4 f4rep24 0.000 0.000 48.752(100%) 0.000 1.327(100%)5 f14rep24 0.000 0.000 43.209(100%) 0.000 1.176(100%)

596 f14rep20 0.000 0.000 1.000(100%) 0.000 0.027(100%)597 f13rep29 0.000 0.000 1.032(100%) 0.000 0.028(100%)598 f13rep21 0.000 0.000 1.000(100%) 0.000 0.027(100%)599 f13rep24 0.000 0.000 1.026(100%) 0.000 0.028(100%)600 f2rep12 0.000 0.000 1.011(100%) 0.000 0.028(100%)

5. Experimental results

5.1. S&P500 series

The S&P500 (Standard & Poor 500) index series presents a reg-ular movement of growth. This growth trend is viewed in Fig. 2.The model chosen for the S&P500 according to hypotheses setout in Section 3.4 follows the assumption of constant returns toscale input oriented. Fig. 3 shows frequency distribution histo-grams of efficiency to the 600 evaluated units following the CCRmodel.

Note that the highest concentration of efficiencies is located in aregion with efficiency below 0.3 (91.83%). This indicates that mostof the evaluated units have been unable to achieve the efficientfrontier. Only one unit assessed (0.17%) was able to achieve it.The results of the 5 highest and 5 lowest evaluated units for theS&P500 series are shown in Table 3, along with the correspondingoverall efficiency (OE), weight peers, observed values, slacks andthe values for the improvement of inefficient units (projectedvalues).

We must remember that the evaluated unit is considered effi-cient if it obtains the optimal objective value z� ¼ 1 and all slacksare null in the solution of the dual model. Table 3 also shows thatonly one unit f 7rep12 is considered efficient, following these condi-tions. The unit f 7rep1 is taken as benchmark for all other units con-sidered inefficient. Still in the Table 3, the 4th column shows theweights attributed to the improvement of inefficient units. Throughthese weights, the inefficient units are projected on the surface ofefficiency. In this process of projection, there are residues (slacks)and through them is built the coordinates for the optimal values ofinputs and outputs (projected values). For example, for the unit(f 9rep10) becomes effective is necessary to reduce the inputs propor-tionally in (1�0.882435) 11.75% and reduce the slacks correspondingto each input. Thus, the optimal value for the input ‘‘MSE’’ (columnprojected value) will be 0.000257 [(0.882435⁄0.00392)�0.00089]while the observed value was 0.00392 (column observed value).

670

671

672

673

2 The notation utilized to represent the Fitness Function as a DMU is fXrepY, whereX is the number of the Fitness Function defined in Table 1 and the Y is the simulationnumber for the respective X Fitness Function (Y is an integer 2 ½1;30�).


Table 4 shows the potential improvement needed for the ineffi-cient units become efficient.

Note that the unit f 1rep7 was the only efficient and does notrequire improvement in their inputs/outputs. Also note that thevalues of the output (POCID) have not additional expansion(shortfalls). For the values of inputs ‘‘MSE’’ and ‘‘ARV’’ is giventhe same improvement conditions, indicating a common charac-teristic for these two variables. Inefficient units need to improvearound 100% in order to achieve the efficiency frontier, indicatingthat optimum values are very far from ideal. It is also importantto note that the input ‘‘MAPE’’ had no slack in its projection(Table 3), but requires improvement in the value of their input.The potential improvement was lower than all other entries indi-cating that the reduction in the consumption of this variable ismore important for obtaining the efficiency of that reduction ofother inputs. This fact can be better seen in Table 5, where it isshown the weights that contributed to find the optimal objectivevalue w� (2).



674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

0 20 40 60 800.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Normalized Index

Fitness Function−f7rep1

Monthly Records

0 20 40 60 800.65

0.7

0.75

0.8

0.85

0.9

0.95

1Fitness Function−f9rep1

Monthly Records

Normalized Index

0 20 40 60 800.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Monthly Records


Normalized Index

Fig. 4. Figure comparing the actual series (solid line) and the prediction generated by the neural network (dashed line) using the DMU more efficient to the S&P500 series.

0 50 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Monthly Records

Normalized Index


0 50 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Monthly Records

Normalized Index


0 50 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Monthly Records

Normalized Index


Fig. 5. Predicted values for S&P500 series to the less efficient DMUs. The solid line is the real data and the dashed line is the generated prediction.

0 100 200 3000

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index

Fig. 6. The annual mean sunspot number time series.



17 June 2014

Q1

The CCR model assigned weights only for input ‘‘MAPE’’ show-ing the importance of this variable. The highest weights were forthe most efficient units and smaller weights for less efficient. Thecontribution of the weight assigned to the product ‘‘output’’ willalways be 100% because it is the only variable considered. Theoverall efficiency (OE) of the unit is obtained by multiplyingthe weight by the value produced by the output POCID. Forexample, the unit more efficient (f7rep1) has 100% [(1.957)⁄(0.511111)].

As defined in Table 1, the evaluated units (DMU) are the func-tions of fitness of the best individuals of the Evolutionary Strategy.These functions values indicate the performance of the modelobtained by correlating the observed data with its prediction. Ifthe model was able to find only one efficient unit projected cor-rectly on the efficiency frontier, then when the original data seriesare compared with predicted values of this efficient unit isexpected to obtain a good fit.

Fig. 4 shows the predicted values (compared with the actualseries) for the three most efficient DMUs to DEA model adopted(f 7rep1; f 9rep10 and f 9rep1), where the dashed line is the predic-tion and the solid lines is the real data.

Visually is noted that the function f 7 obtained the best fit com-pared the other two DMUs.

Fig. 5 presents the predicted values for the three DMUs less effi-ciently and visually note that these Fitness Functions can not traina model with sufficient accuracy to adjust the actual series data.


5.2. Sunspot series

The time series of sunspots shows a quasi-periodic behavioralong time as observed in Fig. 6.

In the DEA analysis, the model employed for the sunspot seriesfollows the assumption of variable returns to scale input oriented.Fig. 7 shows the efficiency distribution for the 600 units evaluatedaccording to the BCC model.

It should be noted that the efficiencies were concentrated onthe right side of the distribution representing about 85% of theevaluated units. From the samples analyzed, 1.33% of the units



710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

0 0.2 0.4 0.6 0.8 10

50

100

150

200

Efficiency

Frequency

Fig. 7. Histogram for the distribution of efficiencies (sunspot series).



17 June 2014

Q1

showed maximum efficiency. The number of efficiency observa-tions made by the Fitness Function are summarized in Table 6,which is classified according to overall measures of efficiency(OE), purely technical efficiency (PTE) and scale efficiency (SE)obtained.

It should be noted that 19 units presented scalar efficiency and8 units presented pure technical efficiency, however, only 4 units(f 1; f 5; f 15; f 17) showed global efficiency. These 4 units met thecriteria of the CCR model, a derivative of the BCC model, and thuswere able to reach a maximum scale of productivity. Of the 8 unitswith pure technical efficiency, 4 needed to improve their globalefficiency in order to reach a maximum scale of productivity. Inthe 30 repetitions, 8 units (f 4; f 7; f 8; f 9; f 10; f 11; f 19; f 20) neverattained scalar efficiency which implies that their inefficiencywas due to the increasing and decreasing factors of their corre-sponding scale of production. In Table 7 the results of the 8 highestand 8 lowest evaluated units for the sunspot series, along withtheir corresponding pure technical efficiencies (PTE), Returns,

Table 6Number of efficient observations by Fitness Functions for the Sunspot Series.

DMU f1 f2 f3 f4 f5 f6 f7 f8 f9 f10

Observations 30 30 30 30 30 30 30 30 30 30OE 1 0 0 0 1 0 0 0 0 0PTE 2 0 0 0 1 0 0 0 0 0DRS 2 0 0 0 1 0 0 0 0 0SE 2 2 2 0 1 1 0 0 0 0

Table 7Metrics for the efficient and less efficient Fitness Functions for the sunspot series and the

Order DMU PE Return Peer Weight (%)

v�mse v�arv

1 f1rep22 1 con. f1rep22 30.989 0.0002 f5rep5 1 con. f5rep5 30.989 0.0003 f15rep2 1 con. f15rep2 0.000 0.0004 f17rep26 1 con. f17rep26 2.947 0.0005 f1rep20 1 dec. f1rep20 0.000 0.0006 f14rep29 1 dec. f14rep29 39.309 0.0007 f19rep19 1 dec. f19rep19 0.000 0.0008 f11rep30 1 dec. f11rep30 41.816 0.000

593 f13rep1 0.4349 dec. f1rep20,f15rep2 0.000 0.000594 f11rep11 0.4271 Inc. f1rep22 0.000 0.000595 f13rep11 0.3959 dec. f1rep20,f15rep2 0.000 0.000596 f13rep30 0.3956 Inc. f1rep22 0.000 0.000597 f13rep21 0.3808 con. 15rep2 0.000 0.000598 f13rep27 0.3796 Inc. f1rep22, f5rep5, f17rep26 0.000 0.737599 f13rep14 0.3673 Inc. f1rep22,f15rep2 0.000 0.000600 f13rep6 0.3092 dec. f1rep20,f15rep2 0.000 0.000


Reference Set (Peer), slacks and the values for the improvementof inefficient units (projected values) are shown.

Constant scalar returns were seen in only 4 DMUs (f 1rep22;f 5rep5; f 15rep2; f 17rep26) of the 8 most efficient DMUs. Theremaining 4 (f 1rep20; f 14rep29; f 19rep19; f 11rep30) units werecharacterized by decreasing returns. For these 4 units, the inputvariation produces inversely proportional variation in the final out-put. In the 6th column of the Table 7 the weights assigned by theBCC model (multiplier form) are shown. Notice that the inputsMSE and MAPE had the highest contributions to the optimal objec-tive value. Under the potential improvement, there is not need toimprove the units that are already efficient. In other words, theprojection of these units on the frontier efficiency does not produceslacks, thus signifying that the adjustment was adequate. On theother hand, the most inefficient units presented a very high per-cent of improvement which is indicative of a high need to improveinputs. For example, the unit f 13rep6 needs to reduce its consump-tion by approximately 694% for the inputs MSE e ARV, 223.4% forthe input MAPE and 761% for the input THEIL. These values indi-cate that the DMUs are more inefficient further away they are fromthe frontier efficiency. For each of these inefficient units, a refer-ence set (based on the efficiency units) at which the inefficientunits would be efficiency. In addition, for f 13rep6, the benchmarkswere f 1rep20 and f 15rep2. For each reference, (k�) determines theweight of the contribution for the constructions of projected val-ues. Table 8 shows the average contributions of each benchmark,aggregated by the Fitness Function.

Observing the Table 8, the unit f 1rep22 have the largest globalaverages (72:22%) contribution to the projection of the inefficientunits. The weight given to this DMU was always more decisivefor all the Fitness Functions groups (f 1; f 2; . . . ; f 20). In secondplace, the DMU f 15rep2 had a global average contribution of10:26%, followed by the DMU f 5rep5 (9:36%). The efficiency unitthat less contributed was the DMU f 19rep19, with 0:17%.

f11 f12 f13 f14 f15 f16 f17 f18 19 f20 SUM

30 30 30 30 30 30 30 30 30 30 6000 0 0 0 1 0 1 0 0 0 41 0 0 1 1 0 1 0 1 0 81 0 0 1 1 0 1 0 1 0 80 1 3 1 2 1 2 1 0 0 19

throughput with respect to best individual.

Potential improvement

v�mape v�theil u�pocid MSE ARV MAPE THEIL POCID

1.766 0.000 0.000 0.00 0.00 0.00 0.00 0.001.766 0.000 0.000 0.00 0.00 0.00 0.00 0.003.308 0.000 1.034 0.00 0.00 0.00 0.00 0.003.020 0.000 1.592 0.00 0.00 0.00 0.00 0.002.968 0.000 1.442 0.00 0.00 0.00 0.00 0.000.894 0.000 8.707 0.00 0.00 0.00 0.00 0.000.230 0.713 8.278 0.00 0.00 0.00 0.00 0.000.326 0.000 6.217 0.00 0.00 0.00 0.00 0.00

1.407 0.000 0.683 �213.17 �213.17 �129.89 �229.00 0.001.456 0.000 0.000 �153.77 �153.77 �134.12 �181.28 0.001.252 0.000 0.608 �280.11 �280.11 �152.58 �299.71 0.001.349 0.000 0.000 �236.20 �236.20 �152.77 �275.76 7.271.260 0.000 0.394 �383.87 �383.87 �162.56 �432.42 0.000.676 0.000 0.238 �163.41 �163.41 �163.41 �183.90 0.001.234 0.000 0.385 �281.54 �281.54 �172.19 �308.14 0.001.000 0.000 0.486 �694.05 �694.05 �223.40 �761.28 0.00



762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

0 20 40 600

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index


0 20 40 600

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index


0 20 40 600

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index


0 20 40 600

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index


0 20 40 600

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index


0 20 40 600

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index


0 20 40 600

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index


0 20 40 600

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index


Fig. 8. Figure comparing the actual series (solid line) and the prediction generated by the neural network (dashed line) using the DMU more efficient to the sunspot series.

0 10 20 30 40 50 60 700

0.2

0.4

0.6

0.8

1

Annual Records

Normalized Index

Sunspot f13rep14 f13rep6

Fig. 9. Sunspot series forecasting with the smallest efficient DMUs.

0 50 100 1500

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index

Fig. 10. The series of monthly milk production.

Table 8Average contribution of the weights peer by Fitness Functions for the sunspot series and the Global Average (GA).

k� Fitness Function

f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14 f15 f16 f17 f18 19 f20 GA

f1rep22 66.95 62.44 93.33 58.32 65.19 79.74 67.20 75.91 75.61 93.78 73.74 87.36 48.16 67.49 74.57 66.67 55.21 78.98 69.02 84.71 72.22f1rep20 5.83 3.67 0.00 2.62 3.82 2.36 2.67 4.10 0.00 0.00 5.45 0.00 11.84 3.00 1.28 5.86 5.63 2.19 5.21 2.08 3.38f5rep5 14.72 16.95 0.00 34.29 14.12 0.48 21.73 2.43 21.69 4.55 6.92 0.00 1.82 11.64 12.40 0.00 14.31 1.02 4.55 3.62 9.36f11rep30 0.71 0.53 0.00 0.00 2.20 0.00 0.00 0.00 0.00 0.00 3.46 0.00 0.00 0.26 0.00 0.00 0.63 0.00 1.25 1.22 0.51f14rep29 1.99 5.12 0.00 0.00 2.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 4.36 0.00 0.00 4.23 0.00 1.27 0.71 0.99f15rep2 9.10 8.09 6.67 2.43 5.82 15.43 2.88 14.74 1.45 1.67 4.25 11.25 32.20 9.76 10.87 23.42 10.30 15.42 11.84 7.67 10.26f17rep26 0.70 3.20 0.00 2.34 6.72 1.99 5.53 2.82 1.25 0.00 6.10 1.39 5.95 3.49 0.87 4.05 9.67 2.39 3.53 0.00 3.10f19rep19 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.33 0.00 0.17



17 June 2014

Q1

By observing the units that achieved a technical pure efficiency,and comparing the sunspot series (original data) to the predictedvalues obtained by the respective units, it is hoped that thesetwo values show a good adjustment.

Fig. 8 presents the predicted values for the sunspot series, basedon the eight most efficient DMUs (Table 7), where the dashed lineis the prediction and the solid lines is the real data.

The Fitness Functions with global efficiency, by the DEAanalysis, obtained a good fit with the real time series data


(Fig. 8). Visually, the units f 11rep30 and f 14rep29, that had higherweights assigned to input ‘‘MSE’’ (see Table 7), shown a good fit tothe sunspot series data (Fig. 8).

In contrast, Fig. 9 shows the forecast for the sunspot seriesbased on the two least efficient DMUs. It is verified that the FitnessFunction f 13 did not reach a good forecast for the sunspot series,where the small efficiency of the Fitness Function implies a poor



778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

0 0.2 0.4 0.6 0.8 10

50

100

150

Efficiency

Frequency

Fig. 11. Histogram for the distribution of efficiencies (milk series).



17 June 2014

Q1

forecasting capacity of the predictive model. Therefore, the FitnessFunction f 13 is not a good choice for the sunspot series.

5.3. Milk series

The time series of monthly milk production in the United Statesshows a seasonal behavior and a tendency. These two features ofthe milk time series can be observed in Fig. 10.

The model chosen for the milk series follows the assumption ofvariable returns to scale input oriented, similar to the sunspot ser-ies. Fig. 11 shows the efficiency distribution of the 600 evaluated

Table 9Number of efficient observations by Fitness Functions for the Milk Series.

DMU f1 f2 f3 f4 f5 f6 f7 f8 f9 f10

Observations 30 30 30 30 30 30 30 30 30 30OE 1 0 0 0 1 0 0 0 0 0PTE 0 1 0 0 0 0 1 0 0 0DRS 0 0 0 0 0 0 1 0 0 0SE 0 0 0 0 0 0 0 0 0 0

Table 10Metrics for the efficient and less efficient Fitness Functions for the milk series and the thr

Order DMU PE Return Peer Weight (%)

v�mse v�arv v�mape

1 f2rep17 1 dec. f2rep17 0.000 0.000 8.7092 f7rep26 1 dec. f7rep26 0.000 0.000 6.3243 f12rep23 1 dec. f12rep23 0.000 0.000 2.1024 f19rep25 1 dec. f19rep25 0.000 0.000 7.2165 f20rep22 1 dec. f20rep22 0.000 0.000 4.345

596 f12rep7 0.1748 dec. f2rep17, f19rep25 0.000 0.000 2.569597 f16rep19 0.1649 dec. f2rep17, f19rep25 0.000 0.000 3.602598 f13rep29 0.1609 dec. f2rep17, f19rep25 0.000 0.000 3.591599 f16rep8 0.1575 dec. f2rep17, f19rep25 0.000 0.000 2.347600 f3rep8 0.1489 dec. f2rep17, f19rep25 0.000 0.000 4.490

Table 11Average contribution of the weights peer by Fitness Functions to the Milk Series and the

k� Fitness Function

f1 f2 f3 f4 f5 f6 f7 f8 f9 f10

f2rep17 57.52 65.38 58.16 59.73 59.57 67.04 49.07 52.96 58.84 51.77f7rep26 0.00 0.00 0.00 0.00 0.00 0.00 3.33 6.67 0.00 0.00f12rep23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00f19rep25 42.48 34.62 41.84 40.27 40.43 32.96 47.60 40.37 41.16 48.23f20rep22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00


units following the BCC model. Note that the majority of the ana-lyzed units had pure technical efficiency (PTE) scores lower than0.6 (approximately 66.51%). Only 5 units (0.83%) were able to reachthe frontier of efficiency following the BCC model.

In Table 9, the number of evaluated units that were efficient,grouped by the Fitness Function, is shown. In this table, note thatonly 4 units showed global efficiencies (CCR model), 5 showed puretechnical efficiency according to the BCC model and 4 units wereefficiency according to the model of decreasing returns to scale.It is important to observe that no unit evaluated by the milk seriespresented scalar efficiency, which indicates that all of the units areoperating above or below the optimal scale. Table 10 shows theresults of the 5 highest and 5 lowest evaluated units for the milkseries, along with their corresponding technical efficiencies (PTE),Returns, Reference Set (Peer), slacks, and the values for theimprovement of inefficient units (projected values). The resultspresented in Table 10 verify that units demonstrating pure techni-cal efficiency (PE) as well as the most inefficient units, had decreas-ing returns to scale, working above the optimal scale. The weightswere attributed by the dual model (multiplier form), the emphasiswas given the input ‘‘MAPE’’, with emphasis on the units f 2rep17and f 19rep25 that had the largest weights. The bigger weightattributed to the ‘‘POCID’’ output was f 19rep25, however this unithad the lowest weight ‘‘MAPE’’ input, in other words for this unitthe highest weight was given to the product.

The seventh column of the Table 10 contains the potentialimprovement of the inefficient units. The efficiency units do not

f11 f12 f13 f14 f15 f16 f17 f18 19 f20 SUM

30 30 30 30 30 30 30 30 30 30 6000 0 0 0 1 0 1 0 0 0 40 1 0 0 0 0 0 0 1 1 50 1 0 0 0 0 0 0 1 1 40 0 0 0 0 0 0 0 0 0 0

oughput with respect to best individual.

Potential Improvement

v�theil u�pocid MSE ARV MAPE THEIL POCID

0.000 0.896 0.000 0.000 0.000 0.000 0.0000.000 4.823 0.000 0.000 0.000 0.000 0.0000.000 20.135 0.000 0.000 0.000 0.000 0.0000.000 0.743 0.000 0.000 0.000 0.000 0.0000.000 6.101 0.000 0.000 0.000 0.000 0.000

0.000 0.264 �2771.815 �2771.815 �471.948 �2793.932 0.0000.000 0.371 �2564.160 �2564.160 �506.196 �2565.303 0.0000.000 0.370 �2159.784 �2159.784 �521.121 �2136.770 0.0000.000 0.242 �1975.774 �1975.774 �534.815 �1949.200 0.0000.000 3.424 �2607.480 �2607.480 �571.432 �2589.380 0.000

Global Average (GA).

f11 f12 f13 f14 f15 f16 f17 f18 f19 f20 GA

20.00 7.41 10.00 35.93 23.70 17.04 32.22 29.63 35.50 32.60 41.203.33 15.00 28.33 3.33 0.00 15.00 5.00 11.67 8.33 3.33 5.170.00 3.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17

73.33 69.26 53.33 57.41 72.96 62.96 61.11 53.70 54.50 60.74 51.463.33 5.00 8.33 3.33 3.33 5.00 1.67 5.00 1.67 3.33 2.00



814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

0 20 40

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


0 20 40

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


0 20 40

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


0 20 40

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


0 20 40

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


Fig. 12. Figure comparing the actual series (solid line) and the prediction generated by the neural network (dashed line) using the DMU more efficient to the milk series.

0 20 400

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


0 20 400

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


0 20 400

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


0 20 400

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


0 20 400

0.2

0.4

0.6

0.8

1

Monthly Records

Normalized Index


Fig. 13. Figure comparing the actual series (solid line) and the prediction generated by the neural network (dashed line) using the DMU less efficient to the milk series.



17 June 2014

Q1

need to be improved. On the other hand, the most inefficient unitsneed a high reduction of their inputs in order for them to reach thefrontier efficiency. For example, f 3rep8 needs to reduce theobserved values of ‘‘MSE’’ inputs by almost 2607%, 2607% of‘‘ARV’’ inputs, 571% of ‘‘MAPE’’ inputs and around 2589% of the‘‘THEIL’’ inputs.

In the fifth column of the Table 10, the reference set is pre-sented for the inefficient units projected unto the frontier effi-ciency. The efficiency units are self-referencing. The inefficientunits have references f 2rep17 e f 19rep25, the same units whichdemonstrated the biggest weight assigned by the ‘‘MAPE’’ inputas previously stated.

In Table 11, the average percentage contribution for each aggre-gated benchmark of the Fitness Function is presented.

Note that the units f 2rep17 (41,20%) and f 19rep25 (51,46%)contributed the most to the inefficient units becoming efficient.


The least contributing units were f 7rep26; f 20rep22 and f 12rep23with average global percentages of 5.17%, 2.00% and 0.17% respec-tively. Fig. 12 shows the fit of the predictions to the original data.The efficiency units with which were associated low weights forthe ‘‘MAPE’’ input (Table 10) were those that needed the fewestadjustments to the original data. The units f 2rep17 and f 19rep25showed good fits for the upper part of the original series butshowed poor fits for the lower periodic regions. Among the effi-cient DMUs, it is possible to see that f 12rep23 presents the worstforecast, reflecting a lower value of v�mape weight (see Table 10).This shows that is possible to sort the efficient units according tothe values of v�mape, where DMU f 2rep17 on average has the bestprediction. The performance of predicted values of the less efficientDMUs is shown in Fig. 13. Neither of the two functions fit well toreal values. This fact is in accordance with the result of the efficien-cies that were very close to zero (as shown in Table 10).



846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951952953954955956957958959960961962963964965966967968969970Q5971972973974975976977978979980



17 June 2014

Q1

From the results of experiments in the three time series, we canconclude that DEA was able to identify the best and the worstFitness Functions. By comparing the prediction with the originalseries was possible to observe if their Fitness Function, responsiblefor guiding EA, was really efficient or not efficient. For S&P500 series(exponential trend), the most units evaluated had low scores overallefficiency. Only the unit f 7rep1 was considered efficient. Note thatthis unit consists by the ‘‘MSE’’ and ‘‘THEIL’’ performance metrics.For sunspot series (quasi-periodic behavior with a constant level),the choice of model with variable returns to scale identified 8 effi-cient units: f 1rep22, f 5rep5; f 15rep2; f 17rep26; f 1rep20; f 14rep29;f 19rep19; f 11rep30. The units f 14rep29 and f 11rep30 had higherweight assigned to input MSE. For milk series (with trend and a welldefined seasonality), the DEA model identified 5 efficient units:f 2rep17; f 7rep26; f 12rep23; f 19rep25; f 20rep22. For this series, theweights of the inputs were assigned only to MAPE input, being theunit f 2rep17 with highest weight. Therefore, these experimentalresults show that the statistics measures of MSE and MAPE, andthe combination of MSE with THEIL are a good choice to composethe Fitness Function for the time series forecasting problem.

In more detail, analyzing the functions regarded as more effi-cient, the experiments results shown that if the time series has astrong trend, then the combination of MSE and THEIL is good tocompose the fitness function. However, if the time series has aoscillate behavior, but a constant level, the MSE performance mea-sure is more appropriate. And, if the time series has a seasonalitycomponent combined with a trend, then the MAPE performancemeasure is more indicated to compose the fitness function.

6. Conclusion

Many intelligent techniques are applied to solve real worldproblems, where a very studied class of real world problems isthe time series forecasting problem.

A very popular approach for the time series forecasting problemwith expert system is to use an Evolutionary Algorithm to adjustthe parameters of a predictive model.

In this way, a vital point on the design of the Evolutionary Algo-rithm is the definition of the fitness function. However, studies forthe characterization of the Fitness Function is a branch poorlyexplored in the literature, where in general just one performancemeasure, commonly the MSE, is used as the statistical measureto guide the Evolutionary Algorithm. Surely, these fitness functionsbased on just one performance measure will work, but are theseFitness Function the best choice to guide the Evolutionary Algo-rithm for search a good solution?

In this paper was employed the DEA procedure to point the Fit-ness Function more efficient used in the Evolutionary Algorithm,with a constant number of iterations, to evolve an ANN as a predic-tive model. Three Time Series with specific features was used asapplication: the sunspot time series with a quasi-periodic behaviorwith a constant level; the S&P500 time series with a exponentialtrend and random shocks or fluctuations (additive white noise)and the milk time series with trend and a well defined seasonality.

After discussing the assumptions and methodology used to setthe EA and DEA, the model was idealized using 20 different func-tions (Table 1), where each one of these function is composed bya combination of up three performance measures. Experimentalresults showed that regardless of the time series feature, the DEAwas able to find the best Fitness Function based on the conceptof efficiency. Graphs comparing the original data to the predictedvalues were used to check the smooth adjustment of Fitness Func-tions efficient.

Some strengths of the proposed method can be highlighted:first, DEA is a non-parametric methodology which does notrequire, a priori, knowledge of the weights involved in the model.


The DEA can define which are the DMUs efficient and not efficientjust observing the group of all DMUs. Secondly, although of thework to use only one hybrid system (EE þ ANN), it was possibleto evaluate five performance measures, where 30 simulations weredone to each one of 20 combination of these performance measure,enabling to observe the behavior of each Fitness Function based onbehavior of these samples (Figs. 3, 7 and 11).

The Fitness Function will create a surface of solution quality(fitness landscape) where the Evolutionary Algorithm will searchby the global maximum. In this sense, an efficient Fitness Functionshould be as smooth as possible and provide a more easily accessi-ble global maximum for any Evolutionary Algorithm. In this way, itis possible expect that an efficient function will be effective regard-less of the type of Evolutionary Algorithm applied. However, thisstatement needs to be confirmed with further experiments usingother Evolutionary Algorithms. Therefore, here was expected thatthere is a correlation between the function and features of the timeseries, but not with the Evolutionary Algorithm (based on theresults of previous work Ferreira et al., 2008; Rodrigues et al.,2009; Rodrigues et al., 2010). This point is the biggest weaknessof the proposed methodology, because this point was not tested.In order to remedy this weakness, new experiments are being con-ducted with other Evolutionary Algorithms such as particle swarmoptimization and genetic algorithms. With these new experimentswill be possible answer whether there is a dependency of the func-tion with the Evolutionary Algorithm used.

In order to evaluate the limitations of the proposed methodol-ogy some future works can be listed. Beyond the methodologycan be applied to evaluate the efficiency of the Fitness Functionfor others Evolutionary Algorithms, the methodology can beapplied to a new data set, in order to find correlation betweenthe new time series features (not analyzed here, like heteroscedas-ticity, chaos, spurious dependencies, etc.) and the best configura-tion of the fitness function. Furthermore, methods for variablesselection (Nataraja & Johnson, 2011; Wagner & Shimshak, 2007;Ueda & Hoshiai, 1997) can be combined with the proposed meth-odology aiming to explore more parameters of the analyzed fore-casting model (ANN). Finally, these works also can be used forothers problems, like classification, clustering, pattern recognition,and others.

References

Abraham, B., & Ledolter, J. (2009). Statistical methods for forecasting. Wiley.Amjady, N., & Keynia, F. (2009). Day-ahead price forecasting of electricity markets

by mutual information technique and cascaded neuro-evolutionary algorithm.IEEE_J_PWRS, 24(1), 306–318.

Areekul, P., Senjyu, T., Toyama, H., & Yona, A. (2010). A hybrid ARIMA and neuralnetwork model for short-term price forecasting in deregulated market.IEEE_J_PWRS, 25(1), 524–530.

Armstrong, J. S., & Collopy, F. (1992). Error measures for generalizing aboutforecasting methods: Empirical comparisons. International Journal ofForecasting, 8(1), 69–80.

Bäck, T. (1996). Evolutionary algorithms in theory and practice: Evolution strategies,evolutionary programming, genetic algorithms. Oxford, UK: Oxford UniversityPress.

Banker, R. D. (1996). Hypothesis tests using data envelopment analysis. Journal ofProductivity Analysis, 7(2), 139–159.

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimatingtechnical and scale inefficiencies in data envelopment analysis. ManagementScience, 30(9), 1078–1092.

Belfore, L. A., II, & Arkadan, A.-R. (1997). Modeling faulted switched reluctancemotors using evolutionary neural networks. IEEE Transactions on IndustrialElectronics, 44(2), 226–233.

Beyer, H.-G., & Schwefel, H.-P. (2002). Evolution strategies – a comprehensiveintroduction. Natural Computing, 1(1), 3–52.

Bhuiyan, M. (2009). An algorithm for determining neural network architectureusing differential evolution. In International conference on business intelligenceand financial engineering, 2009. BIFE’09 (pp. 3–7). http://dx.doi.org/10.1109/BIFE.2009.10.

Bogetoft, P., & Otto, L. (2010). Benchmarking with DEA, SFA, and R. Springer.Boussofiane, A., Dyson, R. G., & Thanassoulis, E. (1991). Applied data envelopment

analysis. European Journal of Operational Research, 52(1), 1–15.


http://dx.doi.org/10.1109/BIFE.2009.10

http://dx.doi.org/10.1109/BIFE.2009.10


981982983984985986 Q6987988989990991992993994995996997998999

10001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061

10621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141

1142



17 June 2014

Q1

Box, G. E. P., & Jenkins, G. M. (1994). Time series analysis: Forecasting and control (3rded.). Upper Saddle River, NJ, USA: Prentice Hall PTR.

Charnes, A. et al. (1978). Measuring the efficiency of decision making units.European Journal of Operational Research, 2(6), 429–444.

Charnes, A. et al. (1994). Data envelopment analysis: Theory, methodology, andapplication. Norwell, MA, USA: Kluwer Academic Publishers.

Chen, Y., Yang, B., Dong, J., & Abraham, A. (2005). Time-series forecasting usingflexible neural tree model. Information Sciences, 174(3), 219–235.

Coll, V., & Blasco, O. M. (2006). Evaluación de la Eficiencia Mediante el AnálisisEnvolvente de Datos, Juan Carlos Martínez Coll.

Cooper, W. W., Seiford, L. M., & Tone, K. (2007). Data envelopment analysis: Acomprehensive text with models, applications, references and dea-solver software.Springer.

da S. Gomes, G. S., & Ludermir, T. B. (2013). Optimization of the weights andasymmetric activation function family of neural network for time seriesforecasting. Expert Systems with Applications, 40(16), 6438–6446.

der Linden, R. V. (2011). The SIDC team, Online catalogue of the sunspot index. URL<http://sidc.oma.be/html/sunspot.html>.

Desheng, W. (2009). Supplier selection: A hybrid model using DEA, decision treeand neural network. Expert Systems with Applications, 36(4), 8503–8508.

Donate, J. P., Li, X., Sánchez, G. G., & de Miguel, A. S. (2013). Time series forecastingby evolving artificial neural networks with genetic algorithms, differentialevolution and estimation of distribution algorithm. Neural Computing andApplications, 22(1), 11–20.

Eiben, A. E., & Smith, J. E. (2003). Introduction to evolutionary computing.SpringerVerlag.

Fan, W., Fox, E. A., Pathak, P., & Wu, H. (2004). The effects of fitness functions ongenetic programming-based ranking discovery for web search. Journal of theAmerican Society for Information Science and Technology, 55(7), 628–636.

Fare, R., Grosskopf, S., & Lovell, C. K. (1994). Production frontiers. CambridgeUniversity Press.

Farrell, M. (1957). The measurement of productive efficiency. Journal of the RoyalStatistical Society. Series A (General), 120(3), 253–290.

Fernandes, B. J., Cavalcanti, G. D., & Ren, T. I. (2013). Autoassociative pyramidalneural network for one class pattern classification with implicit featureextraction. Expert Systems with Applications, 40(18), 7258–7266<http://dx.doi.org/10.1016/j.eswa.2013.06.080> .

Ferreira, T. A. E., Vasconcelos, G. C., & Adeodato, P. J. L. (2008). A new intelligentsystem methodology for time series forecasting with artificial neural networks.Neural Processing Letters, 28(2), 113–129.

Golany, B., & Roll, Y. (1989). An application procedure for DEA. Omega, 17(3),237–250.

Gonzalez, B., Donate, J., Cortez, P., Sanchez, G., & de Miguel, A. (2012). Parallelizationof an evolving artificial neural networks system to forecast time series usingopenmp and mpi. In 2012 IEEE conference on evolving and adaptive intelligentsystems (EAIS) (pp. 186–191). http://dx.doi.org/10.1109/EAIS.2012.6232827.

Grzesiak, L., Meganck, V., Sobolewski, J., & Ufnalski, B. (2007). Genetic algorithm forparameters optimization of ann-based speed controller. In EUROCON, 2007. Theinternational conference on #34; Computer as a tool #34 (pp. 1700–1705). http://dx.doi.org/10.1109/EURCON.2007.4400689.

Guo, Y., Kang, L., Liu, F., Sun, H., & Mei, L. (2007). Evolutionary neural networksapplied to land-cover classification in zhaoyuan, china. In IEEE symposium oncomputational intelligence and data mining, 2007. CIDM 2007 (pp. 499–503).http://dx.doi.org/10.1109/CIDM.2007.368916.

Haykin, S. (1998). Neural networks: A comprehensive foundation (2nd Edition.). UpperSaddle River, NJ, USA: Prentice Hall PTR.

Hinojosa, V., & Hoese, A. (2010). Short-term load forecasting using fuzzy inductivereasoning and evolutionary algorithms. IEEE_J_PWRS, 25(1), 565–574.

Kittelsen, S. A. C. (1993). Stepwise DEA. choosing variables for measuring technicalefficiency in norwegian electricity distribution, Memorandum 06/1993, OsloUniversity, Department of Economics.

Kitts, B., Edvinsson, L., & Beding, T. (2001). Intellectual capital: From intangibleassets to fitness landscapes. Expert Systems with Applications, 20(1), 35–50.

Liao, G. -C. (2012). Application a novel evolutionary computation algorithm for loadforecasting of air conditioning. In Power and energy engineering conference(APPEEC), 2012 Asia-Pacific (pp. 1–4). http://dx.doi.org/10.1109/APPEEC.2012.6307573.

Lima, A., Cannon, A., & Hsieh, W. (2012). Downscaling temperature andprecipitation using support vector regression with evolutionary strategy. InThe 2012 international joint conference on neural networks (IJCNN) (pp. 1–8).http://dx.doi.org/10.1109/IJCNN.2012.6252383.

Luo, Y., Bi, G., & Liang, L. (2012). Input/output indicator selection for DEA efficiencyevaluation: An empirical study of chinese commercial banks. Expert Systemswith Applications, 39(1), 1118–1123.

Ma, H., Chan, J., Saha, T., & Ekanayake, C. (2013). Pattern recognition techniques andtheir applications for automatic classification of artificial partial dischargesources. IEEE Transactions on Dielectrics and Electrical Insulation, 20(2), 468–478.http://dx.doi.org/10.1109/TDEI.2013.6508749.

Makui, A., & Noushabadi, M. (2012). An empirical study for ranking insurance firmsusing a hybrid of data envelopment analysis and neural network. ManagementScience Letters, 2(8), 2923–2928.


Mandal, P., Senjyu, T., Urasaki, N., Funabashi, T., & Srivastava, A. K. (2007). A novelapproach to forecast electricity price for PJM using neural network and similardays method. IEEE_J_PWRS, 22(4), 2058–2065.

Merz, P. (2004). Advanced fitness landscape analysis and the performance ofmemetic algorithms. Evolutionary Computation, 12(3), 303–325.

N.A.S. Service(Nass). (2011). Milk production. URL http://www.nass.usda.gov/.Nataraja, R. N., & Johnson, A. L. (2011). Guidelines for using variable selection

techniques in data envelopment analysis. European Journal of OperationalResearch, 215(3), 662–669.

Nowlan, S. J., & Hinton, G. E. (1992). Simplifying neural networks by soft weight-sharing. Neural Computation, 4(4), 473–493.

Pai, P.-F., & Hong, W.-C. (2005). Forecasting regional electricity load based onrecurrent support vector machines with genetic algorithms. Electric PowerSystems Research, 74(3), 417–425.

Petkovic, D., Cojbašic, Zarko, & Lukic, S. (2013). Adaptive neuro fuzzy selection ofheart rate variability parameters affected by autonomic nervous system. ExpertSystems with Applications, 40(11), 4490–4495<http://dx.doi.org/10.1016/j.eswa.2013.01.055> .

Rechenberg, I. (1978). Evolutionsstrategien. In Simulationsmethoden in der Medizinund Biologie. Medizinische Informatik und Statistik (Vol. 8, pp. 83–114).Springer.

Rodrigues, A., de Mattos Neto, P. S. G., & Ferreira, T. A. E. (2009). A prime step in thetime series forecasting with hybrid methods: The fitness function choice. InInternational joint conference on neural networks, 2009. IJCNN 2009 (pp. 2703–2710).

Rodrigues, A., Silva, D. A., de Mattos Neto, P. S. G., & Ferreira, T. A. E. (2010). Anexperimental study of fitness function and time series forecasting usingartificial neural networks. In Proceedings of the 12th annual conferencecompanion on Genetic and evolutionary computation, GECCO’10(pp. 2015–2018). New York, NY, USA: ACM.

Sarkis, J., & Cordeiro, J. J. (2012). Ecological modernization in the electrical utilityindustry: An application of a bads—goods {DEA} model of ecological andtechnical efficiency. European Journal of Operational Research, 219(2), 386–395.

Schwefel, H. P. (1981). Numerical optimization of computer models. New York, NY,USA: John Wiley & Sons Inc..

Shadbolt, J., & Taylor, J. G. (2002). Neural networks and the financial markets:Predicting, combining, and portfolio optimisation. Springer.

Shyu, J., & Chiang, T. (2012). Measuring the true managerial efficiency of bankbranches in taiwan: A three-stage DEA analysis. Expert Systems withApplications, 39(13), 11494–11502.

Simar, L., & Wilson, P. (1998). Sensitivity analysis of efficiency scores: How tobootstrap in nonparametric frontier models. Management Science, 44(1), 49–61.

Simar, L., & Wilson, P. (2000). A general methodology for bootstrapping in non-parametric frontier models. Journal of Applied Statistics, 27(6), 779–802.

Simar, L., & Wilson, P. W. (2002). Non-parametric tests of returns to scale. EuropeanJournal of Operational Research, 139(1), 115–132.

Sotiroudis, S., Goudos, S., Gotsis, K., Siakavara, K., & Sahalos, J. (2013). Application ofa composite differential evolution algorithm in optimal neural network designfor propagation path-loss prediction in mobile communication systems.Antennas and Wireless Propagation Letters, IEEE, 12, 364–367. http://dx.doi.org/10.1109/LAWP.2013.2251994.

S.P.S. Index (2011). <http://www.standardandpoors.com>.Stepnicka, M., Cortez, P., Donate, J. P., & Stepnicka, L. (2013). Forecasting seasonal

time series with computational intelligence: On recent methods and thepotential of their combinations. Expert Systems with Applications, 40(6),1981–1992.

Thanassoulis, E. (2001). Introduction to the theory and application of dataenvelopment analysis: A foundation text with integrated software. Norwell, MA,USA: Kluwer Academic Publishers..

Theil, H. et al. (1966). Applied economic forecasting (Vol. 4). Amsterdam: North-Holland Publishing Company.

Tomczak, E. (2011). Application of {ANN} and {EA} for description of metal ionssorption on chitosan foamed structure—equilibrium and dynamics of packedcolumn. Computers & Chemical Engineering, 35(2), 226–235<http://dx.doi.org/10.1016/j.compchemeng.2010.05.012> .

Touloo, M., Sohrabi, B., & Nalchigar, S. (2009). A new method for ranking discoveredrules from data mining by DEA. Expert Systems with Applications, 36(5),9105–9112.

Ueda, T., & Hoshiai, Y. (1997). Application of principal component analysis forparsimonious summarization of DEA inputs and/or outputs. Journal of theOperations Research Society of Japan, 40(4), 466–478.

Wagner, J. M., & Shimshak, D. G. (2007). Stepwise selection of variables in dataenvelopment analysis: Procedures and managerial perspectives. EuropeanJournal of Operational Research, 180(1), 57–67.

Wang, P., Zhang, J., Xu, L., Wang, H., Feng, S., & Zhu, H. (2011). How to measureadaptation complexity in evolvable systems – {A} new synthetic approach ofconstructing fitness functions. Expert Systems with Applications, 38(8),10414–10419.

Yan, W. (2012). Toward automatic time-series forecasting using neural networks.IEEE Transactions on Neural Networks and Learning Systems, 23(7), 1028–1039.


http://sidc.oma.be/html/sunspot.html



http://dx.doi.org/10.1109/EAIS.2012.6232827

http://dx.doi.org/10.1109/EURCON.2007.4400689

http://dx.doi.org/10.1109/EURCON.2007.4400689

http://dx.doi.org/10.1109/CIDM.2007.368916

http://dx.doi.org/10.1109/APPEEC.2012.6307573

http://dx.doi.org/10.1109/APPEEC.2012.6307573

http://dx.doi.org/10.1109/IJCNN.2012.6252383

http://dx.doi.org/10.1109/TDEI.2013.6508749

http://www.nass.usda.gov/



http://dx.doi.org/10.1109/LAWP.2013.2251994

http://dx.doi.org/10.1109/LAWP.2013.2251994

http://www.standardandpoors.com

http://dx.doi.org/10.1016/j.compchemeng.2010.05.012

http://dx.doi.org/10.1016/j.compchemeng.2010.05.012


measurement of fitness function efficiency using data envelopment analysis

Documents