Computational Intelligenceand Energy Systems: intelligent solutions for

complex problems

Matteo De FeliceUnità Modellistica Energetica Ambientale

UTMEA - ENEA

1Tuesday, May 31, 2011

Sommario

Cos’è la Computational Intelligence (CI)?

Quali sono le applicazioni della CI ai sistemi complessi?

2Tuesday, May 31, 2011

CI: paradigmi

NN

EC FS

SI AIS

Soft Computing

Computational Intelligence

IA?

3Tuesday, May 31, 2011

Visione d’insieme

NN

EC FS

SI AIS

Temi principali

4Tuesday, May 31, 2011

CI e letteratura

1994 1996 1998 2000 2002 2004 2006 2008 20100

1

2

3

4

5x 10 3

year

Evolutionary ComputationSwarm IntelligenceArtificial Neural Networks

Dati dalla Thomson Reuters ISI considerando Computer Science & Technology (Gennaio 2010)

Due journals sulla CI nei primi 10 in CS (IF 2009)

5Tuesday, May 31, 2011

La diffusione della CI

Problemi sempre più complessi

Più potenza di calcolo disponibile

6Tuesday, May 31, 2011

ma...Assenza di una teoria consolidata

Frammentazione degli algoritmi

Approccio poco sistematico e confronti poco “robusti”

PSO APSO CPSO DPSO EPSO FPSO GPSO HPSO IPSO LPSO MPSO NPSO OPSO PPSO QPSO RPSO SPSO TPSO UPSO VPSO WPSO GA AGA BGA CGA DGA EGA FGA HGA IGA KGA LGA MGA OGA PGA QGA RGA SGA VGA ...

7Tuesday, May 31, 2011

Applicazioni Principali

1) Modellazione & Forecasting

2) Ottimizzazione

CalcoloEvolutivo

Reti Neurali

& Logica Fuzzy

8Tuesday, May 31, 2011

Quadro generale

Reti Neurali Evolutive

Ensemble

Reti neurali evolutive con topologia a rete complessa

Artificial Neural Networks and Support Vector Machines ensembling: a comparison

2008Evolving predictive neural models for complex processes

Evolving Complex Neural Networks

2009 9Tuesday, May 31, 2011

2009Modellazione temperature con NN

Ambient temperature modelling with soft computing techniques

Combining Back-Propagation and Genetic Algorithms to Train Neural Networks for Ambient Temperature Modeling in Italy

2010Ottimizzazione dello start-upcentrale a ciclo combinato

Combining Back-Propagation and Genetic Algorithms to Train Neural Networks for Ambient Temperature Modeling in Italy

10Tuesday, May 31, 2011

2011Reti Neurali e Load Forecast

Climate Variables in Energy Modeling

Short-Term Load Forecasting with Neural Network Ensembles: a Comparative Study

11Tuesday, May 31, 2011

Altri Progetti

IdentificazioneStructural System

in Ing. Sismica

Reti Neurali Evolutive e

Applicazioni alla Finanza

Algoritmi Evolutivi Spazialmente

Strutturati

Ottimizzazione traiettorie missioni

interplanetarie

12Tuesday, May 31, 2011

Ottimizzazione13Tuesday, May 31, 2011

Process Optimization

Come migliorare la ‘performance’ di un processo tramite i suoi parametri?

ProcessProcess

Parameters (X)

Environment

Measurement

14Tuesday, May 31, 2011

Ottimizzazione tradizionale

Metodi Line-search and trust-region (serve l’Hessiano!)

Metodi Quasi-newton (Hessiano approssimato)

Metodi Derivative-free

15Tuesday, May 31, 2011

...ma il real-world è:

1) ‘Rumoroso’

2) Dinamico

3) Difficile da esaminare

16Tuesday, May 31, 2011

Evolutionary Computation (EC)

Ottimizzazione Black-box

Singolo e Multi-Obiettivo

Anche funzioni discontinue e non differenziabili

Meta-euristica Population-based

17Tuesday, May 31, 2011

Metaeuristica

Ottimizzazione Stocastica

Algoritmi usati per trovare soluzioni a problemi “difficili”

Esempio: Hill-Climbing, Tabu Search, Simulated-Annealing

18Tuesday, May 31, 2011

Real-World problems

19Tuesday, May 31, 2011

Metodi di ottimizzazioneLipschitzian Optimization

DIRECT AlgorithmApplications

Taxonomy of Methods

Yves Brise Lipschitzian Optimization, DIRECT Algorithm, and Applications20Tuesday, May 31, 2011

ApplicazioneOttimizzazione dello startup di una centrale a ciclo combinato (CCPP)

Minimizzazione del tempo di avvio, consumi, emissioni e stress termico

Massimizzazione della produzione di energiaM. De Felice, I. Bertini, A. Pannicelli, and S. Pizzuti, "Soft Computing based optimisation of combined cycled power plant start-up operation with fitness approximation methods," Applied Soft Computing, 2011.

I. Bertini, M. De Felice, F. Moretti, and S. Pizzuti, "Start-Up Optimisation of a Combined Cycle Power Plant with Multiobjective Evolutionary Algorithms," in Applications of Evolutionary Computation, 2010, pp. 151-160.

21Tuesday, May 31, 2011

Procedura1. Definizione di un indice di

performance

2. Impostazione simulatore sw

3. Algoritmo EC tramite simulatore

22Tuesday, May 31, 2011

Indice Performance

Informazioni dagli esperti di processo

Knowledge modeling con funzioni fuzzy

0 0.5 1 1.5 2 2.5 3x 104

0

0.5

1

F1

0 0.5 1 1.5 2 2.5 3x 105

0

0.5

1

F2

0 5 10 15x 109

0

0.5

1

F3

0 5 10 15 20 25 300

0.5

1

F4

0 50 100 150 200 250 3000

0.5

1

F5

23Tuesday, May 31, 2011

Singolo-obiettivo

Algoritmo Genetico

operazione di mutazione Gaussiano

Funzione di fitness approssimata per velocizzare l’ottimizzazione(da 2070 a 36 ore/CPU)

24Tuesday, May 31, 2011

Risultati

Tempo avvio Consumi Prod.

Energia Emissioni Stress Termico

Esperti 21070 143557 2.5•109 25 10

GA 16569 115070 1.86•109 18.8 78.4

Var. Norm. -25% -16% -16% -30% 2%

25Tuesday, May 31, 2011

Multi-obiettivo

3.9 4 4.1 4.2 4.3 4.4 4.5 4.6x 109

12.2

12.25

12.3

12.35

12.4

12.45

12.5

12.55

12.6

12.65

Energy Production (KJ)

Emis

sion

s (m

g s

/ N m

3 )

RealNSGA 2WSGARAND

26Tuesday, May 31, 2011

Modellazione & Forecasting

27Tuesday, May 31, 2011

Modellazione con NNs||F (x)− f(x)|| < �, ∀x

6.50 1 2 3 4 5 6

1.2

-1.2

-0.6

0

0.6

X Axis

Y A

xis

y = sin(x)NN(x)

28Tuesday, May 31, 2011

Modellazione con NNs

Errore (MSE)

Metodi empirici per decidere la topologia della rete

System

NeuralNetwork

Input u(k) Output y(k)

Disturbances

29Tuesday, May 31, 2011

Regressione con NN

Si una una NN per fare regressione non-lineare

30Tuesday, May 31, 2011

Time Series Forecasting

Possiamo fare una previsione dei dati futuri usando quelli osservati

Altre informazioni utili (!)

31Tuesday, May 31, 2011

Approcci per le NN

NeuralNetwork

Input attime t

y(t+1)y(t+2)

...

y(t+N)

Direct Method

NeuralNetwork

Input attime t output t+1

delay

output tIterative Method

NeuralNetwork

Input attime t output t+1

delay

output tIterative Method

32Tuesday, May 31, 2011

Short-Term Load Forecasting

Dati Orari

Obiettivo: predizione del carico fino a 24 ore

0 200 400 600 800 1000 1200 1400 1600 1800 20000

20

40

60

hours

kW

33Tuesday, May 31, 2011

Modelli Seasonal

ΦP (Bs)φ(B)∇D

s ∇dxt = α+ΘQ(Bs)θ(B)et

0 10 20 30 40 500.5

0

0.5

1

Implementazione in R

34Tuesday, May 31, 2011

Modello NN

Campioni passati

Informazioni aggiuntive

Rete Neurale

Previsione

35Tuesday, May 31, 2011

Rete Neurale

Pesi wi Pesi wo

Funzioni di attivazione f differenziabili

36Tuesday, May 31, 2011

Backpropagation

[Werbos, 1974]

Forward phase: il segnale si propaga “in avanti”

Backward phase: si calcola l’errore e lo si propaga “all’indietro”, modificando i pesi

37Tuesday, May 31, 2011

Modello NN

240 2 4 6 8 10 12 14 16 18 20 22

36

10

15

20

25

30

X Axis

Y A

xis

y(k+1)

y(k)

y(k-1)

Come scegliere i lags?

38Tuesday, May 31, 2011

Data Analysis1. ACF

2. Distribution

3. Multivariate analysis

0 10 20 30 40 500.5

0

0.5

1

hourkW

1 5 9 13 17 21 24

5

10

15

20

25

30

35

40

45

50

0

0.05

0.1

0.15

0.2

0.25

0 20 40 60 80 1000

10

20

30

40

50

60

occupancy

load

(kW

)

y = 0.0013*x2 + 0.26*x + 12

39Tuesday, May 31, 2011

Domanda...

Come ridurre la varianza delle reti neurali?

40Tuesday, May 31, 2011

Ensembling

41Tuesday, May 31, 2011

Ensembling

1. Calibrazione del modello usando sottoinsiemi dei dati (Bagging)

2. Uso dei dati pesato per importanza (Adaboosting)

3. Interazione e cooperazione tra gli stimatori

42Tuesday, May 31, 2011

Ensembling

[Hansen & Salomon, 1990]

Majority voting (classificazione)

Combinazione lineare (regressione)

F (x,D) =1

N

N�

i=1

Fi(x,D)

43Tuesday, May 31, 2011

Ensembling

Media

44Tuesday, May 31, 2011

ApplicazioniSTLF dell’edificio ENEA Casaccia (C59)

Presentato al IEEE Symposium on CI Applications in Smart Grid

M. De Felice and X. Yao, "Neural Networks Ensembles for Short-Term Load Forecasting," in IEEE Symposium Series in Computational Intelligence 2011 (SSCI 2011), 2011

45Tuesday, May 31, 2011

Tecniche

Predittore naive:

modello SARIMA (Seasonal ARIMA):

Reti Neurali (NN)

NN Ensembles

ΦP (Bs)φ(B)∇D

s ∇dxt = α+ΘQ(Bs)θ(B)et

46Tuesday, May 31, 2011

Dati misurati da Settembre a Novembre 2009

Training (13 settimane) e testing (una settimana divisa in T1 e T2)

20582010 2013 2016 2019 2022 2025 2028 2031 2034 2037 2040 2043 2046 2049 2052 2055

40

10

15

20

25

30

35

hours

kW

24 hours

training part

Metodologia

47Tuesday, May 31, 2011

Misure d’errore

Errore Assoluto (MAE e MSE)

Error Percentuale (MAPE)

Scaled Error (MASE)

48Tuesday, May 31, 2011

Negative Correlation Learning

[Liu & Yao, 1999]

Modifica alla funzione di backpropagation

Penalty term λ

ei =M�

n=1

(Fi(xn)− yn)2 + λpi

49Tuesday, May 31, 2011

Regularized NCL

[Chen & Yao, 2009]

NCL con Regolarizzazione

ei =1

N

M�

n=1

(Fi(xn)− yn)2 − 1

N

M�

n=1

(Fi(x)− F (xn))2+

+αiwTi wi

50Tuesday, May 31, 2011

ErroriMAE MSE

NN (Media) 2.34 (0.79)2.49 (1.47)

10.9 (17.88)21.67 (59.29)

NN Ensemble 1.381.09

2.952.4

RNCL 1.471.07

3.342.82

Naive 2.112.28

7.616.4

SARIMA 1.891.24

5.522.17

51Tuesday, May 31, 2011

Dati AggiuntiviInformazioni aggiuntive: occupanti edificio, ora del giorno, giorno della settimana, giorni lavorativi.

NN: input aggiuntivi

SARIMA: termine lineare addizionale

52Tuesday, May 31, 2011

Dati Aggiuntivi

0 20 40 60 80 100 120 1400

1

2

3

4

Forecasting window

Abso

lute

erro

r

SARIMA external dataSARIMA

0 20 40 60 80 100 120 1400

1

2

3

4

forecast window

abso

lute

erro

r

MLP Ensemble external dataMLP Ensemble

0 20 40 60 80 100 120 1400

1

2

3

4

forecast window

abso

lute

erro

r

53Tuesday, May 31, 2011

Errori – dati aggiuntiviMAE MSE

NN (Media) 2.46 (0.83)2.34 (1.00)

12.13 (16.80)11.61 (10.61)

NN Ensemble 1.420.75

3.301.27

RNCL 1.330.92

2.71.62

Naive 2.112.28

7.616.4

SARIMA 1.911.20

5.612.07

54Tuesday, May 31, 2011

Errori giornalieri

5 10 15 200

1

2

3

(a) SARIMA T1

5 10 15 200

1

2

3

(b) MLP Ensembling T1

5 10 15 200

1

2

3

(c) RNCL T1

5 10 15 200

1

2

3

(d) SARIMA T2

5 10 15 200

1

2

3

(e) MLP Ensembling T2

5 10 15 200

1

2

3

(f) RNCL T2

Fig. 6. Univariate approach: 24-hours ahead forecasting absolute errors on both T1 and T2. In light grey the area between the 1st and the 3rd quartiles.

hour of the day

1 5 9 13 17 21 24

20

40

60

80

100

120

140

1

2

3

4

5

6

7

8

(a) SARIMA

hour of the day

1 5 9 13 17 21 24

20

40

60

80

100

120

140

1

2

3

4

5

6

7

8

(b) MLP Ensembling

hour of the day

1 5 9 13 17 21 24

20

40

60

80

100

120

140

1

2

3

4

5

6

7

8

(c) RNCL

Fig. 7. Absolute errors (in kW) made during testing parts T1 and T2 for the univariate approach models. On the Y axis there are the various forecastingwindows and on the X axis the hour of the day of each of the 24 prediction errors. Note that the color scale is not the same in each plot.

TABLE IIMODELS PERFORMANCES: APPROACH WITH OCCUPANCY, HOUR OF THE DAY AND WORKDAY FLAG (ALL THE ERROR VALUES ARE ROUNDED TO TWO

DECIMALS). IN BRACKETS THE STANDARD DEVIATION WHERE NEEDED AND IN BOLD THE BEST MODEL ERROR FOR THE TESTING PART.

Training Testing T1 Testing T2Model MAE MSE MAE MSE Max MAE MSE Max

Naive (no ext. data) 2.45 14.97 2.11 7.61 7.35 2.28 6.4 6.36SARIMA 1.13 4.31 1.91 5.61 8.00 1.20 2.07 5.18

ANN MLP best training 0.36 0.70 3.51 20.28 18.00 2.20 11.83 24.53Average ANN MLP 1.20 (0.31) 3.25 (1.52) 2.46 (0.83) 12.13 (16.80) 13.84 (16.62) 2.34 (1.00) 11.61 (10.61) 13.00 (6.01)MLP Ensemble 0.74 1.47 1.42 3.30 7.98 0.75 1.27 4.79

ANN RBF best training 1.06 2.48 1.36 3.03 6.43 0.88 1.61 7.05Average ANN RBF 1.65 (0.86) 7.71 (10.61) 1.97 (1.01) 7.99 (13.19) 8.22 (5.71) 1.77 (1.39) 8.98 (21.24) 10.74 (6.18)

RNCL 1.15 3.35 1.33 2.71 5.37 0.92 1.62 4.52

information is provided in Figure 9, where, after a comparisonwith Figure 7, we can see in which part of the datasetadditional data has reduced the error.

VII. DISCUSSION

In Figure 10 testing absolute errors are shown, arranged inascending order, for all the MLP and RBF networks on testing

set T1 (we omitted T2 for sake of clearness). It’s evident howneural network ensembles exhibits an error lower or at leastequal than the best network, both for MLP and RBFs. Thismeans that ensembling allows, thank to the exploitation ofall the information ‘contained’ within the trained networks,to achieve an effective forecasting overcoming the drawbacksof neural networks: overfitting and the high variability of the

55Tuesday, May 31, 2011

Errori giornalieri

5 10 15 200

1

2

3

(a) SARIMA T1

5 10 15 200

1

2

3

(b) MLP Ensembling T1

5 10 15 200

1

2

3

(c) RNCL T1

5 10 15 200

1

2

3

(d) SARIMA T2

5 10 15 200

1

2

3

(e) MLP Ensembling T2

5 10 15 200

1

2

3

(f) RNCL T2

Fig. 6. Univariate approach: 24-hours ahead forecasting absolute errors on both T1 and T2. In light grey the area between the 1st and the 3rd quartiles.

hour of the day

1 5 9 13 17 21 24

20

40

60

80

100

120

140

1

2

3

4

5

6

7

8

(a) SARIMA

hour of the day

1 5 9 13 17 21 24

20

40

60

80

100

120

140

1

2

3

4

5

6

7

8

(b) MLP Ensembling

hour of the day

1 5 9 13 17 21 24

20

40

60

80

100

120

140

1

2

3

4

5

6

7

8

(c) RNCL

Fig. 7. Absolute errors (in kW) made during testing parts T1 and T2 for the univariate approach models. On the Y axis there are the various forecastingwindows and on the X axis the hour of the day of each of the 24 prediction errors. Note that the color scale is not the same in each plot.

TABLE IIMODELS PERFORMANCES: APPROACH WITH OCCUPANCY, HOUR OF THE DAY AND WORKDAY FLAG (ALL THE ERROR VALUES ARE ROUNDED TO TWO

DECIMALS). IN BRACKETS THE STANDARD DEVIATION WHERE NEEDED AND IN BOLD THE BEST MODEL ERROR FOR THE TESTING PART.

Training Testing T1 Testing T2Model MAE MSE MAE MSE Max MAE MSE Max

Naive (no ext. data) 2.45 14.97 2.11 7.61 7.35 2.28 6.4 6.36SARIMA 1.13 4.31 1.91 5.61 8.00 1.20 2.07 5.18

ANN MLP best training 0.36 0.70 3.51 20.28 18.00 2.20 11.83 24.53Average ANN MLP 1.20 (0.31) 3.25 (1.52) 2.46 (0.83) 12.13 (16.80) 13.84 (16.62) 2.34 (1.00) 11.61 (10.61) 13.00 (6.01)MLP Ensemble 0.74 1.47 1.42 3.30 7.98 0.75 1.27 4.79

ANN RBF best training 1.06 2.48 1.36 3.03 6.43 0.88 1.61 7.05Average ANN RBF 1.65 (0.86) 7.71 (10.61) 1.97 (1.01) 7.99 (13.19) 8.22 (5.71) 1.77 (1.39) 8.98 (21.24) 10.74 (6.18)

RNCL 1.15 3.35 1.33 2.71 5.37 0.92 1.62 4.52

information is provided in Figure 9, where, after a comparisonwith Figure 7, we can see in which part of the datasetadditional data has reduced the error.

VII. DISCUSSION

In Figure 10 testing absolute errors are shown, arranged inascending order, for all the MLP and RBF networks on testing

set T1 (we omitted T2 for sake of clearness). It’s evident howneural network ensembles exhibits an error lower or at leastequal than the best network, both for MLP and RBFs. Thismeans that ensembling allows, thank to the exploitation ofall the information ‘contained’ within the trained networks,to achieve an effective forecasting overcoming the drawbacksof neural networks: overfitting and the high variability of the

56Tuesday, May 31, 2011

Ensemble: altro esempio

60 80 100 120 140 160 180 200 220

20

40

60

80

100

testing hours

kW

57Tuesday, May 31, 2011

TO-DO

Ensemble: usare tutte le stime per creare una pdf

Ibridizzazione con metodi statistici classici: analisi multivariate, modelli stagionali, Holt-Winters

58Tuesday, May 31, 2011

The Big View

Forecasting & Modeling

59Tuesday, May 31, 2011

Passi principali

1. Definizione target (short-term, medium-term, seasonal)

2. Raccolta dati e analisi

3. Definizione e comparazione tecniche

4. Valutazione

5. Simulazione

Statistical Analysys

High-dimensionalityData Mining

Time Series MethodsNNsHybrid Methods

Cost AnalysisPerformance Measures

Software Simulator

Multi-Agent Systems

60Tuesday, May 31, 2011

PPSN 2012

12th International Conference on “Parallel Problem Solving From Nature”, Taormina

Paper submission: 15 Marzo 2012 (Proceedings Springer)

http://www.dmi.unict.it/ppsn2012/

61Tuesday, May 31, 2011

http://matteodefelice.name/research

62Tuesday, May 31, 2011