real-time price forecast with big data · pdf filereal-time price forecast with big data . ......

CERTS Review

REAL-TIME PRICE FORECAST WITH BIG DATA

A STATE SPACE APPROACH

Lang Tong (PI), Robert J. Thomas, Yuting Ji, and Jinsub Kim School of Electrical and Computer Engineering, Cornell University Jie Mei, Georgia Institute of Technology

August 7, 2013

CERTS Review

DATA QUALITY AND ITS EFFECTS ON MARKET OPERATIONS

DENY OF SERVICE ATTACK ON REAL-TIME ELECTRICITY MARKET COVER UP PROTECTION AGAINST TOPOLOGY ATTACK

Lang Tong (PI), Robert J. Thomas, and Jinsub Kim Cornell University

August 8, 2013

Project overview Objectives

Accurate short-term probabilistic forecasting of real-time LMP. Incorporate real-time measurements (e.g. SCADA/PMU). Scalable computation techniques.

Summary of results A real-time LMP model with forecasting and measurement

uncertainties. A Markov chain abstraction of real-time LMP computation. Monte Carlo sampling techniques. Preliminary simulations.

Outline

Motivation Load and real-time LMP as random processes Benchmark techniques Comments on the state of the art.

Probabilistic forecasting of real-time LMP Simulation studies Summary and future work

Sample paths of load and LMP

Day ahead vs. real-time

Presenter

Presentation Notes

Day ahead LMP is a poor prediction of real-time LMP

Benchmark techniques

Time series ARMA, ARIMA, ARMAX, GARCH

Machine learning Neural networks, support vector machines (SVM)

Hybrid techniques Jump (switching) models

Load forecasting Forecasting Method

MAPE

ANN [1] ANN [2]

1.46% 1.24%

SVM [3] 2.09%

[1] P. Mandal, T. Senjyu, N. Urasaki, and T. Funabashi, “A neural network based several-hour-ahead electric load forecasting using similar days approach,” Int. Journal. of Electric Power and Energy System, vol. 28, no. 6, pp. 367–373, July 2006. [2] P. Mandal, T. Senjyu, N. Urasaki, and T. Funabashi, “Short-Term Price Forecasting for Competitive Electricity Market,” Power Symposium, 2006. NAPS 2006. 38th North. [3] S. Fan, C. Mao and L. Chen, “Next-day electricity-price forecasting using a hybrid network,” IET Generation, Transmission & Distribution, Volume 1, Issue 1, January, 2007.

Day ahead LMP forecasting Forecasting Method

MAPE

ANN [1] ANN [2] ANN [4]

15.96% 12.92% 8.67%

SVM [3] 11.31%

GARCH [5] 11.55%

ARMAX [6] 9.01% [4] P. Mandal, T. Senjyu, N. Urasaki, and T. Funabashi, “A neural network based several-hour-ahead electric load forecasting using similar days approach,” Int. Journal. of Electric Power and Energy System, vol. 28, no. 6, pp. 367–373, July 2006. [5] R. Garcia, J. Contreras, “A GARCH Forecasting Model to Predict Day-Ahead Electricity Prices,” Int. Journal of Electric Power and Energy System, vol. 20, no. 2, MAY 2005. [6] J. Zhang, C. Yu, G. Hou, “Application of Chaotic Particle Swarm Optimization in the Short-term Electricity Price Forecasting,” Int. Journal of Electric Power and Energy System, vol. 28, no. 6, pp. 367 373 July 2006

Real-time LMP forecasting

Actual and forecast six-hours-ahead electricity price for Victorian electricity market (Nov 1st to 7th, 2003).

Actual and forecast six-hours-ahead electricity price for PJM market (Jan 8th to 14th, 2006).

[4] F. Sarafraz, H. Ghasemi, H. Monsef, “Locational Marginal Price Forecasting by Locally Linear Neuro Fuzzy Model ” EEEIC 2011 10th Rome

Summary of the state-of-the-art

Extensive literature: Mostly black box techniques Primarily providing point forecasting Rarely deal with on LMP and network effects

Extremely accurate load forecasting (1-3%) Relatively poor price forecasting (10-20%)

Outline Motivation Probabilistic forecasting of real-time LMP

A stylized real-time ex post LMP model Information structure LMP states and Markov chain representations Probabilistic forecast

Preliminary simulations Summary and future work

A stylized real-time ex post LMP model

State estimation

Ex ante dispatch

Ex post eligible generators

Ex post LMP

Ex post LMP via IDCOPF

Information structure and Markov chain

Probabilistic forecasting

Outline

Motivation Probabilistic forecasting of real-time LMP Preliminary simulations

IEEE 16 bus with NYISO sample load profiles Varying load errors and correlations Monte Carlo techniques to estimate transition

matrices

Summary and future work

Load and price distributions

0 119 200 28815 4220

22

24

26

28

30

32

34

36

38

time index (5 mins interval in 24 hours)

MW

Scaled average actual load profile (upper/lower bounds for predicted load profile) from NYISO June 1st

actual load profileupper boundlower bound

1 2 3 4 5 6 7 8 9 10110

0.5

1

Markov State (LMP) index

Pro

babi

lity

LMP distribution at time 42

1 2 3 4 5 6 7 8 9 10110

0.5

1


Pro

babi

lity


1 2 3 4 5 6 7 8 9 10110

0.5

1

Markov State (LMP) indexP

roba

bilit

y


1 2 3 4 5 6 7 8 9 10110

0.5

1


Pro

babi

lity


Presenter

Presentation Notes

- Actual load profile and upper/lower bounds for realizations - Marginal distribution at 4 time instants

LMP trajectories

0 50 100 150 200 25015

20

25

30

35

40

45

50


$/M

W

LMP trajectories

loadbus1bus2bus3bus4bus5bus6bus7bus8bus9bus10bus11bus12bus13bus14

Presenter

Presentation Notes

Load profile LMP trajectories bus by bus

Time inhomogeneous Markov Chain

Presenter

Presentation Notes

Dynamics of system: all markov state trajectories (for all 5000 training monte carlo runs) Each blue circle corresponds to a markov state Each link corresponds to a transition

Transition Matrices

Ending Markov State (LMP) indexSta

rting

Mar

kov

Sta

te (L

MP

) ind

ex Transition matrix at time 15

2 4 6 8 10 12

2

4

6

8

10

12 -4

-3

-2

-1


rting

Mar

kov

Sta

te (L

MP

) ind


2 4 6 8 10 12

2

4

6

8

10

12-3

-2

-1


rting

Mar

kov

Sta

te (L

MP

) ind

e Transition matrix at time 117

2 4 6 8 10 12

2

4

6

8

10

12 -8

-6

-4

-2

0


rting

Mar

kov

Sta

te (L

MP

) ind


2 4 6 8 10 12

2

4

6

8

10

12-3

-2

-1

0

Presenter

Presentation Notes

Transition matrices from training (computed by counting)

6 hours ahead prediction error

20 40 60 80 100 120 140 160 180 20010

-6

10-4

10-2

100


Pro

babi

lity

of in

corre

ct p

redi

ctio

n Prediction error for best and second best prediction

Best prediction

2nd best prediction

20 40 60 80 100 120 140 160 180 2000

2

4

6

8

10

time index (5 mins interval in 12 hours)n A

bsol

ute

Per

cent

age

Erro

r (M

AP

E

MAPE of best (most likely) prediction

Ending Markov State (LMP) index

Sta

rting

Mar

kov

Sta

te (L

MP

) ind

ex

Prediction transition matrix at time 21

2 4 6 8 10 12

2

4

6

8

10

12

-3

-2.5

-2

-1.5

-1

-0.5

0

Ending Markov State (LMP) index

Sta

rting

Mar

kov

Sta

te (L

MP

) ind

ex

Prediction transition matrix at time 21

2 4 6 8 10 12

2

4

6

8

10

12

-3

-2.5

-2

-1.5

-1

-0.5

0

Presenter

Presentation Notes

Plot above shows probability (in log scale) of incorrect price region prediction, blue is most likely, red is second most likely Plot below shows MAPE of most likely price. Transition matrix here is different from the previous one, this one is calculated from the product of 6 hours transition matrix One is interesting (at time 21), easily make mistake The other is boring case (at time 150), hardly make mistake (no uncertainness)

LMP prediction for different time horizon

1 2 3 4 5 6 7 8 9 10 11 12 13 140

10

20

30

40

50

Bus index

$/M

W

LMP prediction for 5 mins ahead

ActualBest prediction

2nd best

1 2 3 4 5 6 7 8 9 10 11 12 13 140

10

20

30

40

50

Bus index

$/M

W

LMP prediction for 1 hour ahead


2nd best

1 2 3 4 5 6 7 8 9 10 11 12 13 140

10

20

30

40

50LMP prediction for 3 hours ahead

Bus index

$/M

W


2nd best

1 2 3 4 5 6 7 8 9 10 11 12 13 140

10

20

30

40

50LMP prediction for 6 hours ahead

Bus index

$/M

W


2nd best

Presenter

Presentation Notes

Different time horizon prediction Show both most likely and second most likely prices

LMP distribution for different prediction horizons

1 2 3 4 5 6 7 8 9 10 11 12 1310

-4

10-2

100


Pro

babi

lity

(Log

)

LMP distribution for 5 mins ahead

1 2 3 4 5 6 7 8 9 10 11 12 1310

-4

10-2

100


Pro

babi

lity

(Log

)

LMP distribution for 1 hour ahead

1 2 3 4 5 6 7 8 9 10 11 12 1310

-4

10-2

100


Pro

babi

lity

(Log

)

LMP distribution for 3 hours ahead

1 2 3 4 5 6 7 8 9 10 11 12 1310

-4

10-2

100


Pro

babi

lity

(Log

)

LMP distribution for 6 hours ahead

Presenter

Presentation Notes

- Predicted lmp distribution (calculated from the product of transition matrices), different from original marginal lmp distribution

Planned research

Forecasting techniques Incorporating load models More sophisticated Monte Carlo techniques (important

sampling, MCMC) Generator behavior models Forecasting with renewable sources Demand response

Extensive simulation studies and comparisons

DATA QUALITY AND ITS EFFECTS ON MARKET OPERATIONS

DENY OF SERVICE ATTACK ON REAL-TIME ELECTRICITY MARKET

Lang Tong (PI), Robert J. Thomas, and Jinsub Kim Cornell University

Presented at CERTS

Review

August 7-8, 2013

Observability

A graph theoretic condition (KCD’80)

Undetectable attack (KJTT’11)

Only half of the meters are needed!

Bad data identification and removal

Framing attack via QCQP

DoS attack on real-time operations

Simulation examples (AC)

Conclusion

Attacks on cyber physical systems (power systems) are real: State attacks Topology attacks Dispatch attacks

A key step toward protection is deploy authentication mechanisms.

real-time price forecast with big data · pdf filereal-time price forecast with big data . ......

Documents