# electricity price forecasting with recurrent neural networks

Post on 21-Apr-2017

12.099 views

Embed Size (px)

TRANSCRIPT

PowerPoint

Taegyun Jeon

TensorFlow-KR / 2016.06.18Gwangju Institute of Science and Technology

Electricity Price Forecastingwith Recurrent Neural Networks

RNN TensorFlow-KR Advanced Track

1

Who is a speaker?Taegyun Jeon (GIST)Research Scientist in Machine Learning and Biomedical Engineering

tgjeon@gist.ac.kr

linkedin.com/in/tgjeon

tgjeon.github.io

Page 2 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksGithub for this tutorial: https://github.com/tgjeon/TensorFlow-Tutorials-for-Time-Series

2

What you will learn about RNNHow to:Build a prediction modelEasy case study: sine functionPractical case study: electricity price forecasting

Manipulate time series dataFor RNN models

Run and evaluate graph

Predict using RNN as regressorPage 3 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

3

ContentsOverview of TensorFlow

Recurrent Neural Networks (RNN)

RNN Implementation

Case studiesCase study #1: sine functionCase study #2: electricity price forecasting

Conclusions

Q & A

Page 4 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

4

ContentsOverview of TensorFlow

Recurrent Neural Networks (RNN)

RNN Implementation

Case studiesCase study #1: sine functionCase study #2: electricity price forecasting

Conclusions

Q & A

Page 5 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

5

TensorFlowOpen Source Software Library for Machine Intelligence[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksPage 6

6

PrerequisiteSoftwareTensorFlow (r0.9) Python (3.4.4)Numpy (1.11.0)Pandas (0.16.2)

TutorialsRecurrent Neural Networks, TensorFlow TutorialsSequence-to-Sequence Models, TensorFlow Tutorials

Blog PostsUnderstanding LSTM Networks (Chris Olah @ colah.github.io)Introduction to Recurrent Networks in TensorFlow (Danijar Hafner @ danijar.com)

BookDeep Learning, I. Goodfellow, Y. Bengio, and A. Courville, MIT Press, 2016Page 7 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

7

ContentsOverview of TensorFlow

Recurrent Neural Networks (RNN)

RNN Implementation

Case studiesCase study #1: sine functionCase study #2: electricity price forecasting

Conclusions

Q & A

Page 8 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

8

Recurrent Neural NetworksNeural Networks

Inputs and outputs are independent Page 9 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksRecurrent Neural Networks

Sequential inputs and outputs......

9

Recurrent Neural Networks (RNN)

Page 10 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksImage from WILDML.com: RECURRENT NEURAL NETWORKS TUTORIAL, PART 1 INTRODUCTION TO RNNS

10

Overall procedure: RNN

Page 11 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks[1] X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks (2010)

11

Overall procedure: RNN

Page 12 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

12

Overall procedure: RNN

Page 13 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

13

Overall procedure: RNN

Page 14 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

14

Overall procedure: RNNInitializationForward PropagationCalculating the lossStochastic Gradient Descent (SGD)Backpropagation Through Time (BPTT)Long-term dependencies vanishing/exploding gradient problem

Page 15 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

15

Vanishing gradient over timeConventional RNN with sigmoidThe sensitivity of the input valuesdecays over timeThe network forgets the previous input

Long-Short Term Memory (LSTM) [2]The cell remember the input as long as it wantsThe output can be used anytime it wants

[2] A. Graves. Supervised Sequence Labelling with Recurrent Neural Networks (2012)Page 16 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

16

Design Patterns for RNNRNN Sequences

Page 17 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksBlog post by A. Karpathy. The Unreasonable Effectiveness of Recurrent Neural Networks (2015)TaskInputOutputImage classificationfixed-sized imagefixed-sized classImage captioningimage input sentence of wordsSentiment analysissentencepositive or negative sentimentMachine translationsentence in English sentence in FrenchVideo classificationvideo sequence label each frame

17

Design Pattern for Time Series Prediction

Page 18 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

RNN

DNN

Linear Regression

18

ContentsOverview of TensorFlow

Recurrent Neural Networks (RNN)

RNN Implementation

Case studiesCase study #1: sine functionCase study #2: electricity price forecasting

Conclusions

Q & A

Page 19 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

19

RNN Implementation using TensorFlowHow we design RNN model for time series prediction?

[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksPage 20

How manipulate our time series data as input of RNN?

20

Regression models in Scikit-Learn

[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksPage 21

X = np.atleast_2d([0., 1., 2., 3., 5., 6., 7., 8., 9.5]).Ty = (X*np.sin(x)).ravel()

x = np.atleast_2d(np.linspace(0, 10, 1000)).T

gp = GaussianProcess(corr='cubic', theta0=1e-2, thetaL=1e-4, thetaU=1e-1, random_start=100)

gp.fit(X, y)y_pred, MSE = gp.predict(x, eval_MSE=True)

21

RNN ImplementationRecurrent StatesChoose RNN cell typeUse multiple RNN cells

Input layerPrepare time series data as RNN input Data splittingConnect input and recurrent layers

Output layerAdd DNN layerAdd regression model

Create RNN model for regressionTrain & Prediction

[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksPage 22

22

1) Choose the RNN cell typeNeural Network RNN Cells (tf.nn.rnn_cell)BasicRNNCell (tf.nn.rnn_cell.BasicRNNCell)activation : tanh()num_units : The number of units in the RNN cell

BasicLSTMCell (tf.nn.rnn_cell.BasicLSTMCell)The implementation is based on RNN Regularization[3] activation : tanh()state_is_tuple : 2-tuples of the accepted and returned states

GRUCell (tf.nn.rnn_cell.GRUCell)Gated Recurrent Unit cell[4]activation : tanh()

LSTMCell (tf.nn.rnn_cell.LSTMCell)use_peepholes (bool) : diagonal/peephole connections[5]. cell_clip (float) : the cell state is clipped by this value prior to the cell output activation.num_proj (int): The output dimensionality for the projection matrices

Page 23 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks[3] W. Zaremba, L. Sutskever, and O. Vinyals, Recurrent Neural Network Regularization (2014)[4] K. Cho et al., Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation (2014)[5] H. Sak et al., Long short-term memory recurrent neural network architectures for large scale acoustic modeling (2014)

23

LAB-1) Choose the RNN Cell type[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksPage 24 Import tensorflow as tf

rnn_cell = tf.nn.rnn_cell.BasicRNNCell(num_units)rnn_cell = tf.nn.rnn_cell.BasicLSTMCell(num_units)rnn_cell = tf.nn.rnn_cell.GRUCell(num_units)rnn_cell = tf.nn.rnn_cell.LSTMCell(num_units)

BasicRNNCellBasicLSTMCellGRUCellLSTMCell

24

2) Use the multiple RNN cellsRNN Cell wrapper (tf.nn.rnn_cell.MultiRNNCell)Create a RNN cell composed sequentially of a number of RNN Cells.

RNN Dropout (tf.nn.rnn_cell.Dropoutwrapper)Add dropout to inputs and outputs of the given cell.

RNN Embedding wrapper (tf.nn.rnn_cell.EmbeddingWrapper)Add input embedding to the given cell.Ex) word2vec, GloVe

RNN Input Projection wrapper (tf.nn.rnn_cell.InputProjectionWrapper)Add input projection to the given cell.

RNN Output Projection wrapper (tf.nn.rnn_cell.OutputProjectionWrapper)Add output projection to the given cell.

Page 25 [TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural Networks

25

LAB-2) Use the multiple RNN cells[TensorFlow-KR Advanced Track] Electricity Price Forecasting with Recurrent Neural NetworksPage 26 rnn_cell = tf.nn.rnn_cell.DropoutWrapper(rnn_cell, input_keep_prob=0.8, output_keep_prob=0.8)

GRU/LSTMInput_keep_prob=0.8output_keep_prob=0.8GRU/LSTMGRU/LSTMGRU/LSTMStacked_lstm = tf.nn.rnn_cell.MultiRNNCell([rnn_cell] * depth)

depth

26

3) Prepare the time series data Split raw data into train, validation, and test datasetsplit_data [6

Recommended