forecasting non-stationary time series without recurrent

Forecasting Non-Stationary Time Series withoutRecurrent Connections

AP Engelbrecht

Department of Industrial Enigneering, andComputer Science Division

Stellenbosch UniversitySouth Africa

[email protected]

Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 1 / 30

Presentation Outline I

1 Introduction

2 The Time Series Used

3 Recurrent Neural Networks

4 Dynamic Optimization Problems

5 Particle Swarm Optimization

6 PSO Training of NNs

7 Empirical Analysis

8 Conclusions


Introduction

The main goal of this study was to investigate if recurrent connectionsor time delays are necessary when training neural networks (NNs) fornon-stationary time series prediction using a dynamic particle swarmoptimization (PSO) algorithm

Consider training of the NN as a dynamic optimization problem, due tothe statistical properties of the time series changing over time

The quantum-inspired PSO (QSO) is a dynamic PSO with the ability totrack optima in changing landscapes


The Time SeriesPlots

International Airline Passengers

(AIP)Australian Wine Sales (AWS)


The Time SeriesPlots (cont)

US Accidental Death (USD) Sunspot Annual Measure (SAM)



Hourly Internet Traffic (HIT)Daily Minimum Temperature

(DMT)



Mackey Glass (MG) Logistic Map (LM)


Feedforward Neural Networks


Recurrent Neural Networks


Dynamic Optimization Problems

Training of a NN is an optimization problem, with the objective to findbest values for weights and biases such that a given error function isminimized

Forecasting a non-stationary time series is a dynamic optimizationprocess, due to the statistical properties of the time series changingover time

Dynamic optimization problems:search landscape properties change over timeoptima change over time, in value and in positionnew optima may appearexisting optima may disappearchanges further characterized by change severity and changefrequency


Dynamic Optimization Problems (cont)

Implications Optimization Algorithms:Need to adjust values assigned to decision variables in order totrack changing optima, without re-optimizingFor NN training, need to adapt weight and bias values to cope withconcept drift, without re-trainingShould have the ability to escape local minimaNeed to continually inject diversity into the search


Particle Swarm OptimizationIntroduction

What is particle swarm optimization (PSO)?a simple, computationally efficient optimization methodpopulation-based, stochastic searchindividuals follow very simple behaviors:

emulate the success of neighboring individuals,but also bias towards own experience of success

emergent behavior: discovery of optimal regions within a highdimensional search space


Particle Swarm OptimizationMain Components

What are the main components?a swarm of particleseach particle represents a candidate solutionelements of a particle represent parameters to be optimized

The search process:Position updates

xi(t + 1) = xi(t) + vi(t + 1), xij(0) ⇠ U(xmin,j , xmax ,j)

Velocity (step size)drives the optimization processreflects experiential knowledge of the particles andsocially exchanged information about promisingareas in the search space


Particle Swarm OptimizationInertia Weight PSO

used either the star (gbest PSO) or social (lbest PSO) topologyvelocity update per dimension:

vij(t + 1) = wvij(t) + c1r1j(t)[yij(t)� xij(t)] + c2r2j(t)[yij(t)� xij(t)]

vij(0) = 0w is the inertia weightc1, c2 are positive acceleration coefficientsr1j(t), r2j(t) ⇠ U(0, 1)note that a random number is sampled for each dimension


Particle Swarm OptimizationPSO Algorithm

Create and initialize an nx -dimensional swarm, S;repeat

for each particle i = 1, . . . ,S.ns do

if f (S.xi) < f (S.yi) then

S.yi = S.xi ;end

for each particle i with particle i in its neighborhood do

if f (S.yi) < f (S.yi) then

S.yi = S.yi ;end

end

end

for each particle i = 1, . . . ,S.ns do

update the velocity and position;end

until stopping condition is true;Engelbrecht (Stellenbosch University) Non-Stationary Time Series Forecasting 3 May 2019 15 / 30

Particle Swarm OptimizationQuantum-Inspired PSO (QSO)

Developed to find and track an optimum in changing searchlandscapesBased on quantum model of an atom, where orbiting electrons arereplaced by a quantum cloud which is a probability distributiongoverning the position of each electronSwarm contains

neutral particles following standard PSO updatescharged, or quantum particles, randomly placed within amulti-dimensional sphere

xi(t + 1) =⇢

xi(t) + vi(t + 1) if Qi = 0By(rcloud ) if Qi 6= 0


Particle Swarm OptimizationCooperative PSO

For large-scale optimization problems, a divide-and-conquer approachto address the curse of dimensionality:

Each particle is split into K separate parts of smaller dimensionEach part is then optimized using a separate sub-swarmIf K = nx , each dimension is optimized by a separate sub-swarm

Cooperative quantum PSO (CQSO) uses QSO in the sub-swarms


PSO Training of NNs

When using PSO to train a NN:each particle represents the weights and biases of one NNobjective function is a cost function, e.g. SSEto prevent hidden unit saturation, use ReLUany activation function in the output units

For non-stationary time series prediction:Used cooperative PSO with QSO in sub-swarmsRNNs used modified hyperbolic tangent:f (net) = 1.7159 tanh(1

3net)


Control Parameters


Dynamic Scenarios


Performance Measure

Used the collective mean error,

Fmean(t) =PT

t=1 F (t)T

where F (t) is the MSE at time t

Number of independent runs: 30


ResultsMG (Mackey Glass)


ResultsHIT (Hourly Internet Traffic)


ResultsDMT (Daily Minimum Temperature)


ResultsSAM (Sunspot Annual Measure)


ResultsLM (Logistic Map)


ResultsAWS (Australian Wine Sales)


ResultsAIP (International Airline Passengers)


ResultsUSD (US Accidental Death)


Conclusions

The aim of the study was to investigate if recurrent connections ordelays are necessary if a dynamic PSO is used to train a NN for timeseries prediction

Main observation:A FFNN trained with the cooperative quantum PSO performedbetter than the RNNs used for most problems and scenariosWhere the CQSO FFNN algorithm did not perform best,differences in performance were not statistically significant

Future work will:expand the study to other variants of recurrent NNs, and moretime seriesdevelop dynamic architecture optimization approaches


forecasting non-stationary time series without recurrent

Documents