jean-baptistefilippi1 vivienmallet23 bahaanader1 · modelevaluationbasedonalargeobservationdataset...
TRANSCRIPT
Model evaluation based on a large observation data set
Jean-Baptiste Filippi1 Vivien Mallet2,3 Bahaa Nader1
1SPE CNRS
2INRIA
3CEREA, joint laboratory ENPC - EDF R&D, Université Paris-Est
Numerical simulation of forest fires, Cargèse, May 2013
Filippi, Mallet, Nader Model evaluation May 2013 1 / 50
Topic
Evaluation of the performance of a propagation model
Questions1 How to rank models? Can we identify the “best” model out of a pool
of models?2 How to evaluate the dynamics of the model when the observation is
the final burned surface?3 Can we evaluate a model regardless of the quality of its inputs?4 Can we carry out probabilistic forecasts?
Filippi, Mallet, Nader Model evaluation May 2013 2 / 50
Notation
ObservationObservation of final burnedsurface at time to
fObserved burned surface:So(to
f )
SimulationFinal simulation time: ts
fSimulated burned surface attime t: S(t)
Area|S| is the area of the surface SΩ is the simulation domain
ScoresClassical scores compare So(to
f ) and S(tsf )
Filippi, Mallet, Nader Model evaluation May 2013 3 / 50
Classical scores
Sørensen similarity index2|So(to
f ) ∩ S(tsf )|
|So(tof )|+ |S(ts
f )|∈ [0, 1]
Jaccard similarity coefficient|So(to
f ) ∩ S(tsf )|
|So(tof ) ∪ S(ts
f )|∈ [0, 1]
Kappa coefficientPa − Pe1− Pe
≤ 1
wherePa =
|So(tof ) ∩ S(ts
f )||Ω|
+|Ω\(So(to
f ) ∪ S(tsf ))|
|Ω|
Pe =|So(to
f )||S(tsf )|
|Ω|2+|Ω\So(to
f )||Ω\S(tsf )|
|Ω|2
Filippi, Mallet, Nader Model evaluation May 2013 4 / 50
Dynamic-aware scoresArrival time agreement
Addition notationArrival time: earliest time at which the front is known to havereached some point; +∞ if the front never reaches the pointObserved arrival time at X : T o(X ), say T o(X ) = to
f if X ∈ So(tof )
Simulated arrival time at X : T (X ), here, T (X ) ≤ tsf if X ∈ S(ts
f )
Arrival time agreement
1− 1|S(ts
f ) ∪ So(tof )|max(ts
f , tof )
[∫S(ts
f )∩So(tof )max(T (X )− T o(X ), 0)dX
+
∫S(ts
f )\So(tof )max(to
f − T (X ), 0)dX
+
∫So(to
f )\S(tsf )
(tsf − T o(X ))dX
]∈ [0, 1]
Filippi, Mallet, Nader Model evaluation May 2013 5 / 50
Dynamic-aware scoresShape agreement
Shape agreement
1− 1tsf
[∫]0,to
f ]
|S(t)\So(tof )|
|S(t)|dt +
∫[to
f ,tsf [
|So(tof )\S(t)||So(to
f )|dt
]∈ [0, 1]
Filippi, Mallet, Nader Model evaluation May 2013 6 / 50
Application of the scores to an idealized case
0 200 400 600 800
0
200
400
600
800
0
200
400
600
800
1000
1200
1400
1600
1800
2000
0 200 400 600 800
0
200
400
600
800
0
200
400
600
800
1000
1200
1400
1600
1800
2000
S = 0.981J = 0.962K = 0.977
ATA = 0.985SA = 0.973
0 200 400 600 800
0
200
400
600
800
0
250
500
750
1000
1250
1500
1750
2000
2250
S = 0.981J = 0.962K = 0.977
ATA = 0.998SA = 0.997
J.-B. Filippi, V. Mallet, and B. Nader (2013). “Representation and evaluation ofwildfire propagation simulations”. In: Under review for Int. J. Wild. Fire
Scoring methods in Python at http://sf.net/projects/pyfirescore/
Filippi, Mallet, Nader Model evaluation May 2013 7 / 50
Simulation of 80 fire cases
Fire casesUsing observations from Prométhée, database of french firesAvailable data for a subset of the fires: date, ignition point, finalcontourUnfortunately, no data on firefightsConsidering 80 Corsican fires from 2003 to 2008
Filippi, Mallet, Nader Model evaluation May 2013 8 / 50
80 fire cases [1/5]
0 2000400060008000100001200014000160000
2000
4000
6000
8000
10000
12000
14000Aullene [990 ha]
0 500 1000 1500 2000 2500 30000
500
1000
1500
2000
2500
3000
3500Oletta [54 ha]
0 100020003000400050006000700080000
1000
2000
3000
4000
5000
6000
7000Olmi Cappella [193 ha]
0 200040006000800010000120001400016000180000
2000
4000
6000
8000
10000
12000
14000Pietracorbara [1083 ha]
0 2000 4000 6000 80001000012000140000
5000
10000
15000
20000
25000Santo Pietro di Tenda [1310 ha]
0 5000 10000 15000 20000 250000
5000
10000
15000
20000
25000Calenzana [1419 ha]
0 200 400 600 800 1000 1200 14000
500
1000
1500
2000Calenzana [18 ha]
0 200 400 600 800 1000 1200 14000
500
1000
1500
2000
2500
3000
3500Solaro [29 ha]
0 500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500Volpajola [50 ha]
0 1000 2000 3000 4000 5000 60000
1000
2000
3000
4000
5000
6000
7000Murzo [180 ha]
0 1000 2000 3000 4000 50000
1000
2000
3000
4000
5000Ghisonaccia [95 ha]
0 500 1000 1500 2000 2500 3000 35000
50010001500200025003000350040004500
Vivario [57 ha]
0 500 10001500200025003000350040000
500
1000
1500
2000
2500Corscia [58 ha]
0 2000 4000 6000 8000 10000 120000
2000400060008000
10000120001400016000
Biguglia [783 ha]
0 100 200 300 400 500 600 700 800 9000
100
200
300
400
500
600Casaglione [2 ha]
0 500 1000 1500 2000 2500 30000
1000
2000
3000
4000
5000Aleria [62 ha]
Filippi, Mallet, Nader Model evaluation May 2013 9 / 50
80 fire cases [2/5]
0 100020003000400050006000700080000
2000
4000
6000
8000
10000
12000Sisco [441 ha]
0 200 400 600 800100012001400160018000
500
1000
1500
2000
2500
3000
3500Linguizzetta [34 ha]
0 500 1000 1500 2000 2500 30000
1000
2000
3000
4000
5000
6000Aleria [80 ha]
0 500 1000 1500 20000
500
1000
1500
2000
2500Prunelli di Fiumorbo [26 ha]
0 500 1000 1500 2000 2500 30000
1000
2000
3000
4000
5000
6000Olmeta di Tuda [81 ha]
0 500 10001500200025003000350040000
1000
2000
3000
4000
5000Calenzana [76 ha]
0 500 1000 1500 2000 2500 30000
10002000300040005000600070008000
Nocario [82 ha]
0 500 1000 1500 2000 25000
500
1000
1500
2000
2500
3000Ventiseri [29 ha]
0 100 200 300 400 500 6000
200
400
600
800
1000Porto Vecchio [3 ha]
0 1000 2000 3000 4000 5000 6000 70000
100020003000400050006000700080009000
Afa [242 ha]
0 500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500Loreto di Casinca [42 ha]
0 500 10001500200025003000350040000
500
1000
1500
2000
2500
3000Calenzana [59 ha]
0 200 400 600 800 1000 1200 14000
500
1000
1500
2000
2500Corbara [10 ha]
0 2000 4000 6000 8000 100000
2000
4000
6000
8000
10000
12000Vero [535 ha]
0 200 400 600 800 10001200140016000
50010001500200025003000350040004500
Canale di Verde [35 ha]
0 500 1000 1500 2000 2500 30000
200400600800
10001200140016001800
Altiani [24 ha]
Filippi, Mallet, Nader Model evaluation May 2013 10 / 50
80 fire cases [3/5]
0 200 400 600 800100012001400160018000
200
400
600
800
1000
1200
1400Patrimonio [13 ha]
0 1000 2000 3000 4000 5000 6000 70000
5001000150020002500300035004000
Pruno [116 ha]
0 500100015002000250030003500400045000
5001000150020002500300035004000
Olcani [70 ha]
0 50 100 150 200 250 3000
50
100
150
200
250Sartene [0 ha]
0 500 1000 1500 2000 2500 30000
200
400
600
800
1000
1200
1400Calenzana [16 ha]
0 100 200 300 400 500 600 7000
200
400
600
800
1000Propriano [3 ha]
0 1000200030004000500060007000800090000
1000
2000
3000
4000
5000
6000Canari [95 ha]
0 200 400 600 800 1000 1200 14000
200400600800
10001200140016001800
Olmeta di Tuda [18 ha]
0 1000 2000 3000 4000 5000 60000
10002000300040005000600070008000
Calenzana [91 ha]
0 500 1000 1500 2000 2500 3000 35000
1000
2000
3000
4000
5000
6000Calenzana [103 ha]
0 1000200030004000500060007000800090000
500
1000
1500
2000
2500
3000Calenzana [143 ha]
0 200 400 600 800 10000
100
200
300
400
500
600
700Piana [2 ha]
0 200 400 600 800100012001400160018000
200400600800
1000120014001600
Ajaccio [14 ha]
0 50 100 150 200 250 3000
50
100
150
200
250
300
350Bastelicaccia [0 ha]
40 60 80 100 120 140 160 180 200 2200
50
100
150
200
250
300Propriano [1 ha]
0 200040006000800010000120001400016000180000
2000400060008000
1000012000140001600018000
Oletta [1126 ha]
Filippi, Mallet, Nader Model evaluation May 2013 11 / 50
80 fire cases [4/5]
0 500 1000 1500 2000 2500 30000
500
1000
1500
2000
2500
3000Soveria [56 ha]
0 500100015002000250030003500400045000
2000
4000
6000
8000
10000
12000
14000Sisco [382 ha]
0 1000 2000 3000 4000 50000
1000
2000
3000
4000
5000
6000
7000Coti Chiavari [185 ha]
0 1000 2000 3000 4000 5000 60000
10002000300040005000600070008000
Lumio [201 ha]
0 500 1000 1500 2000 2500 30000
5001000150020002500300035004000
Santa Maria Poggio [59 ha]
0 1000200030004000500060007000800090000
2000
4000
6000
8000
10000
12000Barbaggio [518 ha]
0 500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500
3000
3500Rutali [60 ha]
0 1000 2000 3000 4000 50000
1000
2000
3000
4000
5000
6000
7000Oletta [186 ha]
0 50 100 150 200 250 3000
50
100
150
200
250
300
350Propriano [0 ha]
0 2000 4000 6000 8000 10000 120000
10002000300040005000600070008000
Calenzana [465 ha]
0 500 10001500200025003000350040000
1000
2000
3000
4000
5000
6000Calvi [79 ha]
0 200 400 600 800 1000 12000
200400600800
1000120014001600
San Giovanni di Moriani [10 ha]
0 500 1000 1500 2000 2500 30000
1000
2000
3000
4000
5000Luri [51 ha]
0 200 400 600 800100012001400160018000
500
1000
1500
2000
2500Saint Florent [25 ha]
0 2000400060008000100001200014000160000
2000400060008000
10000120001400016000
Peri [750 ha]
0 500 1000 1500 2000 2500 30000
5001000150020002500300035004000
Poggio d'Oletta [53 ha]
Filippi, Mallet, Nader Model evaluation May 2013 12 / 50
80 fire cases [5/5]
0 500 1000 1500 20000
500
1000
1500
2000
2500
3000
3500Borgo [41 ha]
0 500100015002000250030003500400045000
2000400060008000
10000120001400016000
Sisco [357 ha]
0 500 1000 1500 2000 2500 30000
500
1000
1500
2000Calenzana [27 ha]
0 2000 4000 6000 8000 10000 120000
2000400060008000
10000120001400016000
Bonifacio [477 ha]
0 100 200 300 400 500 600 700 8000
100200300400500600700800
Propriano [4 ha]
0 2000400060008000100001200014000160000
2000400060008000
10000120001400016000
Tolla [942 ha]
0 500 1000 1500 2000 25000
500
1000
1500
2000
2500Santa Lucia di Mercurio [19 ha]
40 60 80 100 120 140 160 180 2000
50
100
150
200
250Alata [0 ha]
0 500100015002000250030003500400045000
500
1000
1500
2000
2500
3000
3500Omessa [116 ha]
0 100020003000400050006000700080000
100020003000400050006000700080009000
Ersa [157 ha]
0 100 200 300 400 500 600 700 800 9000
500
1000
1500
2000
2500Manso [11 ha]
0 200 400 600 800 10000
500
1000
1500
2000
2500Olmeta di Tuda [13 ha]
0 500 10001500200025003000350040000
1000
2000
3000
4000
5000Santo Pietro di Tenda [111 ha]
0 500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500Calenzana [28 ha]
0 200 400 600 800100012001400160018000
200
400
600
800
1000
1200
1400Bisinchi [14 ha]
0 500 1000 1500 2000 25000
500
1000
1500
2000
2500
3000
3500Montegrosso [44 ha]
Filippi, Mallet, Nader Model evaluation May 2013 13 / 50
Simulation of 80 fire casesSimulations
Land use cover: “inventaire forestier national” (IFN, from IGN) andglobal land cover (GLC)Elevation at 25 m resolution from IGNWind velocity and direction
Taken at Ajaccio or Bastia meteorological station, or from ECMWFsimulationsAnd then computed by Windninja (US Forest Service and ColoradoState University)
Running ForeFire (SPE), until the final burned area is attainedFour models for the rate of spread
Balbi model (cf. his talk, tomorrow), “stationary”Balbi model, “non-stationary”, i.e., with front depth and dependenceon the front geometryRothermel modelSimple rate of spread equal to 3% of wind velocity
Filippi, Mallet, Nader Model evaluation May 2013 14 / 50
80 fire cases [1/5]
0 2000400060008000100001200014000160000
2000
4000
6000
8000
10000
12000
14000Aullene [990 ha]
0 500 1000 1500 2000 2500 30000
500
1000
1500
2000
2500
3000
3500Oletta [54 ha]
0 100020003000400050006000700080000
1000
2000
3000
4000
5000
6000
7000Olmi Cappella [193 ha]
0 200040006000800010000120001400016000180000
2000
4000
6000
8000
10000
12000
14000Pietracorbara [1083 ha]
0 2000 4000 6000 80001000012000140000
5000
10000
15000
20000
25000Santo Pietro di Tenda [1310 ha]
0 5000 10000 15000 20000 250000
5000
10000
15000
20000
25000Calenzana [1419 ha]
0 200 400 600 800 1000 1200 14000
500
1000
1500
2000Calenzana [18 ha]
0 200 400 600 800 1000 1200 14000
500
1000
1500
2000
2500
3000
3500Solaro [29 ha]
0 500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500Volpajola [50 ha]
0 1000 2000 3000 4000 5000 60000
1000
2000
3000
4000
5000
6000
7000Murzo [180 ha]
0 1000 2000 3000 4000 50000
1000
2000
3000
4000
5000Ghisonaccia [95 ha]
0 500 1000 1500 2000 2500 3000 35000
50010001500200025003000350040004500
Vivario [57 ha]
0 500 10001500200025003000350040000
500
1000
1500
2000
2500Corscia [58 ha]
0 2000 4000 6000 8000 10000 120000
2000400060008000
10000120001400016000
Biguglia [783 ha]
0 100 200 300 400 500 600 700 800 9000
100
200
300
400
500
600Casaglione [2 ha]
0 500 1000 1500 2000 2500 30000
1000
2000
3000
4000
5000Aleria [62 ha]
Filippi, Mallet, Nader Model evaluation May 2013 15 / 50
80 fire cases [2/5]
0 100020003000400050006000700080000
2000
4000
6000
8000
10000
12000Sisco [441 ha]
0 200 400 600 800100012001400160018000
500
1000
1500
2000
2500
3000
3500Linguizzetta [34 ha]
0 500 1000 1500 2000 2500 30000
1000
2000
3000
4000
5000
6000Aleria [80 ha]
0 500 1000 1500 20000
500
1000
1500
2000
2500Prunelli di Fiumorbo [26 ha]
0 500 1000 1500 2000 2500 30000
1000
2000
3000
4000
5000
6000Olmeta di Tuda [81 ha]
0 500 10001500200025003000350040000
1000
2000
3000
4000
5000Calenzana [76 ha]
0 500 1000 1500 2000 2500 30000
10002000300040005000600070008000
Nocario [82 ha]
0 500 1000 1500 2000 25000
500
1000
1500
2000
2500
3000Ventiseri [29 ha]
0 100 200 300 400 500 6000
200
400
600
800
1000Porto Vecchio [3 ha]
0 1000 2000 3000 4000 5000 6000 70000
100020003000400050006000700080009000
Afa [242 ha]
0 500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500Loreto di Casinca [42 ha]
0 500 10001500200025003000350040000
500
1000
1500
2000
2500
3000Calenzana [59 ha]
0 200 400 600 800 1000 1200 14000
500
1000
1500
2000
2500Corbara [10 ha]
0 2000 4000 6000 8000 100000
2000
4000
6000
8000
10000
12000Vero [535 ha]
0 200 400 600 800 10001200140016000
50010001500200025003000350040004500
Canale di Verde [35 ha]
0 500 1000 1500 2000 2500 30000
200400600800
10001200140016001800
Altiani [24 ha]
Filippi, Mallet, Nader Model evaluation May 2013 16 / 50
80 fire cases [3/5]
0 200 400 600 800100012001400160018000
200
400
600
800
1000
1200
1400Patrimonio [13 ha]
0 1000 2000 3000 4000 5000 6000 70000
5001000150020002500300035004000
Pruno [116 ha]
0 500100015002000250030003500400045000
5001000150020002500300035004000
Olcani [70 ha]
0 50 100 150 200 250 3000
50
100
150
200
250Sartene [0 ha]
0 500 1000 1500 2000 2500 30000
200
400
600
800
1000
1200
1400Calenzana [16 ha]
0 100 200 300 400 500 600 7000
200
400
600
800
1000Propriano [3 ha]
0 1000200030004000500060007000800090000
1000
2000
3000
4000
5000
6000Canari [95 ha]
0 200 400 600 800 1000 1200 14000
200400600800
10001200140016001800
Olmeta di Tuda [18 ha]
0 1000 2000 3000 4000 5000 60000
10002000300040005000600070008000
Calenzana [91 ha]
0 500 1000 1500 2000 2500 3000 35000
1000
2000
3000
4000
5000
6000Calenzana [103 ha]
0 1000200030004000500060007000800090000
500
1000
1500
2000
2500
3000Calenzana [143 ha]
0 200 400 600 800 10000
100
200
300
400
500
600
700Piana [2 ha]
0 200 400 600 800100012001400160018000
200400600800
1000120014001600
Ajaccio [14 ha]
0 50 100 150 200 250 3000
50
100
150
200
250
300
350Bastelicaccia [0 ha]
40 60 80 100 120 140 160 180 200 2200
50
100
150
200
250
300Propriano [1 ha]
0 200040006000800010000120001400016000180000
2000400060008000
1000012000140001600018000
Oletta [1126 ha]
Filippi, Mallet, Nader Model evaluation May 2013 17 / 50
80 fire cases [4/5]
0 500 1000 1500 2000 2500 30000
500
1000
1500
2000
2500
3000Soveria [56 ha]
0 500100015002000250030003500400045000
2000
4000
6000
8000
10000
12000
14000Sisco [382 ha]
0 1000 2000 3000 4000 50000
1000
2000
3000
4000
5000
6000
7000Coti Chiavari [185 ha]
0 1000 2000 3000 4000 5000 60000
10002000300040005000600070008000
Lumio [201 ha]
0 500 1000 1500 2000 2500 30000
5001000150020002500300035004000
Santa Maria Poggio [59 ha]
0 1000200030004000500060007000800090000
2000
4000
6000
8000
10000
12000Barbaggio [518 ha]
0 500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500
3000
3500Rutali [60 ha]
0 1000 2000 3000 4000 50000
1000
2000
3000
4000
5000
6000
7000Oletta [186 ha]
0 50 100 150 200 250 3000
50
100
150
200
250
300
350Propriano [0 ha]
0 2000 4000 6000 8000 10000 120000
10002000300040005000600070008000
Calenzana [465 ha]
0 500 10001500200025003000350040000
1000
2000
3000
4000
5000
6000Calvi [79 ha]
0 200 400 600 800 1000 12000
200400600800
1000120014001600
San Giovanni di Moriani [10 ha]
0 500 1000 1500 2000 2500 30000
1000
2000
3000
4000
5000Luri [51 ha]
0 200 400 600 800100012001400160018000
500
1000
1500
2000
2500Saint Florent [25 ha]
0 2000400060008000100001200014000160000
2000400060008000
10000120001400016000
Peri [750 ha]
0 500 1000 1500 2000 2500 30000
5001000150020002500300035004000
Poggio d'Oletta [53 ha]
Filippi, Mallet, Nader Model evaluation May 2013 18 / 50
80 fire cases [5/5]
0 500 1000 1500 20000
500
1000
1500
2000
2500
3000
3500Borgo [41 ha]
0 500100015002000250030003500400045000
2000400060008000
10000120001400016000
Sisco [357 ha]
0 500 1000 1500 2000 2500 30000
500
1000
1500
2000Calenzana [27 ha]
0 2000 4000 6000 8000 10000 120000
2000400060008000
10000120001400016000
Bonifacio [477 ha]
0 100 200 300 400 500 600 700 8000
100200300400500600700800
Propriano [4 ha]
0 2000400060008000100001200014000160000
2000400060008000
10000120001400016000
Tolla [942 ha]
0 500 1000 1500 2000 25000
500
1000
1500
2000
2500Santa Lucia di Mercurio [19 ha]
40 60 80 100 120 140 160 180 2000
50
100
150
200
250Alata [0 ha]
0 500100015002000250030003500400045000
500
1000
1500
2000
2500
3000
3500Omessa [116 ha]
0 100020003000400050006000700080000
100020003000400050006000700080009000
Ersa [157 ha]
0 100 200 300 400 500 600 700 800 9000
500
1000
1500
2000
2500Manso [11 ha]
0 200 400 600 800 10000
500
1000
1500
2000
2500Olmeta di Tuda [13 ha]
0 500 10001500200025003000350040000
1000
2000
3000
4000
5000Santo Pietro di Tenda [111 ha]
0 500 1000 1500 2000 2500 3000 35000
500
1000
1500
2000
2500Calenzana [28 ha]
0 200 400 600 800100012001400160018000
200
400
600
800
1000
1200
1400Bisinchi [14 ha]
0 500 1000 1500 2000 25000
500
1000
1500
2000
2500
3000
3500Montegrosso [44 ha]
Filippi, Mallet, Nader Model evaluation May 2013 19 / 50
Distribution of the Sørensen similarity indexesScores sorted independently for each model
0 10 20 30 40 50 60 70 800.00.10.20.30.40.50.60.70.80.9
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 20 / 50
Distribution of the Jaccard similarity coefficientsScores sorted independently for each model
0 10 20 30 40 50 60 70 800.00.10.20.30.40.50.60.70.8
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 21 / 50
Distribution of the Kappa coefficientsScores sorted independently for each model
0 10 20 30 40 50 60 70 800.2
0.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 22 / 50
Distribution of the shape agreementsScores sorted independently for each model
0 10 20 30 40 50 60 70 800.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 23 / 50
Distribution of the Sørensen similarity indexesScores sorted according to the average score
0 10 20 30 40 50 60 70 800.00.10.20.30.40.50.60.70.80.9
Scor
e
Filippi, Mallet, Nader Model evaluation May 2013 24 / 50
Distribution of the Jaccard similarity coefficientsScores sorted according to the average score
0 10 20 30 40 50 60 70 800.00.10.20.30.40.50.60.70.8
Scor
e
Filippi, Mallet, Nader Model evaluation May 2013 25 / 50
Distribution of the Kappa coefficientsScores sorted according to the average score
0 10 20 30 40 50 60 70 800.2
0.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Filippi, Mallet, Nader Model evaluation May 2013 26 / 50
Distribution of the shape agreementsScores sorted according to the average score
0 10 20 30 40 50 60 70 800.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Filippi, Mallet, Nader Model evaluation May 2013 27 / 50
Monte Carlo simulations
Pertubed parametersWind direction: additive perturbation, clipped normal distribution,with zero mean and standard deviation of 40
Wind velocity: log-normal distribution, assuming [12v , 2v ] is a
0.95-confidence intervalFinal burned area: log-normal distribution, assuming [2
3v , 32v ] is a
0.95-confidence intervalFuel load: log-normal distribution, assuming [ 1
1.1v , 1.1v ] is a0.95-confidence intervalMoisture content: log-normal distribution, assuming [ 1
1.1v , 1.1v ] is a0.95-confidence interval
Filippi, Mallet, Nader Model evaluation May 2013 28 / 50
Monte Carlo simulations
Monte Carlo experimentSelecting the four models with equal probabilityRunning on 75 cases (out of the 80 cases), with at least1150 simulations per caseAbout 89,000 simulations in total
Filippi, Mallet, Nader Model evaluation May 2013 29 / 50
Distribution of the Sørensen similarity indexesScores sorted independently for each model
0 5000 10000 15000 20000 250000.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 30 / 50
Distribution of the Jaccard similarity coefficientsScores sorted independently for each model
0 5000 10000 15000 20000 250000.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 31 / 50
Distribution of the Kappa coefficientsScores sorted independently for each model
0 5000 10000 15000 20000 250000.2
0.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 32 / 50
Distribution of the shape agreementsScores sorted independently for each model
0 5000 10000 15000 20000 250000.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 33 / 50
Distribution of the Sørensen similarity indexesScores of the best simulations, sorted independently for each model
0 10 20 30 40 50 60 70 800.00.10.20.30.40.50.60.70.80.9
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 34 / 50
Distribution of the Jaccard similarity coefficientsScores of the best simulations, sorted independently for each model
0 10 20 30 40 50 60 70 800.00.10.20.30.40.50.60.70.80.9
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 35 / 50
Distribution of the Kappa coefficientsScores of the best simulations, sorted independently for each model
0 10 20 30 40 50 60 70 800.00.10.20.30.40.50.60.70.80.9
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 36 / 50
Distribution of the shape agreementsScores of the best simulations, sorted independently for each model
0 10 20 30 40 50 60 70 800.0
0.2
0.4
0.6
0.8
1.0
Scor
e
Balbi non stationaryBalbi stationaryRothermel3-percent
Filippi, Mallet, Nader Model evaluation May 2013 37 / 50
Distribution of the Jaccard similarity coefficientsAgainst input parameters, for case “Patrimonio (2005-02-11)”
50 0 50 1001502002503003504000.10.00.10.20.30.40.50.60.70.8
0 2 4 6 8 10 120.10.00.10.20.30.40.50.60.70.8
12 14 16 18 20 22 240.10.00.10.20.30.40.50.60.70.8
0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.40.10.00.10.20.30.40.50.60.70.8
0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.40.10.00.10.20.30.40.50.60.70.8
Filippi, Mallet, Nader Model evaluation May 2013 38 / 50
Distribution of the shape agreementsAgainst input parameters, for case “Patrimonio (2005-02-11)”
50 0 50 1001502002503003504000.50.60.70.80.91.0
0 2 4 6 8 10 120.50.60.70.80.91.0
12 14 16 18 20 22 240.50.60.70.80.91.0
0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.40.50.60.70.80.91.0
0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.40.50.60.70.80.91.0
Filippi, Mallet, Nader Model evaluation May 2013 39 / 50
Distribution of the shape agreementsAgainst input parameters, for case “Olcani (2006-01-01)”
0 50 100 150 200 250 3000.20.00.20.40.60.81.01.2
0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.00.20.00.20.40.60.81.01.2
60 70 80 90 100 110 120 130 1400.20.00.20.40.60.81.01.2
0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.40.20.00.20.40.60.81.01.2
0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.40.20.00.20.40.60.81.01.2
Filippi, Mallet, Nader Model evaluation May 2013 40 / 50
Distribution of the shape agreementsAgainst input parameters, for case “Sisco (2003-08-14)”
50 0 50 1001502002503003504000.20.30.40.50.60.70.80.91.0
0.0 0.5 1.0 1.5 2.0 2.50.20.30.40.50.60.70.80.91.0
350 400 450 500 550 600 650 700 7500.20.30.40.50.60.70.80.91.0
0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.40.20.30.40.50.60.70.80.91.0
0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.40.20.30.40.50.60.70.80.91.0
Filippi, Mallet, Nader Model evaluation May 2013 41 / 50
75 fire cases [1/5]
2000 4000 6000 8000100001200014000
2000
4000
6000
8000
10000
12000
500 1000 1500 2000 2500
500
1000
1500
2000
2500
3000
1000 2000 3000 4000 5000 6000 7000
1000
2000
3000
4000
5000
6000
5000 10000 15000 20000
5000
10000
15000
20000
20040060080010001200
500
1000
1500
20040060080010001200
500
1000
1500
2000
2500
3000
500 1000 1500 2000 2500 3000
500
1000
1500
2000
1000 2000 3000 4000 5000
1000
2000
3000
4000
5000
6000
1000 2000 3000 4000
1000
2000
3000
4000
50010001500200025003000
500
1000
1500
2000
2500
3000
3500
4000
500 1000 1500 2000 2500 3000 3500
500
1000
1500
2000
200040006000800010000
2000
4000
6000
8000
10000
12000
14000
100 200 300 400 500 600 700 800
100
200
300
400
500
5001000150020002500
1000
2000
3000
4000
2004006008001000120014001600
500
1000
1500
2000
2500
3000
5001000150020002500
1000
2000
3000
4000
5000
Filippi, Mallet, Nader Model evaluation May 2013 42 / 50
75 fire cases [2/5]
500 1000 1500
500
1000
1500
2000
5001000150020002500
1000
2000
3000
4000
5000
500100015002000250030003500
1000
2000
3000
4000
5001000150020002500
1000
2000
3000
4000
5000
6000
7000
500 1000 1500 2000
500
1000
1500
2000
2500
100 200 300 400 500
100
200
300
400
500
600
700
800
900
100020003000400050006000
1000
2000
3000
4000
5000
6000
7000
8000
500 1000 1500 2000 2500 3000 3500
500
1000
1500
2000
2500
20040060080010001200
500
1000
1500
2000
500 1000 1500 2000 2500 3000
500
1000
1500
2000
200400600800100012001400
500
1000
1500
2000
2500
3000
3500
4000
500 1000 1500 2000 2500
200
400
600
800
1000
1200
1400
1600
200 400 600 800 1000 1200 1400 1600
200
400
600
800
1000
1200
1000 2000 3000 4000 5000 6000
500
1000
1500
2000
2500
3000
3500
500 1000150020002500300035004000
500
1000
1500
2000
2500
3000
3500
50 100 150 200 250
50
100
150
200
Filippi, Mallet, Nader Model evaluation May 2013 43 / 50
75 fire cases [3/5]
500 1000 1500 2000 2500
200
400
600
800
1000
1200
100200300400500600
200
400
600
800
1000 2000 3000 4000 5000 6000 7000 8000
1000
2000
3000
4000
5000
200 400 600 80010001200
200
400
600
800
1000
1200
1400
1600
10002000300040005000
1000
2000
3000
4000
5000
6000
7000
50010001500200025003000
1000
2000
3000
4000
5000
1000 2000 3000 4000 5000 6000 7000 8000
5001000150020002500
200 400 600 800
100
200
300
400
500
600
200 400 600 800 1000120014001600
200
400
600
800
1000
1200
1400
50 100 150 200 250
50
100
150
200
250
300
60 80100120140160180200
50
100
150
200
250
200040006000800010000120001400016000
2000
4000
6000
8000
10000
12000
14000
16000
500 1000 1500 2000 2500
500
1000
1500
2000
2500
5001000150020002500300035004000
2000
4000
6000
8000
10000
12000
1000 2000 3000 4000
1000
2000
3000
4000
5000
6000
10002000300040005000
1000
2000
3000
4000
5000
6000
7000
Filippi, Mallet, Nader Model evaluation May 2013 44 / 50
75 fire cases [4/5]
500 1000150020002500
500
1000
1500
2000
2500
3000
3500
10002000300040005000600070008000
2000
4000
6000
8000
10000
500 1000 1500 2000 2500 3000
500
1000
1500
2000
2500
3000
1000200030004000
1000
2000
3000
4000
5000
6000
50 100 150 200 250
50
100
150
200
250
300
2000 4000 6000 8000 10000
1000
2000
3000
4000
5000
6000
7000
500100015002000250030003500
1000
2000
3000
4000
5000
200 400 600 800 1000
200
400
600
800
1000
1200
1400
5001000150020002500
1000
2000
3000
4000
2004006008001000120014001600
500
1000
1500
2000
2000400060008000100001200014000
2000
4000
6000
8000
10000
12000
14000
500 1000150020002500
500
1000
1500
2000
2500
3000
3500
500 1000 1500
500
1000
1500
2000
2500
3000
5001000150020002500300035004000
2000
4000
6000
8000
10000
12000
14000
500 1000 1500 2000 2500
500
1000
1500
100 200 300 400 500 600 700
100
200
300
400
500
600
700
Filippi, Mallet, Nader Model evaluation May 2013 45 / 50
75 fire cases [5/5]
2000400060008000100001200014000
2000
4000
6000
8000
10000
12000
14000
500 1000 1500 2000
500
1000
1500
2000
60 80100120140160180
50
100
150
200
500 1000 1500 2000 2500 3000 3500 4000
500
1000
1500
2000
2500
3000
1000200030004000500060007000
1000
2000
3000
4000
5000
6000
7000
8000
100200300400500600700800
500
1000
1500
2000
200400600800
500
1000
1500
2000
500100015002000250030003500
1000
2000
3000
4000
500 1000 1500 2000 2500 3000
500
1000
1500
2000
200 400 600 800 1000 1200 1400 1600
200
400
600
800
1000
1200
500 1000 1500 2000
500
1000
1500
2000
2500
3000
Filippi, Mallet, Nader Model evaluation May 2013 46 / 50
Probability densityFor case “Olcani (2006-01-01)”
500 1000150020002500300035004000
500
1000
1500
2000
2500
3000
3500
Filippi, Mallet, Nader Model evaluation May 2013 47 / 50
Probability densityFor case “Patrimonio (2005-02-11)”
200 400 600 800 1000 1200 1400 1600
200
400
600
800
1000
1200
Filippi, Mallet, Nader Model evaluation May 2013 48 / 50
Reliability diagram
0.0 0.2 0.4 0.6 0.8 1.0Computed probability
0.0
0.2
0.4
0.6
0.8
1.0O
bser
ved
prob
abili
ty
Filippi, Mallet, Nader Model evaluation May 2013 49 / 50
Conclusions
Scoring methodsIt is possible to take into account the model dynamics
Arrival time agreement and shape agreement
Evaluation on 80 firesIt seems possible to rank models, without any (over)tuning3-percent rate of spread significantly worse
Monte Carlo simulationsMay also be used to rank modelsToward probabilistic forecasts and uncertainty estimation
Filippi, Mallet, Nader Model evaluation May 2013 50 / 50