Crown copyright Met Office Verifying modelled currents using a threshold exceedance approach Dr Ray Mahdon An exploration of the Gerrity Skill Score NPE - Cross-cutting research on verification techniques Presentation Session Code: SCI-PS153.03 Crown copyright Met Office Verifying modelled currents using a threshold exceedance approach An exploration of the Gerrity Skill Score Table of Contents Introduction Data Source & Locations Differing Current Regimes Time Series, Continuous Statistics & Simple Cat. Metrics Neighbourhood Methods Bias Removal Questions Multi-Cat. Metric Gerrity Skill Score & Ocean Currents Threshold Choices Crown copyright Met Office Introduction Some questions we are trying to answer.. Does the model capture extreme events or weather-windows? In which locations or time of year do the models have the best performance; is there a significant difference in regime, time or area? Surface currents forecasts important for commercial or defence weather-windows e.g. Current speed below 1kt for 12 hours. e.g. Does not exceed 1kt more than x times Good for site-specific & threshold based analysis Crown copyright Met Office 26-56N,19W-5E MyOcean - Puertos Del Estado Matxitxako Donostia Data Source & Locations Slope Current General Ocean Circulation Eddies Shelf Circulation Wind & Tidal Currents Crown copyright Met Office Data, Time Series & Continuous Statistics Hourly frequency, Jan 2012 Jun 2014 (30 months) Collocated model & In-Situ moored observation surface currents Continuous statistics are helpful to describe overall behaviour e.g. q-q & histogram plots describe climatology Timeseries can show seasonal patterns or significant events Do not quantify the performance of a system when exceeding thresholds is of interest We focus on surface currents validation is relatively sparse for this parameter Categorical Metric Assessment Simple 2x2 (binary) contingency table per chosen threshold Crown copyright Met Office Neighbourhoods: 1x1, 3x3, 5x5,..,NxN Neighbourhood Sampling T+0 T+1 T-1 Spatial Neighbourhoods Temporal Neighbourhoods Time averaging & shifting Combinations spatial & temporal neighbourhoods trialled Crown copyright Met Office Simple Categorical Metrics CSI & ETS require un-biased input data Over what period should a tidally dominated field be normalised: 1 tidal cycle; spring-neap cycle; astronomical cycle? How to handle ve currents? MISSES HITS F. ALARMS ETS CSI CORR. REJ. Improvements from temporal averaging hour-hour assessment not good as CSI ETS says model mostly correct by chance! Crown copyright Met Office Multi-Categorical Metric Method The Gerrity Skill Score Crown copyright Met Office Gerrity* Skill Score (GSS) Refinement of binary categorical methods Does not depend on the forecast distribution Rewards/penalises for rare(extreme)/disparate events does not reward conservative forecasting Large choice of threshold divisions Good observation (sample) climatology required Contingency table distribution leads to scoring matrix Equitable (i.e., random & constant forecasts score a value of 0) CT1 OBS T1