product validation report - servizio...

Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1)

Products Validation Programme

Italian Meteorological Service Italian Department of Civil Defence

H-SAF Product Validation Report (PVR)

PR-ASS-1 - Instantaneous and accumulated precipitation at ground computed by a NWP model

Zentralanstalt für Meteorologie und

Geodynamik

Vienna University of Technology Institut für Photogrammetrie

und Fernerkundung

Royal Meteorological Institute of Belgium

European Centre for Medium-Range Weather Forecasts

Finnish Meteorological Institute

Finnish Environment Institute

Helsinki University of Technology

Météo-France CNRS Laboratoire Atmosphères, Milieux, Observations Spatiales

CNRS Centre d'Etudes Spatiales de la BIOsphere

Bundesanstalt für Gewässerkunde

Hungarian Meteorological Service

CNR - Istituto Scienze dell’Atmosfera

e del Clima Università di Ferrara

Institute of Meteorology and Water Management

Romania National Meteorological Administration

Slovak Hydro-Meteorological Institute

Turkish State Meteorological Service

Middle East Technical University

Istanbul Technical University

Anadolu University

30 May 2010


H-SAF Product Validation Report PVR-06

Product PR-ASS-1

Instantaneous and accumulated precipitation at ground computed by a NWP model

INDEX

Page

Acronyms [not including those in the Appendix] 06

1. The EUMETSAT Satellite Application Facilities and H-SAF 08

2. Introduction to product PR-ASS-1 09

2.1 Principle of the product 08

2.2 Algorithm principle 09

2.3 Main operational characteristics 10

3. Validation strategy, methods and tools 11

3.1 Validation team and work plan 11

3.2 Validation philosophy 11

3.2.1 Objectives and problems 11

3.2.2 Tools to be used for validation 12

3.2.3 Techniques to bring observations comparable 13

3.2.4 Structuring the results of the validation activity 14

3.3 Definition of statistical scores 16

3.4 Inventory of validation facilities 18

3.4.1 Facilities in Belgium (IRM) 18

3.4.2 Facilities in Germany (BfG) 22

3.4.3 Facilities in Hungary (OMSZ) 24

3.4.4 Facilities in Italy (UniFe) 27

3.4.5 Facilities in Poland (IMWM) 29

3.4.6 Facilities in Slovakia (SHMÚ) 31

3.4.7 Facilities in Turkey (ITU) 35

4. Validation of the product release as at the end of the Development Phase 39

4.1 Introduction 39

4.2 Validation in Belgium (IRM) 40

4.3 Validation in Germany (BfG) 42

4.4 Validation in Hungary (OMSZ) 48

4.5 Validation in Italy (UniFe) 55

4.6 Validation in Poland (IMWM) 58

4.7 Validation in Slovakia (SHMÚ) 63

4.8 Validation in Turkey (ITU) 66

5. Overview of findings 69

5.1 Synopsis of validation results 69

5.2 Summary conclusions on comparative elements 72


Appendix - Collection of validation experiment reports [low-level of editing] [4 pages] 75

Yellow: statistical analyses over several months Blue: case studies in specified few days

§ Validation experiments on PR-ASS-1 Period Institute

2. Contribution not expected (exclusive task of CNMCA) Belgium/IRM

3. Contribution not expected (exclusive task of CNMCA) Germany/BfG

4. Contribution not expected (exclusive task of CNMCA) Hungary/OMSZ

5.1 Half-year statistics September 2008 - February 2009 Italy/CNMCA

5.2 Round-year statistics December 2008 - November 2009 Italy/CNMCA

6. Contribution not expected (exclusive task of CNMCA) Poland/IMWG

7. Contribution not expected (exclusive task of CNMCA) Slovakia/SHMÚ

8. Contribution not expected (exclusive task of CNMCA) Turkey/ITU

List of Tables

Table 01 - List of H-SAF products 08

Table 02 - Validation Team for precipitation products 11

Table 03 - List of ground data used for precipitation products validation in Belgium 18

Table 04 - Precipitation data available at BfG 22

Table 05 - Location of the 16 meteorological radar of the DWD 22

Table 06 - Characteristics of the three meteorological Doppler radars in Hungary 24

Table 07 - Accuracy requirements for product PR-ASS-1 [RMSE (%)] 39

Table 08 - Summary results of PR-ASS-1 validation in Belgium by IMR 40

Table 09 - Summary results of the validation of PR-ASS-1rate in Germany by BfG 42

Table 10 - Summary results of the validation of PR-ASS-1accumulated in Germany by BfG 44

Table 11 - Summary results of the validation of PR-ASS-1rate in Hungary by OMSZ 48

Table 12 - Summary results of the validation of PR-ASS-1accumulated in Hungary by OMSZ 51

Table 13 - Summary results of the validation of PR-ASS-1rate in Italy by UniFe 55

Table 14 - Summary results of the validation of PR-ASS-1accumulated in Italy by UniFe 55

Table 15 - Summary results of the validation of PR-ASS-1rate in Poland by IMWM 58

Table 16 - Summary results of the validation of PR-ASS-1accumulated in Poland by IMWM 58

Table 17 - Summary results of the validation of PR-ASS-1rate in Slovakia by SHMÚ 63

Table 18 - Summary results of the validation of PR-ASS-1accumulated in Slovakia by SHMÚ 64

Table 19 - Summary results of the validation of PR-ASS-1accumulated in Turkey by ITU and TSMS

over inner land

66

Table 20 - Summary results of the validation of PR-ASS-1accumulated in Turkey by ITU and TSMS

over coastal zones

66

Table 21 - Comparative results of validation in several Countries/Teams split by season. PR-

ASS-1 00 h (rain rate)

70

Table 22 - Comparative results of validation in several Countries/Teams split by season. PR-

ASS-1 accumulated precip.

71

Table 23 - Simplified compliance analysis for product PR-ASS-1 73

Table 24 - Synthesis of all validation results, including yearly average 74

List of Figures

Fig. 01 - Conceptual scheme of the EUMETSAT application ground segment 08

Fig. 02 - Current composition of the EUMETSAT SAF network (in order of establishment) 08

Fig. 03 - COSMO-ME model formulation and current area coverage 09

Fig. 04 - Cloud microphysical processes considered in the two-category ice scheme 10

Fig. 05 - Structure of the Precipitation products validation team 11

Fig. 06 - The network of 4100 rain gauges used for H-SAF precipitation products validation 12


Fig. 07 - The network of 40 C-band radar used for H-SAF precipitation products validation 13

Fig. 08 - Classes and sub-classes for evaluating Precipitation Rate products. Applicable to PR-

OBS-1, PR-OBS-2, PR-OBS-3, PR-OBS-4 and PR-ASS-1rate

15

Fig. 09 - Classes and sub-classes for evaluating Accumulated Precipitation products. Applicable

to PR-OBS-5 and PR-ASS-1accumulated

16

Fig. 10 - Meteorological radar in Belgium 18

Fig. 11 - RMI raingauges: daily ( ) and AWS ( ) 19

Fig. 12 - SETHY AWS network in Walloon Region 19

Fig. 13 - Left: Gaussian filter; right: sketch of the up-scaling procedure. The circle corresponds to

the range of the weather radar. The square in the middle is a common area such that it is

entirely included in the selected PR-OBS-2 files. The grey rectangle, the tilted dark grey

rectangle and the black ellipse are explained in the text

20

Fig. 14 - Left panel: radar coverage in Germany as of 01/03/2007. Right panel: location of

ombrometers for online calibration in RADOLAN; squares: hourly data provision

(about 500), circles: event-based hourly data provision (about 800 stations): red: AMDA

III, blue: aggregational network federal states (Bartels et al., 2004)

23

Fig. 15 - Flowchart of online calibration RADOLAN (adapted from Bartels et al. 2004) 23

Fig. 16 - The automatic rain gauge network in Hungary 24

Fig. 17 - Location and coverage of the three meteorological Doppler radars in Hungary 24

Fig. 18 - Map of SHMÚ raingauge stations: green – operational (98) , blue – climatological

(586), red - hydrological stations in H-SAF selected test basins (37). White points show

regular grid of experimental NOAA Snow water equivalent data

31

Fig. 19 - Example of Slovak radar network coverage - left circle corresponds to radar site Malý

Javorník, right one corresponds to Kojšovská hoľa 31

Fig. 20 - Example of 5-days cumulative precipitation constructed from raingauge measurements

by means of 3D extrapolation method 32

Fig. 21 - Map of SHMÚ hydrological stations: red - hydrological stations in H-SAF selected test

basins (34) 33

Fig. 22 - General hydrological model of HBV type 33 Fig. 23 - Example of pick up pixels from radar measurement for selected area 34 Fig. 24 - Example of separated ALADIN forecast pixel to smaller pixels 34 Fig. 25 - The Susurluk and Western Black Sea catchments selected for the precipitation product

validation in Turkey 35

Fig. 26 - Network of Meteorological stations in Susurluk (on the left) and Western Black Sea (on

the right) catchments 36

Fig. 27 - Position of 193 AWOS sites used for ground truth for the precipitation product

validation in western Turkey 36

Fig. 28 - H02 product footprint centres with a sample footprint area as well as the AWOS ground

observation sites 37

Fig. 29 - Meshed structure of the sample H02 product footprint 38

Fig. 30 - Mean Error of PR-ASS-1, 24h accumulated precipitation (monthly values over Belgium

in mm day-1) 40

Fig. 31 - Root Mean Square Error of PR-ASS-1, 24 accumulated precipitation (monthly values

over Belgium in mm day-1) 41

Fig. 32 - Time evolution of Mean Error and Standard Deviation for the three precipitation

categories 42

Fig. 33 - Time evolution of the Root Mean Square Error (mm/h and %) for the three precipitation

categories 43

Fig. 34 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success

Index 43

Fig. 35 - Time evolution of the Mean Error, function of the accumulation period 44

Fig. 36 - Time evolution of the Standard Deviation, function of the accumulation period 45

Fig. 37 - Time evolution of the Root Mean Square Error (mm), function of the accumulation 45


period

Fig. 38 - Time evolution of the Root Mean Square Error (%), function of the accumulation period 45

Fig. 39 - Time evolution of the Correlation Coefficient, function of the accumulation period 46

Fig. 40 - Time evolution of the Probability of Detection, function of the accumulation period 46

Fig. 41 - Time evolution of the False Alarm Rate, function of the accumulation period 46

Fig. 42 - Time evolution of the Critical Success Index, function of the accumulation period 47

Fig. 43 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for

medium and light precipitation 48

Fig. 44 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for

heavy precipitation 49

Fig. 45 - Time evolution of the Correlation Coefficient for the three categories of precipitation 49

Fig. 46 - Time evolution of Probability of Detection, False Alarm Rate and Critical Success

Index for rain rate 50

Fig. 47 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for the

integration intervals 3, 6, 12 and 24 hours 52

Fig. 48 - Time evolution of the Correlation Coefficient for the integration intervals 3, 6, 12 and

24 hours 52


Index for integration intervals 3, 6, 12 and 24 hours (four panels) 54

Fig. 50 - Probability Density Function for instantaneous rain rate and accumulation periods 3 h

and 24 h for the two months January and July 2009, for satellite-derived and ground

measurements

56

Fig. 51 - Time evolution of continuous statistical scores for the precipitation intensity 57

Fig. 52 - Time evolution of the Correlation Coefficient for instantaneous rain rate and integration

intervals 3 and 24 hours 57

Fig. 53 - Mean error (ME) of PR-ASS-1 v.1.0 rain rate product for the period of Jan 2009 – Mar

2010 for Poland 59

Fig. 54 - RMSE % of PR-ASS-1 v.1.0 rain rate product for the period of Jan 2009 – Mar 2010

for Poland 59

Fig. 55 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-

ASS-1 v.1.0 rain rate product using Polish RG data in the period of Jan 2009 – Mar

2010

60

Fig. 56 - Mean error (ME) of PR-ASS-1 3 h cumulated precipitation v.1.0 for the period of

Jan 2009 – Mar 2010 for Poland 61

Fig. 57 - RMSE (%) of PR-ASS-1 3-hour cumulated precipitation v.1.0 for the period of

Jan 2009 – Mar 2010 for Poland 61

Fig. 58 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-

ASS-1 3-hour cumulated precipitation v.1.0 product using Polish RG data for the period

of Jan 2009 – Mar 2010

62

Fig. 59 - Time evolution of Mean Error, Root Mean Square Error (%) and dichotomous scores for

rain rate 63

Fig. 60 - Time evolution of Mean Error and Root Mean Square Error (%), function of the

integration interval 3, 6, 12 and 24 h 65


Index, function of the integration interval (3, 6, 12 and 24 h) 65

Fig. 62 - Continuous and multi-categorical statistics for inner land (3 and 24 hourly

accumulation) 67

Fig. 63 - Continuous and multi-categorical statistics for coastal zones (3 and 24 hourly

accumulation) 67

Fig. 64 - Correlation coefficients for inner land and coastal zones (3 and 24 hourly accumulation) 67


Acronyms [not including those in the Appendix]

ACC Fraction correct Accuracy

AMSU-B Advanced Microwave Sounding Unit - B (on NOAA up to 17)

ATDD Algorithms Theoretical Definition Document

AU Anadolu University (in Turkey)

AWOS Automated Weather Observing Stations

AWS Automatic Weather Station

BfG Bundesanstalt für Gewässerkunde (in Germany)

CAF Central Application Facility (of EUMETSAT)

CC Correlation Coefficient

CESBIO Centre d'Etudes Spatiales de la BIOsphere (of CNRS, in France)

CM-SAF SAF on Climate Monitoring

CNMCA Centro Nazionale di Meteorologia e Climatologia Aeronautica (in Italy)

CNR Consiglio Nazionale delle Ricerche (of Italy)

CNRS Centre Nationale de la Recherche Scientifique (of France)

COSMO Consortium for Small-Scale Modelling

COSMO-ME Consortium for Small-Scale Modelling - version for Mediterranean

CSI Critical Success Index

DPC Dipartimento Protezione Civile (of Italy)

DWD Deutscher Wetterdienst

DWR Dry to Wet Ratio

ECMWF European Centre for Medium-range Weather Forecasts

ETS Equitable Threat Score

EUM Short for EUMETSAT

EUMETSAT European Organisation for the Exploitation of Meteorological Satellites

FAR False Alarm Rate

FBI Frequency BIas

FMI Finnish Meteorological Institute

GEO Geostationary Earth Orbit

GIS Geographic Information System

GPM Global Precipitation Measurement mission

GRAS-SAF SAF on GRAS Meteorology

HAWK Hungarian Advanced Weather Workstation

HBV Hydrologiska Byrans Vattenbalansavdelning (hydrological model)

HRON Hydrological model for the Hron basin

H-SAF SAF on Support to Operational Hydrology and Water Management

HSS Heidke skill score

IFOV Instantaneous Field Of View

IMWM Institute of Meteorology and Water Management (in Poland)

IPF Institut für Photogrammetrie und Fernerkundung (of TU-Wien, in Austria)

IR Infra Red

IRM Institut Royal Météorologique (of Belgium) (alternative of RMI)

ISAC Istituto di Scienze dell‟Atmosfera e del Clima (of CNR, Italy)

ITU İstanbul Technical University (in Turkey)

LATMOS Laboratoire Atmosphères, Milieux, Observations Spatiales (of CNRS, in France)

LEO Low Earth Orbit

LSA-SAF SAF on Land Surface Analysis

ME Mean Error

Météo France National Meteorological Service of France

METU Middle East Technical University (in Turkey)

MSG Meteosat Second Generation

MTF Modulation Transfer Function


MW Micro-Wave

NMA National Meteorological Administration (of Romania)

NOAA National Oceanic and Atmospheric Administration (Agency and satellite)

NWC Nowcasting

NWC-SAF SAF in support to Nowcasting & Very Short Range Forecasting

NWP Numerical Weather Prediction

NWP-SAF SAF on Numerical Weather Prediction

O3M-SAF SAF on Ozone and Atmospheric Chemistry Monitoring

OMSZ Hungarian Meteorological Service

ORR Operations Readiness Review

OSI-SAF SAF on Ocean and Sea Ice

Pixel Picture element

POD Probability of Detection

POFD Probability Of False Detection

PUM Product User Manual

PVR Product Validation Report

REP-3 H-SAF Products Valiadation Report

RMI Royal Meteorological Institute (of Belgium) (alternative of IRM)

RMS Root Mean Square

RMSE Root Mean Square Error

SAF Satellite Application Facility

SD Standard Deviation

SETHY Service d'Ètudes Hydrologiques, Walloon Ministry of Public Works - Belgium SEVIRI Spinning Enhanced Visible and Infra-Red Imager (on Meteosat from 8 onwards)

SHMÚ Slovak Hydro-Meteorological Institute

SRE Scale Recursive Estimation

SYKE Suomen ympäristökeskus (Finnish Environment Institute)

TB or TB Brightness Temperature (used for MW and, inappropriately, for IR)

TKK Teknillinen korkeakoulu (Helsinki University of Technology)

TSMS Turkish State Meteorological Service

TU-Wien Technische Universität Wien (in Austria)

UniFe University of Ferrara (in Italy)

UTC Universal Coordinated Time

ZAMG Zentralanstalt für Meteorologie und Geodynamik (of Austria)


1. The EUMETSAT Satellite Application Facilities and H-SAF

The “EUMETSAT Satellite Application Facility on Support to Operational Hydrology and Water

Management (H-SAF)” is part of the distributed application ground segment of the “European

Organisation for the Exploitation of Meteorological Satellites (EUMETSAT)”. The application ground

segment consists of a “Central Application Facility (CAF)” and a network of eight “Satellite

Application Facilities (SAFs)” dedicated to development and operational activities to provide satellite-

derived data to support specific user communities. See Fig. 01.

Fig. 01 - Conceptual scheme of the EUMETSAT application ground segment.

Fig. 02 reminds the current composition of the EUMETSAT SAF network (in order of establishment).

Nowcasting & Very

Short Range Forecasting Ocean and Sea Ice Ozone & Atmospheric Chemistry Monitoring Climate Monitoring Numerical Weather

Prediction GRAS Meteorology Land Surface Analysis Operational Hydrology

& Water Management

Fig. 02 - Current composition of the EUMETSAT SAF network (in order of establishment).

The H-SAF was established by the EUMETSAT Council on 3 July 2005; its Development Phase started

on 1st September 2005 and ends on 31 August 2010. The list of H-SAF products is shown in Table 01.

Table 01 - List of H-SAF products

Code Acronym Product name

H01 PR-OBS-1 Precipitation rate at ground by MW conical scanners (with indication of phase)

H02 PR-OBS-2 Precipitation rate at ground by MW cross-track scanners (with indication of phase)

H03 PR-OBS-3 Precipitation rate at ground by GEO/IR supported by LEO/MW

H04 PR-OBS-4 Precipitation rate at ground by LEO/MW supported by GEO/IR (with flag for phase)

H05 PR-OBS-5 Accumulated precipitation at ground by blended MW and IR

H06 PR-ASS-1 Instantaneous and accumulated precipitation at ground computed by a NWP model

H07 SM-OBS-1 Large-scale surface soil moisture by radar scatterometer

H08 SM-OBS-2 Small-scale surface soil moisture by radar scatterometer

H09 SM-ASS-1 Volumetric soil moisture (roots region) by scatterometer assimilation in NWP model

H10 SN-OBS-1 Snow detection (snow mask) by VIS/IR radiometry

H11 SN-OBS-2 Snow status (dry/wet) by MW radiometry

H12 SN-OBS-3 Effective snow cover by VIS/IR radiometry

H13 SN-OBS-4 Snow water equivalent by MW radiometry

Decentralised processing

and generation of products

EUM Geostationary Systems

Systems of the EUM/NOAA

Cooperation

Centralised processing

and generation of products

Data Acquisition and Control

Data Processing EUMETSAT HQ

Meteorological Products Extraction

EUMETSAT HQ

Archive & Retrieval Facility (Data Centre)

EUMETSAT HQ

USERS

Application Ground Segment

other data sources

Satellite Application

Facilities (SAFs)


2. Introduction to product PR-ASS-1

2.1 Principle of the product

Product PR-ASS-1 (Instantaneous and accumulated precipitation at ground computed by a NWP

model) is the output of the operational COSMO-ME NWP model in use at CNMCA. Its main

characteristics are shown in Fig. 03.

Model Equations

Basic hydro-thermodynamical equations in advection form,

no scale approximations (i.e. fully compressible and non-hydrostatic),

substraction of horizontally homogeneous basic state at rest.

Prognostic Variables

Horizontal and vertical wind components (u,v,w)

Temperature (T)

Pressure perturbation (p', deviation from the reference state)

Specific humidity (qv),

Specific cloud water content (qc)

Specific cloud ice content (qi)

Specific rain content (qr)

Specific snow content (qs)

Turbulent kinetic energy (tke)

Optionally, specific graupel content (qg)

Coordinate System

Rotated geographical (lat/lon) coordinate system horizontally,

Generalized terrain-following height-coordinate vertically.

Grid Structure

Arakawa C-grid, Lorenz vertical grid staggering

Spatial Discretization

Second-order horizontal and vertical differencing (centred)

Time Integration

3 time-level (Leapfrog) split explicit using extensions proposed by Skamarock and Klemp (1992).

Additional Options: - 2 time-level Runge-Kutta 3rd-order scheme (regular or TVD) with

various options for high-order spatial discretization (Forstner and Doms, 2004),

- 3 time-level semi-implicit scheme (Thomas et al., 2000), - 2 time-level Runge-Kutta 2nd order split-explicit scheme (Wicker and

Skamarock, 1998).

Numerical Smoothing

Rayleigh damping layer at upper boundary

4th order linear horizontal diffusion with option for a monotonic version including an orographic limiter,

3-D divergence damping and off-centering in split steps

Lateral Boundaries

1-way nesting using the lateral boundary formulation to Davies and Turner (1977).

Initialization

Diabatic digital filtering initialization scheme (Lynch et al. 1997)

Grid-Scale Clouds and Precipitation

Cloud water formation dissipation by saturation adjustment.

Precipitation formation by a bulk-parameterization including water vapour, cloud water, cloud ice, rain and snow as hydrometeor categories (Doms 2002; Baldauf and Schultz 2004; Reinhardt and Seifert 2006).

Subgrid-Scale Clouds

Subgrid-scale cloudiness (fractional cloud cover) is interpreted by an empirical function depending on relative humidity. A corresponding cloud water content is also diagnosed.

Moist Convection

Mass-flux convection scheme after Tiedtke (1989) with closure based on moisture convergence.

Option for a modified closure based on CAPE.

Radiation

δ-two stream radiation scheme based on Ritter and Geylen (1989) for short- and longwave fluxes; full cloud-radiation feedback.

Turbulent Diffusion

Level 2.5-scheme with a prognostic treatment of turbulent kinetic energy; effects of subgrid-scale condenstation and evaporation are included.

Optionally, diagnostic K-closure (at hierarchy level 2) for vertical diffusion.

Subgrid-scale orography

Blocked flow and gravity wave drag (Lott and Miller, 1997).

Surface Layer

Scheme based on turbulent kinetic energy; includes effects from subgrid-scale thermal circulations.

Optionally, constant flux layer parameterization.

Soil Processes

Multi-layer soil model including freezing of soil water (Schrodin and Heise, 2001).

Optionally, two-layer soil model after Jacobsen and Heise (1984) with Penman-Monteith type transpiration. Snow and interception storage are included. Climate values changing monthly (but fixed during forecast) in third layer.

Current area coverage

Fig. 03 - COSMO-ME model formulation and current area coverage.

The role of PR-ASS-1 is to provide a background precipitation field regular in space and time, unlike

satellite-derived observations that are available at changing times and locations, depending on the

specific orbit. The product has been developed, is operationally running and is being progressively

improved at CNMCA. It is a “best effort” product: e.g., the covered area (as shown in Fig. 03), and the

number of runs/day will not meet H-SAF requirement by the end of the Development Phase; but it is


fully operational product. The output consists of five figures at 3-hour intervals: the rate and the

accumulated precipitation over 3. 6. 12 and 24 hours.

For more information, please refer to the Products User Manual (specifically, PUM-06).

2.2 Algorithm principle

The baseline algorithm for PR-ASS-1 processing is described in ATDD-06. Only essential elements are

highlighted here. Fig. 04 illustrates the module that most concerns the precipitation products.

2.3 Main operational characteristics

The operational characteristics of PR-ASS-1 are discussed in PUM-06. Here are the main highlights.

The horizontal resolution ( x), for a product generated by a NWP model, depends on the scale of

motion correctly represented in the model. For COSMO-ME, as concerns precipitation products, it is

estimated as ~ 30 km. The sampling interval consist of the model grid mesh ~ 7 km. Thus:

resolution x ~ 30 km - sampling distance: 7 km.

The observing cycle ( t). All products are outputted at 3-hour intervals. This is the sampling interval

for the product). However, they derive from forecasting cycles that currently are run at 00 and 12 UTC,

therefore the output products are correlated. It is correct to quote an observing cycle ~ 12 h (to be

reduced to t ~ 6 h in the near future, when runs at 06 and 18 UTC will be added). Thus:

observing cycle t ~ 12 h - sampling time: 3 h.

The timeliness ( ), for a product from a NWP process, is intended as the difference between the nominal

time of the run start and the availability of forecast products, inclusive of the window cut-off for

observation collection, analysis, initialisation, processing and output stabilisation. For COSMO-ME we

currently have:

timeliness ~ 4 h.

The accuracy (RMS) is the convolution of model features and observational data availability and

quality. Its evaluation is the task of the validation activity.

Fig. 04 - Cloud microphysical processes considered in the two-category ice scheme.


3. Validation strategy, methods and tools

3.1 Validation team and work plan

Whereas the previous operational characteristics have been evaluated on the base of system

considerations (number of satellites, their orbits, access to the satellite) and instrument features (IFOV,

swath, MTF and others), the evaluation of accuracy requires validation, i.e. comparison with the ground

truth or with something assumed as “true”. PR-ASS-1, as any other H-SAF product, has been submitted

to validation entrusted to a number of institutes (see Fig. 05).

Precipitation products validation group

Leader: Italy (DPC)

Belgium IRM

Germany

BfG

Hungary OMSZ

Italy

UniFe

Poland IMWM

Slovakia

SHMÚ

Turkey ITU

Fig. 05 - Structure of the Precipitation products validation team.

Table 02 lists the persons involved in the validation of H-SAF precipitation products

Table 02 - Validation Team for precipitation products

Silvia Puca (Leader) Dipartimento Protezione Civile (DPC) Italy [email protected]

Emmanuel Roulin Institut Royal Météorologique (IRM) Belgium [email protected]

Angelo Rinollo Institut Royal Météorologique (IRM) Belgium [email protected]

Thomas Maurer Bundesanstalt für Gewässerkunde (BfG) Germany [email protected]

Peer Helmke Bundesanstalt für Gewässerkunde (BfG) Germany [email protected]

Eszter Lábó Hungarian Meteorological Service (OMSZ) Hungary [email protected]

Federico Porcu' Ferrara University, Department of Physics (UniFe) Italy [email protected]

Bozena Lapeta Institute of Meteorology and Water Management (IMWM) Poland [email protected]

Ján Kaňák Slovenský Hydrometeorologický Ústav (SHMÚ) Slovakia [email protected]

Ľuboslav Okon Slovenský Hydrometeorologický Ústav (SHMÚ) Slovakia [email protected]

Ahmet Öztopal Istanbul Technical University (ITU) Turkey [email protected]

Ibrahim Sonmez Turkish State Meteorological Service (TSMS) Turkey [email protected]

The Precipitation products validation programme started with a first workshop in Rome, 20-21 June

2006, soon after the H-SAF Requirements Review (26-27 April 2006). The first activity was to lay

down the Validation plan, that was finalised as early as 30 September 2006, i.e. about one year after the

start of the H-SAF Development Phase. After the first Workshop, other ones followed, at roughly

yearly intervals, often joined with the Hydrological validation group.

At the 1st H-SAF Workshop (Rome,16-18 October 2007), a first set of significant validation exercises

was presented. An internal document, called REP-3 (H-SAF Products Validation Report) started being

compiled since then. Now, moving to the end of the H-SAF Development Phase, REP-3 has been

restructured into this Product Validation Report (PVR) split into 13 volumes, one for each H-SAF

product. The validation experiments recorded in REP-3 constitute “Appendixes” to the various

volumes. Because of the initial aim of REP-3 (internal document at working level) the editorial level of

the Appendixes is of rather low standard.

3.2 Validation philosophy

3.2.1 Objective and problems

The products validation activity has to serve multiple purposes:

most urgent, to provide input to the product developers for improving calibration for better quality

of baseline products, and for guidance in the development of more advanced products;


also urgent, to characterise the products error structure in order to enable the Hydrological

validation programme to appropriately use the data; the Education & Training programme, part of

the Hydrological validation programme, was particularly instrumental for this;

to build up the background information necessary for online quality control of the products before

distribution;

in general, to enable attaching the necessary information on error structure to accompany H-SAF

products distribution in an open environment, after the initial phase of distribution limited to the so-

called “beta users”.

Validation is obviously a hard work in the case of precipitation, both because the sensing principle from

space is very much indirect, and because of the natural space-time variability of the precipitation field

(sharing certain aspects with fractal fields), that places severe sampling problems. It is known that an

absolute „ground truth‟ does not exist. For the performance evaluation of the H-SAF precipitation

products the radar and rain gauge measurements have been assumed as „ground truth‟. This is due to the

large use and experience of these data by the hydrologists, the main users of the products. Comparison

with results of numerical models obviously suffer of the incompatible scales between the natural

phenomenon and the model (for hydrostatic NWP models) or the limits of atmospheric predictability

when entering the scale of convection (for Cloud Resolving Models). A mixture of all this techniques is

generally used, and the results change with the climatic situation and the type of precipitation. It is

therefore necessary a European cooperation for this programme.

3.2.2 Tools to be used for validation

The areas chosen for the validation task include the basins where the hydrological validation is

performed. The data used for the validation of the satellite precipitations products are:

Ground data:

- automatic rain gauges with different time resolution: 5 min, 10 min, 15 min, 30 min;

- meteorological radars with different time resolution: 5 min, 10 min, 30 min.

Data for cloud types classification, containing information about water content in vertical column

and for the discrimination of the synoptic situation are also foreseen. The main products used to

derived these information are: products from the Nowcasting SAF; output from NWP models;

SEVIRI composite images.

Fig. 06 provides a view of the raingauge network used for precipitation products validation in H-SAF.

Fig. 06 - The network of 4100 rain gauges used for H-SAF precipitation products validation.


Fig. 07 provides a view of the radar network utilised for precipitation products validation in H-SAF.

Fig. 07 - The network of 40 C-band radar used for H-SAF precipitation products validation.

3.2.3 Techniques to bring observations comparable

Due to the time and space structure of precipitation and to the sampling characteristics of both the

precipitation products and ground data used for validation, care has to be taken to bring data

comparable. At a given place, precipitation occurs intermittently and at highly fluctuating rates. Over

space, precipitation is distributed with a high variability, in cells of high intensity nested in larger area

with lower rain rate. Aimed at observing this complex phenomenon, the satellite-based products are

defined with a spatial resolution of several kilometres and with different sampling rate. On the other

hand, reference ground data used to validate precipitation data from satellite are also characterized by

their own spatial resolution ranging from point information measured on rain-gauge networks to grids

with cells of several hundreds of meters to several kilometres for weather radar. Furthermore, none of

these reference observations are without error. For this reason it was decided to compare the satellite

data with ground data on the satellite product native grid. All the institutes applied the same up-scaling

method to compare the satellite precipitation estimations with ground data.

There are several approach to bring the observation comparable. The simplest consists in comparing

untransformed data, e.g. comparing areal data to observations at a nearest gauge station, or

instantaneous images with information available within a time window. Doing so, part of the error has

to be attributed to the differences between sample volumes: this “representativeness error” may be

estimated by using high spatial and temporal resolution gauge data (e.g. Kitchen and Blackall 1992)1 or

may be simulated in numerical experiments (e.g. Tustison et al. 2001)2.

An alternative approach consists in upscaling reference observations to areal averages corresponding to

the resolution of the precipitation products but in an equal-area map projection. For rain-gauge data,

this step requires the use appropriate interpolation scheme (e.g. Thiessen polygons, kriging, etc.). For

radar images, it requires to average the values measured at radar pixels included in each of the product

1 Kitchen M. and R.M. Blackall, 1992: “Representativeness errors in comparisons between radar and gauge

measurements of rainfall”. J. Hydrol., 134, 13-33. 2 Tustison B., D. Harris and E. Foufoula-Georgiou, 2001: “Scale issues in verification of precipitation forecasts”. J.

Geophys. Res., 106, 11,775-11,784.


pixel. For cumulate products, data from radar images have to be further integrated over the time

intervals and using advection procedures to correct the effect of time sampling.

The Scale Recursive Estimation (SRE) (Primus et al. 20013, Tustison et al. 20034, Gupta et al. 20065)

can be used for the validation since observations are available at one ore more scales different than the

scale of the H-SAF products. This methodology consists in filtering noisy observations taking into

account the scale-dependant variability and the nested spatial structure of precipitation. It provides

optimal estimates of precipitation at the desired scale i.e. unbiased and with the minimum variance as

well as it gives information about uncertainty at that scale. Nevertheless, it may require some

resampling of the data to make it compatible with a cascade structure.

There is a trade-off to be found between pooling the data in space and time in order to have a validation

sample large enough and stratifying into sub-samples with comparable situations so as to avoid that the

performance results be biased towards the dominant regime. The validation data may be separated in

seasons, night and day, sea and land, geographical regions, rainfall intensity and cloud or precipitation

type.

As mentioned before for the validation exercises inside this project the radar and rain gauge data were

up-scaled taking into account the satellite scanning geometry and IFOV resolution of AMSU-B scan,

SSMI and SEVIRI. Radar and rain gauge instruments provide many measurements within a single

satellite IFOV, those measurements were averaged following the satellite antenna pattern of AMSU-B,

SSMI and SEVIRI. This activity was developed in collaboration with the precipitation product

developers.

Two codes were developed by the validation group for upscaling ground data data vs AMSU-B and

SSMI IFOV. All institutes involved in precipitation product validation activity uses these two codes

developed by University of Ferrara and RMI6.

About the SEVIRI data a common code was not developed, but all institutes involved in precipitation

product validation activity uses the same up-scaling technique which was indicated by CNR-ISAC. A

common code will be developed in CDOP.

3.2.4 Structuring the results of the validation activity

During the development phase a twofold validation strategy was applied: one based on large statistics

(multi-categorical and continuous), and one on selected case studies. Both components were, and still

are, considered complementary in assessing the accuracy of the implemented algorithms. Large

statistics help in identifying existence of pathological behaviour, selected case studies are useful in

identifying the roots of such behaviour where present.

Common validation

To produce a large statistical analysis of the H-SAF Precipitation Products it was necessary to define a

„common validation methodology’ in order to make comparable the results obtained by several

institutes and to better understand their meanings.

To achieve these goal it was necessary:

standardization of the up-scaling techniques of radar and rain gauge data vs AMSU, SSMI and

SEVIRI data,

introduction of quality filter,

3 Primus I., D. McLaughlin and D. Enthekabi, 2001: “Scale-recursive assimilation of precipitation data”. Adv. Water

Resour., 24, 941-953. 4 Tustison B., E. Foufoula-Georgiou and D. Harris, 2003: “Scale-recursive estimation for multisensor Quantitative

precipitation Forecast verification: a preliminary assessment”. J. Geophys. Res., 108, 8377-8390. 5 Gupta R., V. Venugopal and E. Foufoula-Georgiou, 2006: “A methodology for merging multisensor precipitation

estimates based on expectation-maximization and scale-recursive estimation”. J. Geophys. Res., 111, D02102,

doi:10.1029/2004JD005568. 6 Van de Vyver H. and E. Roulin, 2009: “Scale recursive estimation for merging precipitation data from radar and

microwave cross-track scanners”. J. Geophys. Res., 114, D08104, doi: 10.1029/2008JD010709.


development and sharing of software packages.

The Common Validation Methodology is based on comparisons with rain gauges and radar data to

produce monthly Continuous verification and Multi-Categorical statistic scores for sea, land and coast

area.

The main steps are:

all the institutes compare the national radar and rain gauge data with the precipitation values

estimated by satellite on the satellite native grid using the same up-scaling techniques;

all the institutes evaluate the monthly continuous scores (below reported) and contingency tables for

the precipitation classes (below reported) producing numerical files called „CS‟ and „MC‟ files;

all the institutes evaluate PDF producing numerical files called „DIST‟ files and plots;

the precipitation product validation leader collect all the validation files (MC, CS and DIST files),

verify the consistency of the results and evaluate the monthly common statistical results;

The results obtained were:

discussed inside the validation group and with product developers by email and two annual

meetings,

reported in the project document,

published in the H-SAF web page.

Case studies

Each Institute, in addition to the common validation methodology, developed a more specific

Validation Methodology based on the knowledge and experience of the Institute itself. This activity is

focused on case studies analysis. Each institute decides whether to use ancillary data such as lightning

data, SEVIRI images, the output of numerical weather prediction and nowcasting products.

The main steps are:

description of the meteorological event;

comparison of ground data and satellite products;

visualization of ancillary data deduced by nowcasting products or lightning network;

discussion of the satellite product performances;

indications to Developers;

making the ground data (if requested) available to satellite product developers.

The results obtained were:

discussed inside the validation group and with product developers by email and two annual

meetings,

reported in the project document,

published in the H-SAF web page.

Subdivision in classes

Since the accuracy of precipitation measurements depends on the type of precipitation or, to simplify

matters, the intensity, the verification is carried out split in more classes. For intensity, user

requirements have been expressed for three classes; however, for working purposes, finer subdivision in

11 sub-classes is used (see Fig. 08).

Class 1 2 3

< 1 mm/h (light precipitation) 1 - 10 mm/h (medium precipitation) > 10 mm/h (intense precipitation)

Subclass 1 2 3 4 5 6 7 8 9 10 11

(mm/h) < 0.25 0.25-0.5 0.5 - 1.0 1.0 - 2.0 2.0 - 4.0 4.0 - 8.0 8.0 - 10 10 - 16 16 - 32 32 - 64 > 64

Fig. 08 - Classes and sub-classes for evaluating Precipitation Rate products. Applicable to PR-OBS-1, PR-OBS-2, PR-OBS-3, PR-OBS-4 and PR-ASS-1rate


For accumulated precipitation, user requirements are unclear in terms of dependence on amount. We

have adopted a 5-class splitting for results presentation and a 10-subclass subdivision for working

purpose (see Fig. 09).

Class 1 2 3 4 5

< 8 mm 8 - 32 mm 32-64 mm 64-128 mm > 128 mm

Subclass 1 2 3 4 5 6 7 8 9 10

(mm) < 1 1 - 2 2 - 4 4 - 8 8 - 16 16 - 32 32 - 64 64 - 128 128 - 256 > 128

Fig. 09 - Classes and sub-classes for evaluating Accumulated Precipitation products. Applicable to PR-OBS-5 and PR-ASS-1accumulated

The evaluation of the statistical scores split by precipitation classes allows to analyse the product

performances not only for precipitation mean values (light precipitation being the more frequent) but

also for higher value, the most interesting for Hydrology.

Each Institute, in addition to the common validation methodology, developed a more specific

Validation Methodology based on the knowledge and experience of the Institute itself. This activity is

focused on case studies analysis. Each institute decides whether to use ancillary data such as lightning

data, SEVIRI images, the output of numerical weather prediction and nowcasting products. Specific

validation activities are indicated by each partner in the appropriate section of Chapter 3.4.

3.3 Definition of statistical scores

It is appropriate to deploy the definitions of the statistical scores utilised in H-SAF product validation

activities. Some apply to “continuous statistics”, some to “dichotomous statistics”. Although neither

rain gauges nor radar constitute a very accurate ground truth, we assume as “true” these observations,

thus the departures of satellite observations will be designated as “errors”

Scores for continuous statistics:

- Mean Error (ME) or Bias

- Standard Deviation (SD)

- Correlation Coefficient (CC)

- Root Mean Square Error (RMSE)

- Root Mean Square Error percent (RMSE %), used for precipitation since error grows with rate.

N

1k

kk )true(satN

1biasorME

N

1k

2

kk MEtruesatN

1SD

N

1k

N

1

2

k

2

k

N

1k

kk

truetruesatsat

truetruesatsat

CC with N

1k

ksatN

1sat and

N

1k

ktrueN

1true ;

N

1k

2

kk truesatN

1RMSE

N

1k

2

k2

kk

true

truesat

N

1%RMSE


Scores for dichotomous statistics

Stemming from the contingency Table: Contingency Table

Observed (ground)

yes no total

yes hits false alarms forecast yes

Forecast (satellite) no misses correct negatives forecast no

total observed yes observed no total

where:

- hit: event observed from the satellite, and also observed from the ground

- miss: event not observed from the satellite, but observed from the ground

- false alarm: event observed from the satellite, but not observed from the ground

- correct negative: event not observed from the satellite, and also not observed from the ground.

A large variety of scores have been defined. The following are used in H-SAF

- Frequency BIas (FBI)

- Probability Of Detection (POD)

- False Alarm Rate (FAR)

- Probability Of False Detection (POFD)

- Fraction correct Accuracy (ACC)

- Critical Success Index (CSI)

- Equitable Threat Score (ETS)

- Heidke skill score (HSS)

- Dry-to-Wet Ratio (DWR).

yesobserved

yesforecast

misseshits

alarmsfalsehitsFBI Range: 0 to ∞. Perfect score: 1

yesobserved

hits

misseshits

hitsPOD Range: 0 to 1. Perfect score: 1

yesforecast

alarmsfalse

alarmsfalsehits

alarmsfalseFAR Range: 0 to 1. Perfect score: 0

noobserved

alarmsfalse

alarmsfalsenegativescorrect

alarmsfalsePOFD Range: 0 to 1. Perfect score: 0

total

negativescorrecthitsACC Range: 0 to 1. Perfect score: 1

alarmfalsemisseshits

hitsCSI Range: 0 to 1. Perfect score: 1

random

random

hitsalarmfalsemisseshits

hitshitsETS with

total

yesforecastyesobservedhitsrandom

ETS ranges from -1/3 to 1. 0 indicates no skill. Perfect score: 1.

random

random

correct)pected(exN

correct)pected(exnegatives)correct(hitsHSS with

no)edno)(observ(forecastyes)astyes)(forec(observedN

1correct)pected(ex random

HSS ranges from -1 to 1. 0 indicates no skill. Perfect score: 1.

yesobserved

no observed

misseshits

negative correctalarm falseDWR Range: 0 to ∞. Perfect score: n/a.


3.4 Inventory of validation facilities

In the following sections the facilities utilised in the various Institutes to perform validation of

precipitation products are described. It is apologised that editing is not well homogenised since the

various sections are recorded as they were contributed by the individual institutes, with minimum

harmonisation effort in respect of length and level of detail.

3.4.1 Facilities in Belgium (IRM)

Ground data

The validation results for Belgium presented in this report were obtained by comparison of the rain rates

products with weather radar data and of the cumulated precipitation products with either cumulated

weather radar data or rain gauge data. Table 03 summarizes the ground data used as well as the domain

over which the validation extends. The last row has been included but refers to results to be presented in

the report on hydrological validation.

Table 03 - List of ground data used for precipitation products validation in Belgium

Product Ground data Validation domain

PR-OBS-1 MW Conical Wideumont Radar 230 km 230 km

PR-OBS-2 MW Cross-Track Wideumont Radar 230 km 230 km

PR-OBS-3 IR+MW Rapid Update Wideumont Radar 230 km 230 km

PR-OBS-5 Cumulated 24h Cum. Wideumont Radar 230 km 230 km

PR-OBS-5 Cumulated 24h SETHY Raingauges Walloon Region

PR-OBS-5 Cumulated 24h RMI Daily Raingauges Test Catchments

Weather Radar

Belgium is well covered with three radars (see Fig. 10). A further radar is currently under construction

in the coastal region. These are Doppler, C-band, single polarization radars with beam width of 1° and a

radial resolution of 250 m. Data are available at 0.6, 0.66 and 1 km horizontal resolution for the

Wideumont, Zaventem and Avesnois radars respectively.

In this report, only the Wideumont radar has been used. The data of this radar are controlled in three

steps. First, a long-term verification is performed as the mean ratio between 1-month radar and gauge

accumulation for all gauge stations at less than 120 km from the radar. The second method consists in

fitting a second order polynomial to the mean 24 h (8 to 8 h local time) radar / gauge ratio in dB and the

Fig. 10 - Meteorological radar in Belgium.


range; only the stations within 120 km and where both radar and gauge values exceed 1 mm are taken

into account. The third method is the same as the second but is performed on-line using the 90

telemetric stations of the SETHY (Ministry of the Walloon Region). Corrected 24 h images are then

calculated. New methods for the merging of radar and raingauge data have been recently evaluated

(Goudenhoofdt and Delobbe 2009)7.

Raingauge

Several raingauge networks are managed in Belgium (see Fig. 11). RMI has a dense network of daily

raingauge and an increasing network of automatic weather stations equipped with tipping bucket

gauges. Other networks are operated by the Regional Authorities in charge of rivers. For the validation

of the PR-OBS-5, we have used hourly data from the SETHY raingauge which are quality controlled

daily at RMI. The daily data are gathered and checked with 1.5 to 2 month delay. These later data are

mainly used in the hydrological validation programme.

For the validation of the PR-OBS-5 cumulated rainfall product, a validation with raingauge data has

been performed, in parallel to the radar validation. The reference data used are hourly rain gauge records

from the SETHY (Walloon Region) network (Fig. 12). The network includes 89 automatic non-heated

stations and 3 heated stations (in coincidence with non-heated ones). Only the non-heated stations have

been considered, for the sake of uniformity. The data have been interpolated in onto a 5 km 5 km grid,

following the Barnes method. The sensitivity parameter in the Barnes procedure has been set to 108,

considering the fact that the mean distance between every station and its closest neighbor is roughly 104

m. The interpolation procedure is iterative. If the mean squared difference between the source field and

the interpolated field falls below 0.01 mm h-1, or if the improvement is below 1% between two steps, the

procedure is stopped, otherwise it goes on for a maximum 20 iterations. The result is a series of files

with interpolated data, one per hour.

The quality of the interpolated data has been checked for several months in the following way: the

interpolation is calculated taking into account all the stations except one, and the value corresponding to

the missing station is estimated. The procedure is repeated for all the stations. A set of 89 reconstructed

values is obtained, and compared with the measured data. The verification refers to the period from

August to November 2008. The interpolation is first assessed in its capacity to reconstruct the rain / no

rain field. Taking 0.01 mm h-1 as a threshold, the probability of correct rain (POD) is 0.79, the false

reconstruction (FAR) is 0.07 and the equitable threat score (ETS) is 0.71. Then, statistical scores are

calculated on a monthly basis. The bias ranges from 0.06 to 0.14 mm h-1, the root mean square from

7 Goudenhoofdt E. and L. Delobbe, 2009: “Evaluation of radar-gauge merging methods for quantitative precipitation

estimates”. Hydrol. Earth Syst. Sci., 13, 195-203.

Fig. 11 - RMI raingauges: daily ( ) and AWS ( ). Fig. 12 - SETHY AWS network in Walloon Region.


0.37 to 1.00 mm h-1 and the mean relative error from 0.09 to 0.19 for mean observed values of 0.50 to

1.00 mm h-1.

As preliminary test for the hydrological validation of PR-OBS-5, the data of the daily raingauge stations

have been interpolated using the Thiessen polygons method and spatially averaged over the two test

catchments. The values obtained have been compared with the corresponding cumulated values from

satellite.

Miscellaneous information

For the analysis of test cases, additional information has been used like the cloud types identified using

the SAF-NWC tools, the expertise of weather forecasters to select and analyze the synoptic conditions,

the SAFIR maps of lightning impacts.

Methodology

From a local point of view, rain rates products based on microwave sensors onboard of Low Earth Orbit

satellites are characterized by a varying coverage and projection. To make the statistics comparables

from one file to the other, a validation domain has been defined which is a square of 230 km 230 km

centered on the Wideumont radar location and only the products covering entirely this common area

have been considered. To be more precise, for every product file, a sub-set of lines and columns

including the common square has been extracted. Then, the radar data have been up-scaled to the

projection of the sub-set of pixels and compared with the product estimates.

The up-scaling of the radar data is performed taking the footprints of the microwave sensors into

account. In Fig. 13, the Gaussian filter corresponding to the first scan position of the AMSU-B antenna

(PR-OBS-2) is represented on the left. The filtering procedure is organized as follows (see the figure, on

the right). First, a part of the radar image is selected (grey). Then, the radar data (0.6 km resolution are

re-sampled onto a tilted grid (2 km resolution) where the Gaussian filter is 1% of maximum (dark

grey). Tilting depends on the scan position and on the satellite overpass mode. Finally the Gaussian

filter is applied. The black ellipse corresponds to half power. Additional information about the up-

scaling equations and about the tilting of the PR-OBS-2 pixels can be found in Van de Vyver and

Roulin (2008).

For PR-OBS-3 and PR-OBS-5, a sub-set of lines and columns has been also extracted which comprises

the common validation area. The up-scaling has been simply performed by averaging the radar values

included in each pixel in the SEVIRI projection. For the validation of PR-OBS-5 using raingauge data,

the ground data have been interpolated as explained above and the comparison has been performed

between the product estimate and the nearest interpolated grid point over a domain corresponding to the

Walloon region in Belgium. Finally, the scores of the continuous statistics, the contingency tables and

the probability distribution functions have been prepared on a monthly basis according to the rules

common to all the teams involved in the precipitation products validation.

Fig. 13 - Left: Gaussian filter; right: sketch of the up-scaling procedure. The circle corresponds to the range of the weather radar. The square in the middle is a common area such that it is entirely included in the selected PR-

OBS-2 files. The grey rectangle, the tilted dark grey rectangle and the black ellipse are explained in the text.


Scale Recursive Estimation

As specific development, we have investigated an application of scale recursive estimation (SRE) to

assimilate rainfall rates during a storm estimated from the data of two remote sensing devices. These are

ground based weather radar and space-born microwave cross-track scanner (PR-OBS-2). Our approach

operates directly on the data and does not require a pre-specified multi-scale model structure. We

introduce a simple and computational efficient procedure to model the variability of the rain rate process

in scales. The measurement noise of the radar is estimated by comparing a large number of datasets with

rain gauge data. The noise in the microwave measurements is roughly estimated by using up-scaled

radar data as reference. Special emphasis is placed on the specification of the multi-scale structure of

precipitation under sparse or noisy data. The new methodology is compared with the latest SRE method

for data fusion of multi-sensor precipitation estimates. Applications to the Belgian region show the

relevance of the new methodology (Van de Vyver and Roulin 2009)8.

8 Van de Vyver H. and E. Roulin, 2009. “Scale recursive estimation for merging precipitation data from radar and

microwave cross-track scanners”. J. Geophys. Res., 114, D08104, doi: 10.1029/2008JD010709.


3.4.2 Facilities in Germany (BfG)

Precipitation data

One of the responsibilities of the Federal Institute of Hydrology (BfG) is the shipping related water level

forecast for the river Rhine at low and middle flows. For tasks like hydrological modeling there is

mainly a need for hourly and daily meteorological data which are provided to BfG by Germany‟s

National Meteorological Service (Deutscher Wetterdienst DWD).

It is intended to conduct precipitation validation activities for the territory of Germany.

Germany has a rather dense network of raingauges and it is covered by 16 radars plus one research radar

at Hohenpeissenberg (see Table 04, Table 05 and Fig. 14, from Bartels et al. 20049).

Table 04 - Precipitation data available at BfG

Data Number Resolution Delay Annotation

Synoptical stations About 200 6h / 12h Near-real-time

TTRR stations About 1000 hourly Near-real-time

PI (Picture Composite International)

European radar sites

15 min, 4 km x 4 km

Provided in hourly intervals

International composite image with ground-proximate radar reflectivity distribution

RW (High Resolution Calibrated Quantitative Composite (national)

16 German radar sites

1 hour, 1 km x 1 km

Near-real-time Quantitative radar composite product from RADOLAN software

Table 05 - Location of the 16 meteorological radar of the DWD

Radar site Launch Model WMO No. Radar site Launch Model WMO No.

München 1987 DWSR-88 C 10871 Rostock 1995 METEOR 360 AC 10169

Frankfurt 1988 DWSR-88 C 10637 Ummendorf 1996 METEOR 360 AC 10356

Hamburg 1990 DWSR-88 C 10147 Feldberg 1997 METEOR 360 AC 10908

Berlin-Tempelhof 1991 DWSR-88 C 10384 Eisberg 1997 METEOR 360 AC 10780

Essen 1991 DWSR-88 C 10410 Flechtdorf 1997 METEOR 360 AC 10440

Hannover 1994 METEOR 360 AC 10338 Neuheilen-bach 1998 METEOR 360 AC 10605

Emden 1994 METEOR 360 AC 10204 Türkheim 1998 METEOR 360 AC 10832

Neuhaus 1994 METEOR 360 AC 10557 Dresden 2000 METEOR 360 AC 10488

9 Bartels H. et al. / Deutscher Wetterdienst, Abteilung Hydrometeorologie, 2004: „Projekt RADOLAN -

Routineverfahren zur Online-Aneichung der Radarniederschlagsdaten mit Hilfe von automatischen

Bodenniederschlagsstationen (Ombrometer)“. Summary report for the project period 1997-2004.


RADOLAN

RADOLAN (Routine procedure for an online calibration of radar precipitation data by means of

automatic surface precipitation stations ‘ombrometers’) is a quantitative radar composite product

provided in near-real time (via ftp) by DWD to BfG. Radar data are calibrated with hourly precipitation

data from automatic surface precipitation stations. For a description of the radar network see

http://www.dwd.de/de/Technik/Datengewinnung/Radarverbund/Standorte.htm .

The process chain from the five-minute-interval radar signals to the final hourly precipitation product is

presented in the Fig. 15. RADOLAN data of hourly precipitation (sampling period hh:51 min to

(hh+1):50 min) have a precision of 0.1 mm/h and cover the whole territory of Germany with a spatial

resolution of 1 km.

Fig. 15 - Flowchart of online calibration

RADOLAN (adapted from Bartels et al. 2004).

Fig. 14 - Left panel: radar coverage in Germany as of 01/03/2007. Right panel: location of ombrometers for online calibration in RADOLAN; squares: hourly data provision (about 500), circles: event-based hourly data provision (about 800 stations): red: AMDA III, blue: aggregational network federal states (Bartels et al., 2004).


3.4.3 Facilities in Hungary (OMSZ)

Ground data description (instrument characteristic and map)

In Hungary, about 90 automatic stations work (Fig. 16), where 10-min precipitation is measured by

tipping bucket rain gauges. This data is used to correct the accumulated precipitation radar data

Fig. 16 - The automatic rain gauge network in Hungary.

The main data used for validation in Hungary would be the data of meteorological radars. There are

three C-band dual polarized Doppler weather radars operated routinely by the OMSZ-Hungarian

Meteorological Service (see Fig. 17 and Table 06).

Table 06 - Characteristics of the three meteorological Doppler radars in Hungary

Year of installation Location Radar type Parameters measured

1999 Budapest Dual-polarimetric, Doppler radar Z, ZDR

2003 Napkor Dual-polarimetric, Doppler radar Z,ZDR,KDP,ΦDP

2004 Poganyvar Dual-polarimetric, Doppler radar Z,ZDR,KDP,ΦDP

Ground data quality and accessibility

Access to Hungarian radar data can be set up through contact with the responsible of the institute within

the HSAF project. It will be provided for developers if required for case studies.

Pogányvár Napkor

Budapest

Fig. 17 - Location and coverage of the three meteorological Doppler radars in Hungary.


The quality of the radar measurements is influenced by several factors:

- the accuracy of the Marshall-Palmer equation in deriving rain rates

- the beam blockage which causes the lack of precipitation in areas behind mountains.

The Hungarian radar data bears the consequences of both problems. The Marshall-Palmer calculations

can result in a factor of multiplying or dividing by 2; whereas the beam blockage can result in serious

underestimation of precipitation amounts (e.g. behind the Börzsöny mountains at the north of Budapest).

Besides, the Hungarian radar data is filtered from WLAN signal, which is also a source of false signals

in the radar.

A filter to disregard signals below 7 dBz is also applied because in general, these data is not coming

from real rain drops, but false targets.

Ground data products

Precipitation intensity is derived from radar reflectivity with the help of an empirical formula, the

Marshall-Palmer equation. From the three radar images a composite image over the territory of Hungary

is derived every 15 minutes applying the maximum method in order to make adjustments in overlapping

regions.

The non-corrected precipitation field can be corrected by rain gauge measurements. As recent

researches have shown it is only possible to produce adequate precipitation fields by the correction of

raw radar data at time scales of the order of a few hours or more, thus we do not make corrections to 15

minutes radar data. In our institute, we only use a correction for the total precipitation over a 12 hour

period.

For the 3h and 6h accumulated products, we use a special method as well: we interpolate the 15-minutes

measurements for 1-minute grid by the help of displacement vectors also measured by the radar, and

then sum up the images which we got after the interpolation. It is more precise especially when we have

storm cells on the radar picture, because a storm cell moves a lot during 15 minutes and thus we do not

get continuous precipitation fields when we sum up only with 15.minutes periods. This provides

satisfying results. However, there is still a need for rain-gauge adjustment because there are obviously

places (behind mountains) that the radar does not see.

The radars are corrected with rain gauge data every 12 hours. The correction method using raingauge

data for 12 hour total precipitation consists of two kinds of corrections: the spatial correction which

becomes dominant in the case of precipitation extended over a large area, whereas the other factor, the

distance correction factor prevails in the case of sparse precipitation. These two factors are weighted

according to the actual situation. The weighting factor depends on the actual effective local station

density, and also on the variance of the differences of the bias between radar and rain gauge

measurements. On the whole, we can say that our correction method is efficient within a radius of 100

km from the radar. In this region, it gives a final underestimation of about 10%, while at bigger

distances, the underestimation of precipitation fields slightly increases. Besides, we also produce 12

hour total composite images: first the three radar data are corrected separately, and then the composite is

made from them. The compositing technique consists of weighting the intensity of each radar at a given

point according to the distance of the given point from the radars.

Ground data interpolation

No interpolation is used for the radar data.

Institute validation methodology

The data used for the discrimination of case studies:

OMSZ-Hungarian Meteorological Service receives operationally the MSG data and several products

based on satellite data are derived and sent to the weather forecasters. The SAFNWC/MSG program

package derives cloud type product and precipitation products (convective rain rate and probability

of precipitation) also. MSG composite images are created operationally. For the description of


meteorological environment, the Cloud Type product is the most useful as it distinguishes high-level

and optically thick clouds from medium- and low-level clouds; and optically thin clouds. It is also

useful for analyzing the cloud systems that have occurred at the time of satellite measurements.

SAFIR lightning system works operationally at Hungarian Meteorological Service. We have five

stations. The accuracy of the lightning network has been significantly increased the last two due to

the large amount of case studies investigated during summer periods. Therefore we intend to use the

lightning data, first of all in visual comparison as descriptor of the synoptic situation.

The main steps of the precipitation products validation procedure are:

Time and space alignment of the data from different sources, represented on different scales – pixel

size of radar and satellite data is not the same. Especially in the case of microwave measurements,

we have large footprints that need to be treated carefully. The time alignment can be solved by

simply matching the closest data available in time to the H-SAF products. There are advanced

methods to solve the up-scaling of ground and/or the down-scaling of satellite precipitation data.

The OMSZ-Hungarian Meteorological Service uses the techniques discussed and proposed by the

members of the validation group. We will contribute to the selection of the appropriate method and

to the investigation of the accuracy of the matching method.

The common task of the Precipitation Validation Group is the statistical validation. The up-scaling

method has a key role in this process. We can only compare the radar, satellite, raingauge, and

lightning if we have previously determined the coherent values from different sources. The data will

be collected into one file containing the following information to calculate statistic values either for

15 minutes samples, and for accumulated total precipitation information (over 3, 6, 12, 24 hours):

- radar-derived precipitation intensity,

- the satellite product values derived by H-SAF.

Statistical characteristics (RMS, BIAS, etc) will be then calculated between the satellite-based

precipitation and radar data for the different cases. These statistics will be performed for different

periods (months, summer-winter, day-night) as well as for some satellite pass in extreme cases .

We will also calculate multi-categorical statistics according to the classes determined by common

agreement by the validation group and the hydrologists. We will prepare contingency tables, scatter

plots and bias distribution functions to illustrate the performance of the precipitation products. These

statistics will be performed for different periods (months, summer-winter, day-night) and maybe for

some specific satellite passes. H-SAF data is going to be verified against radar data which preferably

will be adjusted to rain gauge data. Our radar data is already gauge-adjusted in the case of 12, and 24h

total precipitation.

The first task to the preparation of case studies is the selection of the periods with significant rain

amounts. This is done by the plotting of the radar and satellite rainfall estimate sums on a daily basis

of each month.

The second task to the preparation of case studies is the collection of all relevant data to the period

in question. This means the collect of radar, lightning data, and of MSG images and SAFNWC

products which correspond to the time period investigated.

The visual comparison including both rain gauge and radar data is an inevitable tool for the

understanding of the validation results of H-SAF products, because it:

- provides an overall picture of the H-SAF database (resolution, territories seen, general values)

- highlights similarities and differences in the structures and in the values

- helps to monitor the meteorological situation as well

To visualize the different products and to make subjective comparison we use the Hungarian Advanced

Weather Workstation (HAWK) developed by OMSZ-Hungarian Meteorological Service which can

interpret all kind of meteorological products together.


3.4.4 Facilities in Italy (UniFe)

Introduction

This report aims to set up a precipitation validation plan for Italian region. The satellite precipitation

estimation products to be validated come from the development and operation activity. A quick

overview of the products characteristics as request from the approved H-SAF development proposal is

provided. Measurements from raingauge and retrievals from weather radar will be considered as

precipitation “truth” and used to validate those products. A description of the availability of raingauge

and radar precipitation data over Italy is provided. A validation plan at different levels is suggested.

Satellite precipitation estimation products

CNR-ISAC and CNMCA are in charge to develop algorithms to estimate precipitation from microwave

(MW) and infrared (IR) sensors on board respectively of polar and geostationary satellite. The spatial

resolution of MW precipitation product is around 15 km whereas the time resolution is about 6 hours.

The precipitation estimation from merging/morphing MW and IR will be generated at around 5 km of

spatial resolution and 15 minutes of time resolution. Both the products have to be validated. Both

instantaneous and cumulated precipitation (3, 6, 12 and 24 hours) have to be validated, however, in case

of MW precipitation estimation due to its very low time resolution only instantaneous precipitation

estimation can be considered and therefore validated.

Data availability and facilities

The Dipartimento Protezione Civile (DPC) and the Centro Nazionale di Meteorologia e Climatologia

Aeronautica (CNMCA) will provide data set to be used as precipitation “truth”. Measurements from

1200 raingauge distributed along the Italian region are considered the main source of precipitation

“truth”. Those data are available at 30 minutes of cumulated time. There is also the possibility to have

some data set of raingauges with a cumulated time of 5 minutes. These data can be used to validate

instantaneous precipitation estimation. For cumulated precipitation validation a longer cumulated time

have to be considered. The radar network will provide precipitation estimation every 30 minutes and the

sum of data inside longer interval is considered as cumulated radar rain. A digital elevation model of the

Italian region is necessary to select land and sea region and also to evaluate the impact of orography on

the precipitation data.

For the following validation procedure the use of IDL, Fortran 90 and C++ software is foreseen. The

first one is useful to visualise the images and therefore for a visual inspection. The second one is useful

in case of validation of very large data set.

Validation recommendation

Precipitation estimation validation means to compare satellite precipitation estimation against other

source of precipitation data assumed as “truth”, as provided by raingauge and radar, and the comparison

procedures have to be specified. Let us to consider two rainfall maps: the first one from raingauge/radar

data and the second one from satellite estimation, three kind of comparison are here suggested.

VISUAL COMPARISON: the two maps are compared by visual inspection. The human eye has a natural

and powerful ability at judging the similarity of two images. IDL software is used to produce and

compare satellite and raingauge/radar rainfall maps,

CONTINUOS STATISTICS COMPARISON: the rainfall rate from satellite estimation and radar/raingauge

are provided in a continuous range and to quantify the similarity of the two images it is convenient

to calculate some parameters considering pixel by pixel. The suggested parameters are: Mean Error

(ME, or bias), Root Mean Square Error (RMSE) and correlation coefficient (CC).

CATEGORICAL STATISTICS COMPARISON: the rainfall rate from satellite estimation and radar/raingauge

data have to be organized in classes of precipitation. At the first step by fixing a rainfall threshold of

0.1 mm/h only two classes of precipitation (rain and no-rain) can be considered. Then the

precipitation classes can be those indicated in Fig. 08 and Fig. 09. Once the continuous data are

organized in classes the contingency table have to be built. Then some statistical parameters have to


be calculated to quantify the similarity of two images. In case of two classes of precipitation the

suggested parameters are: FAR, POD, BIAS, ETS and HSS. In case of more than two classes the

useful parameters are: HSS and correlation coefficient.

For a reliable validation plan an ensemble of cases (each of them consisting in estimated and “truth‟

maps) have to be considered. The larger the ensemble is the more reliable the validation is. The visual

comparison for several pairs of images become not easy and therefore only the 2 and 3 procedures are

suggested. However it does worth to mention that there are two ways to carry out those procedures: a)

all the pixels from the ensemble of cases are considered together. We obtain an ensemble of pixels of

satellite precipitation estimation and an ensemble of pixels of precipitation “truth”. The two ensemble

are compared by calculating the previously indicated parameters. b) for each case the comparison is

carried out by calculating the above statistics parameters. The average values of those statistics

parameters are calculated over the ensemble of cases.

It is also suggested to carry out the validation for two different areas: sea and land. This because the

precipitation “truth” over the land is mainly based on raingauges whereas for the sea only radar

estimation are available. A further distinction will be made considering orography, by using a digital

elevation model (e.g. GTOPO30 by the U.S. Geological Survey).

Validation: complete plan

The precipitation “truth” data set is built for a complete year at 15 minutes of time resolution.

Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over Italy during the

considered year can be collected and used to generate the corresponding MW-based precipitation

estimation. All of them can be validated. MW+IR algorithm: for each time of the day and for each

day of the year it is possible to validate the MW+IR-based precipitation estimation. In this case the

variability of performance along the day and along the year can be assessed. Moreover the

degradation of MW+IR performance far from the MW overpass can be evaluated.

Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3, 6, 12 and 24 hours

intervals can be validated by using the corresponding cumulated “truth” rain. The variability of

performance along the year can be assessed as well.

Validation: minimal plan

The precipitation “truth” data set is built for summer and winter at around 12 and 24 UTC.

Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over Italy in that

period can be collected and used to generate the corresponding MW-based precipitation estimation.

All of them can be validated.

Instantaneous rain validation (MW+IR algorithm): for 12 and 24 UTC times and for each day of

summer and winter it is possible to validate the MW+IR-based precipitation estimation. In this case

it is expected to get information about the maximum variability of performance.

Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3,6,12 and 24 hours

intervals for summer and winter can be validated by using the corresponding cumulated “truth” rain.

Validation: specific plan

The precipitation “truth” data set is built for particular region of Italy and/or for particular month and/or

particular time of the day.

Instantaneous rain validation (MW algorithm): all the polar satellite overpasses over that region and

in that period can be collected and used to generate the corresponding MW-based precipitation

estimation. All of them can be validated. MW+IR algorithm: for that region and that period it is

possible to validate the MW+IR-based precipitation estimation.

Cumulated rain validation (MW+IR algorithm): cumulated estimated rain at 3, 6, 12 and 24 hours

intervals for that region and that period can be validated by using the corresponding cumulated

“truth” rain.


3.4.5 Facilities in Poland (IMWM)

Precipitation data measurements and observations in Poland

The validation of the H-SAF precipitation products will be carried out in the IMWM on the base of the

ground measurements networks that include:

Telemetry network of automatic rain gauges.

Network of standard rain gauges.

Pluviograph network.

Meteorological radars.

The telemetry network encompasses 430 posts with automatic rain-gauges (2 instruments at each post).

The posts are located all over the whole territory of Poland, however the network is more dense in the

Southern Poland where the flood danger is very high. The measurements are available in the near-real

time and its frequency may be configured at the telemetric network. In standard operational mode the

rainfall and water level data are provided every 10 minutes from the post in the Southern Poland and

every 1 hour from the other ones. The quality of the data is checked on the level of processing in

operational mode.

The network of standard rain gauges consists of 61 synoptic stations and 210 climatological stations, all

equipped with rain-gauges operated by observers as well as 1027 rain-gauge posts. The network is

operational and the data are available in real time from the stations situated in the Southern Poland and

off-line from other parts of country. The synoptic stations provide the standard, 6 hour cumulative

values. The data from other posts are available once per day at normal mode and every 3 hours during a

flood. The quality of the data is checked on the level of processing in the annual mode.

The pluviograph network includes 102 stations located all over the whole country. The network status is

historical. The data are collected only during the period between the 1st of May and the 31st of October

with 10 minute time resolution. The quality of the data is checked on the level of processing in the

annual mode.

The meteorological radar network consists of 8 Doppler radars located all over the country. The data are

available in the real time with 10 minute resolution for analysis and 15 minute resolution for the

forecast. Generally, radar measurements are processed calibrated through/in NIMROD system.

Quality and Accessibility

The data from telemetric posts, rain gauges and synoptic stations located in the Southern Poland are

operationally collected in the database of the System of Hydrology SH located in the Krakow Branch of

IMWM. The data from all standard measuring stations and post are collected in the central data base of

the IMWM in Warsaw. The quality checked data are available from this data base with at least half of a

year delay. The radar measurements are collected in the Radar Centre of IMWM in Warsaw.

The databases available in the IMWM have been check in order to estimate their usefulness for H-SAF

purposes. For data collection, the database existing in the frame of the System of Hydrology will be

used. However, the present database capacity does not allow introducing new users. Therefore it has

been decided that, before the further database development, the data required for H-SAF will be

archived separately in order to simplify their use in the future.

Validation

The validation of the precipitation products will be carried out in the Satellite Research Department of

IMWM in Krakow. Due to its characteristics and availability of quality checked measurements, the 10

minute data from the telemetric network will be used for the validation. It is assumed The standard

observations will play supporting role especially in long term analysis.

The validation process/chain will be performed for satellite projection and include two parts. Firstly, the

analysis, common for all valiadation partners, will be carried out. In the frame of this part of validation

the continuous statistical analysis for all data set and separately for rainfall classes (convective,


stratiform) and different seasons will be performed. The following characteristics are to be obtained:

correlation coefficient, STD, ME, root mean square error. For merging between satellite and ground

measurements, the technique proposed by the validation group will be applied.

Next, the categorical statistical analysis will be performed both for the classes proposed by the

validation group and the classes agreed with Polish hydrologist.

In the second part of the validation it is assumed that the satellite derived cumulative precipitation field

will be compared with the one obtained on the base of the ground measurements. In the analysis the GIS

techniques will be used. The works on the spatialisation methods for precipitation are being developed

in the Krakow Branch of IMWM;

The use of the radar data in validation heavily depends on the scheme of the Radar Operational Centre

team. They may provide the unique methodology of satellite-radar comparison but there are still issues o

be decided.


3.4.6 Facilities in Slovakia (SHMÚ)

Instruments

Slovak Hydrometeorological Institute (SHMÚ) has a large network of the ground stations able to

measure rainfall (see Fig. 18). 684 of these raingauges will be used for validation purpose during this

project covering the complete territory of Slovak Republic. 98 gauges are used operationally and 586

are for research purpose (climatology). It is expected that the number of operational gauges will increase

during the project. Data are available in 1 min, 10 min, 1, 6, 12, 24 hours interval depending on the

gauge type. SHMÚ is performing the offline automatic and manual quality check.

Fig. 18 - Map of SHMÚ raingauge stations: green – operational (98) , blue – climatological (586), red - hydrological stations

in H-SAF selected test basins (37). White points show regular grid of experimental NOAA Snow water equivalent data.

Radar data are planed to be used in this project too. The Slovak meteorological radar network consists

of 2 meteorological radars (see Fig. 19). One is situated at Maly Javornik near city Bratislava and

second is on top of Kojsovska hola close to the city Kosice. Both are Doppler, C-band radars with the

beam width 1 degree and the radial resolution from 256 m (operational). The newer radar at Kojsovska

hola is able to measure the dual polarization variables (non operational). The radar measured

precipitation intensity is available for the whole radar network every 15 minutes, cumulative

precipitation is done for a period of 1, 3, 6 and 24 hours. There is no quality check applied to radar data

operationally. The radar rainfall intensity and the radar cumulative precipitation are not yet corrected or

adjusted to the gauges observations, it will be a part of this project.

Fig. 19 - Example of Slovak radar network coverage - left circle corresponds to radar site Malý Javorník, right one corresponds to Kojšovská hoľa.


Methods

Satellite data are usually the raster data with defined projection and pixel size/resolution. Radar data are

also raster data, but usually in a different projection and resolution from satellite data. Therefore the first

step is to transform both data to the common projection and pixel resolution map optimal for data

comparison. SHMÚ has good experiences how to perform necessary re-projections operationally.

Raingauge data are point data, which can be mapped by means of 2D or 3D interpolation which use the

method of regularised splain with tension. When 3D interpolation is used, orography is reflected in the

outputs. This method is generally called kriging. In this way, the original point data are transformed to

the raster file comparable with satellite and radar data (see Fig. 20).

Fig. 20 - Example of 5-days cumulative precipitation constructed from raingauge measurements by means of 3D

extrapolation method.

Radar data should be validated and adjusted before being used for the validation of other products.

SHMÚ has made calibration study on radar site Maly Javornik based on regression between set of

raingauges and radar 24-hour precipitation products in 150 km radar range few years ago. The

corrections on distance from radar and partial beam blockage by terrain were subject of this study. We

plan to make similar study for the whole territory of Slovakia. Output should be used for the operational

calibration and quality control of radar precipitation measurements. We are also planning to test method

for precipitation estimation based on polarisation techniques for new radar in eastern part of Slovakia.

Quality control of radar data will be based on NWC SAF products like cloud types classification and

products containing information about water content in vertical column.

SHMÚ is planning to use three methods to perform final validation of satellite precipitation products:

Direct comparison of satellite data with point raingauge measurements, calculation of standard

statistical parameters RMSE, ME and other.

Comparison of satellite raster data with precipitation raster based data (interpolated raingauges,

corrected radar precipitation fields,…) and with calculation of standard statistical parameters.


Dividing of precipitation amounts to the defined intervals (classes) according to precipitation

intensity and to evaluate the following parameters: probability of detection (POD), equitable threat

score (ETS), false alarm ratio (FAR) and critical success index (CSI).

Hydrological validation

Slovak Hydrometeorological Institute (SHMÚ) has a large hydrological network of the ground stations

able to measure water level (see Fig. 21). There are 212 stations measuring in online mode. Some of

them (on the Myjava, Kysuca, Hron and Topl‟a river) will be used as input data for H-SAF application.

The data from these stations are available in 15 min interval. SHMÚ is performing the offline automatic

and manual quality check.

Fig. 21 - Map of SHMÚ hydrological stations: red - hydrological stations in H-SAF selected test basins (34).

Hydrological model HRON are conceptual (HBV type) model, semi-distributed with altitude zones

(Fig. 22).

Fig. 22 - General hydrological model of HBV type.

(pictures are from http://www.smhi.se/en/index.htm)

http://www.smhi.se/en/index.htm


Fig. 23 - Example of pick up pixels from radar measurement for selected area.

Model input data are discharge and average data of precipitation

and temperature. We calculate average value from ground stations

and from radar measures. Meteorological forecasted data are taken

from weather prediction model ALADIN.

Average catchment precipitation value from ground station is

calculated using Thiessen polygons method and arithmetic mean

with different station weights assigned a priori. Average

temperature value is calculated as weighted arithmetic mean.

Catchment average precipitation value of 1-hour precipitation

forecast from radar product is calculated as average value of pixels

from the selected area (mask of subcatchment) (see Fig. 24).

Model ALADIN supplies precipitation and temperature forecasts to

hydrological model HRON. Since the resolution of forecast is too

low for selected areas, the large pixel is separated into smaller

pixels for better average value results (see Fig. 24). SHMÚ is

planning to use satellite precipitation products in forecasting hydrological models in similar way.

Mask of selected area. Radar measurement sector with selected area border.

Radar measurement in the selected area only.

Fig. 24 - Example of separated ALADIN

forecast pixel to smaller pixels.


3.4.7 Facilities in Turkey (ITU)

Locations and instruments

For the validation of precipitation products, a first set of locations were selected as Susurluk and

Western Black Sea catchments which represent different physiographic and climatic conditions of

Turkey (see Fig. 25). The Susurluk catchment is influenced by Mediterranean type of climate with mild-

wet winters and hot-dry summers. The basin is covered by forest lands together with fertile alluvial

plains. On the other hand, the Western Black-Sea catchment is characterized by high rainfall dominating

almost two thirds of the year and humid summers. The catchment is covered by highly dense forests and

agricultural lands. The catchment has mountains with low elevations rarely exceeding 1,500 m.

Fig. 25 - The Susurluk and Western Black Sea catchments selected for the precipitation product validation in Turkey.


Validation of precipitation products is based mainly on the comparison with the AWOS and weather

radar data measurements. Turkey has an extensive coverage of meteorological observations, including

automated weather observing stations (AWOS), synoptic and climatological stations, and 4 radars. At

present nearly 450 meteorological, 60 airport, 180 climate, and 7 upper air (radiosonde) stations are in

operation. The meteorological observation network has been strengthened by installation of AWOS in

recent years, and in the near future almost entire country will be covered by an AWOS network to

replace the conventional network.

As shown in Fig. 26, the Western Black Sea catchment has 23 AWOS and 7 synoptic stations in

addition to radar coverage. Similarly, the Susurluk basin has 17 AWOS and 3 synoptic stations and

radar coverage.

Fig. 26 - Network of Meteorological stations in Susurluk (on the left) and Western Black Sea (on the right) catchments.

Starting from January 2009, the validation area has been extended to the middle and western Turkey.

193 Automated Weather Observation Station (AWOS) located in the western part of Turkey are used for

the validation of the precipitation products. The locations of the AWOS sites are shown in Fig. 27.

Fig. 27 - Position of 193 AWOS sites used for ground truth for the precipitation product validation in western Turkey.

Validation Methodology

Due to the time and space structure of precipitation and sampling characteristics of both the

precipitation products and observations used for validation, care has to be taken to bring data to a

comparable level. At a given place, precipitation occurs intermittently and at highly fluctuating rates.

Over space, precipitation is distributed with a high variability, in cells of high intensity nested in larger

area with lower rain rate. Aimed at observing this complex phenomenon, the satellite-based products are

defined with a spatial resolution of several kilometres and with different sampling rate and accumulation


time. On the other hand, reference observations against which the products are going to be validated are

also characterized by their own spatial resolution ranging from point information measured on rain-

gauge networks to grids with cells of several hundreds of meters to several kilometres for weather radar.

Furthermore, none of these reference observations are without error.

Each precipitation product within the H-SAF project represents a foot print geometry. Among these,

H01 and H02 products represent an elliptical geometry while H03 and H05 have a rectangular

geometry. On the other hand, the ground observation (rain-gauge) network consists of point

observations. The main problem in the precipitation product cal/val activities occurs in the dimension

disagreement between the product space (area) and the ground observation space (point). To be able to

compare both cases, either area to point (product to site) or point to area (site to product) procedure has

to be defined. However, the first alternative seems easier. The basic assumption in such an approach is

that the product value is homogenous within the product footprint. Fig. 28 presents satellite foot print

(IFOV) centres of the H02 product, an elliptical footprint for the corresponding centre (area within the

yellow dots) and AWOS ground observation sites. The comparison statistic can be performed by

considering just the sites in the footprint area. Although this approach is reasonable on the average but it

is less useful in spatial precipitation variability representation. The comparison is not possible when no

site is available within the footprint area.

Fig. 28 - H02 product footprint centres with a sample footprint area as well as the AWOS ground observation sites.

Alternatively, the point to area approach is more appealing for the realistic comparison of the

precipitation product and the ground observation. This approach is simply based on the determination of

the true precipitation field underneath the product footprint area. To do so, the footprint area is meshed

and precipitation amounts are estimated at each grid point by using the precipitation observations at the

neighbouring AWOS sites as shown in Fig. 29. A 3x3 km grid spacing is considered for the products

with elliptical geometry while 2x2 km spacing is considered for the products with rectangular geometry.

At each grid point, the precipitation amount is estimated by,

n

i

mi

n

i

imi

m

rW

ZrW

Z

1

,

1

,

)(

)(

(1)

where Zm is the estimated value and W(ri,m) is the spatially varying weighting function between the i-th

site and the grid point m.


Fig. 29 - Meshed structure of the sample H02 product footprint.

Determination of the W(ri,m) weighting function in Equation 1 is crucial. In open literature, various

approaches are proposed for determining this function. For instance, Thiebaux and Pedder 198710

suggested weightings in general as,

Rrfor

RrforrR

rR

rW

mi

mi

mi

mi

mi

,

,2

,

2

2

,

2

,

0

)( (2)

where R is the radius of influence, r is the distance from centre to the point and is a power parameter

that reflects the curvature of the weighting function. Another form of geometrical weighting function

was proposed by Barnes 196411 as,

R

rrW

mi

mi

,

, 4exp)( (3)

Unfortunately none of these functions are observation dependent but suggested on the basis of the

logical and geometrical conceptualizations only. They are based only on the configuration, i.e. geometry

of the measurement stations and do not take into consideration the natural variability of the

meteorological phenomenon concerned. In addition, the weighting functions are always the same from

site to site and time to time. However, in reality, it is expected that the weights should reflect to a certain

extent the regional and temporal dependence behaviour of the phenomenon concerned.

For the validation activities, the point cumulative semi-variogram technique proposed by Şen and Habib

199812 is used to determine the spatially varying weighting functions. In this approach, the weightings

not only vary from site to site, but also from time to time since the observed data is used. In this way,

the spatial and temporal variability of the parameter is introduced more realistically to the validation

activity.

10

Thiebaux H.J. and M.A. Pedder, 1987: “Spatial objective analysis”. Academic Press, 299 pp. 11

Barnes S.L., 1964: “A technique for maximizing details in numerical weather map analysis”. J. App. Meteor., 3,

pp.396-409. 12

Şen Z. and Z. Habib, 1998: “Point cumulative semivariogram of areal precipitation in mountainous regions”. Journal

of Hydrology 205 (1–2), 81–91.


4. Validation of the product release as at the end of the Development Phase

4.1 Introduction

This Chapter collects the results of the validation experiments as at the end of the H-SAF Development

Phase. The validation is performed on the product release in force at the time of the Operations

Readiness Review (ORR). The organisation was as follows:

The validation period is 1st January 2009 to 31 March 2010, in order to cover all seasons with

margins. Granularity: one month.

The results of the previous validation cycles, including case studies, are recorded in the Appendix to

this PVR-06, reproduced from the so-called “REP-3 (H-SAF Products Validation Report)”, last

issue dated 28 February 2010. The Appendix is a simple transcription of the experiments, thus it is

characterised by a low level of editing, as it was appropriate to a project-internal working document.

REP-3 was split in volumes, each H-SAF product being addressed by one volume. PR-ASS-1 was

addressed by REP-3/07 (Results of validation activities for product PR-ASS-1). However, only

internal tests made in the framework of the COSMO Consortium were reported there, whereas this

PVR-06 is based on results achieved by the Units of the H-SAF validation team for precipitation.

This Chapter 4 is structured by Country / Team, one section each. Each section records the main

statistical scores across the January 2009 - March 2010 period, and provides comments, possibly

supported by further analysis material. The next Chapter 5 (Overview of findings) provides comparative

features among the results from the various Countries / Teams, so that the User of the product is

informed of the variability of the performances with climatological and morphological conditions, as

well as with seasonal effects.

Unlike the previous validation campaigns, that placed priority to supporting the product development,

thus made large use of case studies, this campaign focused on the objective of the ORR of assessing the

degree of compliance of the product quality with the User requirements. For product PR-ASS-1 the

User requirements are recorded in Table 07.13 As a matter of fact, the original requirements do not

specify the range of interest for precipitation rate, nor the integration time for the accumulated

precipitation. We have complemented the requirements having regard to other precipitation rate and

accumulated precipitation products.

Table 07 - Accuracy requirements for product PR-ASS-1 [RMSE (%)]

Product threshold target optimal Additional specifications

Precipitation rate 100 50 25 To be validated for RR < 1.0, 1-10 and > 10 mm/h.

To be verified for RR in the range 1-10 mm/h

Accumulated precipitation 200 100 50 To be validated for integration over 3, 6, 12 and 24 h.

To be verified for integration over 24 h

This formulation of the requirement implies that the main score to be evaluated is the Root Mean Square

Error that, since grows with intensity or amount, is better evaluated as percent, i.e. RMSE (%).

Supportive scores are: the ordinary RMSE (mm), the Mean Error (or bias, ME), and the Standard

Deviation (SD). In addition, the Correlation Coefficient (CC), the Probability Of Detection (POD), the

False Alarm Rate (FAR) and the Critical Success Index (CSI, necessary to compare POD - FAR

coupled scores), also are reported, although rather unstable quantities for this type of geophysical

parameter that does not at all comply with Gaussian characteristics.

Each Country / Team should conclude its Section by listing the main features of the product, function of

whatever the Team considers as a significant change of conditions associated to change of performance.

The purpose is to characterise the applicability of the product for a correct use, especially in hydrology.

13

There is evidence that the user requirements for precipitation observation from space adopted by authoritative bodies

(WMO, EUMETSAT, the GPM planning board) are overstated. However, currently another reference is not available.

The situation will be re-assessed during CDOP-1.


4.2 Validation in Belgium (IRM)

The facilities available to IRM, and the methodology adopted for validation, are described in section

3.4.1. The ground truth is provided by meteorological radar.

In respect of product PR-ASS-1, IRM has validated only the 24-hour accumulated precipitation. The

results are reported in Table 08. It is noted that in February and March 2010 the validation could not be

performed because of unavailability of the radar.

Table 08 - Summary results of PR-ASS-1 validation in Belgium by IMR

H06 v1.0 Belgium Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm

Radar Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

No. samples 9952 15604 16595 12504 15288 13576 22295 9602 6468 13383 10204 21066 11321 16256 12595

ME (mm) -0.37 -0.61 -3.28 -2.56 -1.22 -1.76 -2.84 -5.19 -1.95 -1.68 0.27 -0.50 -0.17 -0.04 -0.38

SD (mm) 6.03 5.25 11.7 9.87 5.19 7.32 6.71 10.9 3.89 6.07 5.88 4.62 3.75 6.04 5.46

RMSE (mm) 6.61 4.78 12.8 11.3 6.78 11.4 8.29 12.8 5.74 7.02 4.36 5.03 3.80 5.40 6.19

24-h cumulate RMSE (%) 142 134 105 202 155 332 137 182 134 170 137 127 116 133 192

CC 0.46 0.61 0.03 0.16 0.28 0.15 0.21 0.04 0.16 0.24 0.79 0.34 0.59 0.54 0.31

POD 0.75 0.73 0.84 0.63 0.61 0.58 0.53 0.35 0.68 0.68 0.78 0.83 0.75 0.90 0.83

FAR 0.37 0.41 0.21 0.31 0.35 0.28 0.28 0.49 0.18 0.18 0.31 0.26 0.54 0.35 0.27

CSI 0.52 0.48 0.69 0.49 0.46 0.47 0.44 0.26 0.59 0.60 0.58 0.64 0.40 0.61 0.64

The monthly values of the Mean Error (ME) and of the Root Mean Square Error (RMSE) are shown in

Fig. 30 and Fig. 31, respectively. The ME shows an underestimation for all months except one. The

results are better during winter. The RMSE is also lower during winter months. The ME and RMSE for

winter are of the same order as found by Roulin and Vannitsem (2005) when analyzing the ECMWF

EPS forecasts for a catchment situated in the validation area of the present study. For the summer

season, the ME and RMSE were lower (|ME| < 1 mm day-1 and RMSE < 7 mm day-1). However, the

validation methodology was different: precipitation was first areally averaged and scores were

computed over 6-months season).

Mean Error

-6,00

-5,00

-4,00

-3,00

-2,00

-1,00

0,00

1,00

200901 200903 200905 200907 200909 200911 201001 201003

Fig. 30 - Mean Error of PR-ASS-1, 24h accumulated precipitation (monthly values over Belgium in mm day-1).


RMSE

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

200901 200903 200905 200907 200909 200911 201001 201003

Fig. 31 - Root Mean Square Error of PR-ASS-1, 24 accumulated precipitation (monthly values over Belgium in mm day-1).


4.3 Validation in Germany (BfG)

The facilities available to BfG, and the methodology adopted for validation, are described in section


BfG has performed full validation of PR-ASS-1, i.e. the precipitation rate and the accumulated

precipitation over 3, 6, 12 and 24 hours. The results are reported in Table 09 for the rate and Table 10

for the accumulated.

Table 09 - Summary results of the validation of PR-ASS-1rate in Germany by BfG

H06 v1.0 Germany Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h

Radar Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

Number of > 10 mm/h 0 0 43 18 1532 1458 3550 1068 879 708 122 70 0 32 167

comparisons 1-10 mm/h 40846 56237 88986 30613 63647 91683 113267 41003 40853 66682 118869 110608 33988 58379 58068

(ground obs.) < 1 mm/h 143498 305101 275032 64891 88870 100240 95569 36959 48806 147546 239426 294369 258754 235152 176178

> 10 mm/h - - -10.5 -11.6 -17.1 -13.2 -15.7 -14.3 -14.5 -12.3 -11.5 -12.1 - -13.7 -11.4

ME (mm/h) 1-10 mm/h -1.10 -0.92 -0.96 -1.07 -1.67 -1.71 -1.84 -2.20 -1.82 -1.37 -1.12 -1.02 -0.90 -0.99 -1.16

< 1 mm/h -0.30 -0.30 -0.26 -0.18 -0.11 -0.12 -0.17 -0.19 -0.17 -0.15 -0.20 -0.25 -0.34 -0.33 -0.27

> 10 mm/h - - 0.69 2.70 15.7 5.66 8.99 7.76 5.12 3.33 2.31 1.30 - 2.13 5.30

SD (mm/h) 1-10 mm/h 1.04 0.86 1.99 1.50 2.03 1.98 2.53 2.11 1.86 1.80 1.33 1.10 0.77 0.95 1.34

< 1 mm/h 0.49 0.44 0.77 0.86 1.01 1.07 1.30 1.23 0.90 0.84 0.69 0.58 0.38 0.52 0.61

> 10 mm/h - - 10.6 11.9 23.2 14.4 18.1 16.2 15.4 12.8 11.7 12.1 - 13.85 12.60

RMSE (mm/h) 1-10 mm/h 1.51 1.26 2.21 1.84 2.62 2.62 3.13 3.05 2.60 2.26 1.74 1.50 1.18 1.37 1.77

< 1 mm/h 0.58 0.53 0.82 0.88 1.01 1.07 1.31 1.25 0.91 0.85 0.71 0.63 0.51 0.62 0.67

> 10 mm/h - - 98 95 92 93 93 97 96 94 96 100 - 100 86

RMSE (%) 1-10 mm/h 87 82 167 101 101 101 130 108 96 95 84 83 83 90 89

< 1 mm/h 118 109 134 181 218 228 282 262 197 186 148 128 101 110 139

> 10 mm/h - - -0.04 -0.30 -0.04 0.16 0.07 -0.06 -0.02 0.00 -0.19 -0.03 - - -0.03

CC 1-10 mm/h 0.14 0.12 0.05 0.09 0.14 0.07 0.08 0.02 0.05 0.11 0.14 0.11 0.19 0.13 0.25

< 1 mm/h 0.14 0.18 0.11 0.09 0.06 0.04 0.02 0.02 0.04 0.09 0.13 0.16 0.16 0.13 0.13

POD ≥ 0.25 mm/h 0.19 0.20 0.21 0.21 0.21 0.21 0.18 0.12 0.17 0.25 0.29 0.25 0.17 0.18 0.21

FAR ≥ 0.25 mm/h 0.67 0.68 0.67 0.79 0.83 0.77 0.81 0.89 0.83 0.75 0.60 0.63 0.70 0.64 0.74

CSI ≥ 0.25 mm/h 0.14 0.14 0.15 0.12 0.11 0.13 0.10 0.06 0.09 0.14 0.20 0.18 0.12 0.13 0.13

The time evolution of the statistical scores are depicted in Fig. 32, Fig. 33 and Fig. 34.

Fig. 32 - Time evolution of Mean Error and Standard Deviation for the three precipitation categories.


Fig. 33 - Time evolution of the Root Mean Square Error (mm/h and %) for the three precipitation categories.

Fig. 34 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success Index.

Comments on PR-ASS-1 rain rate

Strong seasonality is observable in quantitative measures of the high precipitation class (> 10 mm/h)

during 2009-2010.

High standard deviation and peak mean error (17.1 mm/h) in the high precipitation class in May

2009 were caused by at least two frontal systems with extended precipitation area that crossed

Germany and were not sufficiently described by the model data.

Mean error is negative all over the period with an average of -1.4 mm/h (-0.2 mm/h) in the mid

(low) precipitation class for the year 2009.

Normalized root mean squared error peaked in July 2009 with the low precipitation class to 282 %.

The overall average for all three classes for 2009 was 129 %.

Performance of PR-ASS-1 in terms of categorical score CSI was highest in November 2009 with

0.20 and lowest in August 2009 with 0.06 with a yearly average of 0.13.

POD and CSI showed little variability and indistinct seasonality, while FAR had a clearer summer

maximum (0.89) and fall minimum (0.60).


Table 10 - Summary results of the validation of PR-ASS-1accumulated in Germany by BfG

H06 v1.0 Germany Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm


No. samples 160542 283005 316996 85757 175815 223199 283683 105681 109352 215188 338513 356182 233475 238020 205093

ME (mm) -1.35 -1.19 -1.16 -1.04 -1.55 -1.84 -1.96 -2.34 -2.01 -1.25 -1.31 -1.26 -1.28 -1.39 -1.27

SD (mm) 1.69 1.48 2.43 2.78 3.76 4.28 4.87 4.41 3.76 3.14 2.41 2.05 1.19 1.61 2.13

RMSE (mm) 2.16 1.90 2.69 2.96 4.07 4.66 5.25 4.99 4.26 3.38 2.75 2.41 1.75 2.13 2.48

03-h cumulate RMSE (%) 90 88 130 128 141 137 157 151 119 126 97 95 87 89 99

CC 0.31 0.30 0.18 0.24 0.27 0.16 0.18 0.04 0.08 0.22 0.31 0.27 0.31 0.26 0.38

POD 0.30 0.30 0.37 0.38 0.37 0.36 0.30 0.25 0.34 0.40 0.41 0.38 0.22 0.28 0.32

FAR 0.40 0.37 0.40 0.61 0.68 0.62 0.67 0.77 0.66 0.54 0.40 0.37 0.44 0.39 0.54

CSI 0.25 0.26 0.30 0.24 0.21 0.23 0.19 0.14 0.20 0.27 0.32 0.31 0.19 0.24 0.23

No. samples 272860 480812 512731 130097 292328 367745 477316 178420 185833 367259 528307 576171 434224 413197 367969

ME (mm) -1.67 -1.43 -1.30 -0.92 -1.32 -1.80 -1.83 -2.19 -1.94 -1.11 -1.40 -1.40 -1.61 -1.66 -1.32

SD (mm) 2.38 2.13 3.73 4.03 4.88 5.90 6.42 6.07 5.10 4.15 3.42 2.85 1.74 2.26 2.81

RMSE (mm) 2.91 2.57 3.95 4.14 5.06 6.17 6.67 6.46 5.45 4.29 3.69 3.18 2.37 2.81 3.11

06-h cumulate RMSE (%) 91 90 167 151 167 167 184 187 148 137 104 99 88 89 106

CC 0.37 0.38 0.21 0.33 0.34 0.23 0.25 0.08 0.10 0.31 0.40 0.38 0.36 0.33 0.42

POD 0.39 0.38 0.48 0.48 0.55 0.46 0.43 0.37 0.46 0.51 0.49 0.47 0.30 0.35 0.40

FAR 0.28 0.28 0.32 0.55 0.63 0.52 0.55 0.68 0.54 0.43 0.33 0.27 0.31 0.31 0.43

CSI 0.34 0.33 0.40 0.30 0.28 0.31 0.28 0.21 0.30 0.37 0.39 0.40 0.26 0.30 0.31

No. samples 459404 830407 861388 218171 507704 594748 817796 306216 306828 626782 891787 975725 751545 731859 662517

ME (mm) -2.18 -1.95 -1.72 -1.07 -1.08 -1.82 -1.62 -1.92 -1.70 -1.04 -1.74 -1.78 -2.10 -2.08 -1.46

SD (mm) 3.23 3.02 5.72 5.45 6.07 7.82 7.96 8.03 6.28 5.11 4.43 3.80 2.41 2.85 3.62

RMSE (mm) 3.90 3.59 5.97 5.56 6.17 8.03 8.13 8.25 6.51 5.21 4.76 4.19 3.20 3.52 3.90

12-h cumulate RMSE (%) 91 91 217 159 194 188 207 216 170 134 103 99 87 90 115

CC 0.46 0.45 0.22 0.43 0.37 0.32 0.29 0.15 0.17 0.39 0.49 0.46 0.42 0.45 0.44

POD 0.46 0.45 0.56 0.58 0.64 0.56 0.54 0.48 0.58 0.59 0.54 0.53 0.37 0.39 0.49

FAR 0.19 0.18 0.23 0.47 0.53 0.44 0.43 0.59 0.44 0.33 0.23 0.16 0.21 0.23 0.29

CSI 0.42 0.41 0.48 0.38 0.38 0.39 0.39 0.29 0.40 0.46 0.47 0.48 0.34 0.35 0.41

No. samples 734563 1278683 1377631 352453 908862 977058 1362757 539123 501401 1052271 1438542 1515607 1195796 1219091 1099085

ME (mm) -2.88 -2.78 -2.21 -1.58 -0.87 -1.80 -1.30 -1.88 -1.35 -1.11 -2.16 -2.50 -2.91 -2.67 -1.85

SD (mm) 4.44 4.34 8.57 7.15 7.48 10.10 9.77 10.52 7.71 6.22 5.59 5.23 3.59 3.81 4.73

RMSE (mm) 5.29 5.16 8.85 7.32 7.53 10.26 9.85 10.69 7.83 6.32 5.99 5.80 4.62 4.65 5.07

24-h cumulate RMSE (%) 99 92 235 181 218 202 232 252 201 140 102 98 89 90 114

CC 0.49 0.50 0.20 0.52 0.36 0.38 0.32 0.15 0.18 0.44 0.51 0.50 0.42 0.46 0.42

POD 0.53 0.53 0.64 0.65 0.66 0.64 0.64 0.58 0.64 0.65 0.60 0.57 0.47 0.44 0.59

FAR 0.14 0.12 0.13 0.42 0.41 0.34 0.29 0.50 0.38 0.25 0.15 0.10 0.13 0.14 0.17

CSI 0.48 0.50 0.59 0.44 0.45 0.48 0.51 0.37 0.46 0.53 0.55 0.54 0.44 0.41 0.52

Fig. 35 to Fig. 42 display the time evolution of the various statistical scores for the different

accumulation periods.

Fig. 35 - Time evolution of the Mean Error, function of the accumulation period.


Fig. 36 - Time evolution of the Standard Deviation, function of the accumulation period.

Fig. 37 - Time evolution of the Root Mean Square Error (mm), function of the accumulation period.

Fig. 38 - Time evolution of the Root Mean Square Error (%), function of the accumulation period.


Fig. 39 - Time evolution of the Correlation Coefficient, function of the accumulation period.

Fig. 40 - Time evolution of the Probability of Detection, function of the accumulation period.

Fig. 41 - Time evolution of the False Alarm Rate, function of the accumulation period.


Fig. 42 - Time evolution of the Critical Success Index, function of the accumulation period.

Comments on PR-ASS-1 accumulated precipitation

Mean error in all accumulated precipitation over 3, 6, 12 and 24 hours was generally negative with a

diffuse bimodal pattern with two maxima in April/May and October and a pronounced minimum in

January 2010 in 24 h precipitation (ME -2.91 mm).

Standard deviation, root mean squared error and normalized root mean squared error all showed

highest values in summer and minima during January/February with absolute values increasing with

the length of accumulation.

Performance of POD, FAR and CSI all showed the strong dependence on accumulation length.

Usually categorical scores of 24 h accumulation were 20-35 % better than those for 3 h

accumulation. This was due to the fact that accumulation masks to a certain degree temporal and

also spatial mismatch of observed and forecast precipitation.

The critical success index varied between 0.14 in August, 3 h accumulation and 0.59 March 2009,

24 h accumulation.

Average CSI for 3 h (6 h, 12 h, 24 h) accumulated precipitation was 0.24 (0.33, 0.41, 0.49) during

the year 2009.


4.4 Validation in Hungary (OMSZ)

The facilities available to OMSZ, and the methodology adopted for validation, are described in section


OMSZ has performed full validation of PR-ASS-1, i.e. the precipitation rate and the accumulated



Table 11 - Summary results of the validation of PR-ASS-1rate in Hungary by OMSZ

H06 v1.0 Hungary Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h


Number of > 10 mm/h 11 3 10 73 458 1184 925 1548 486 206 86 60 17 13 31

comparisons 1-10 mm/h 6036 4550 11019 9200 20968 45070 24181 31095 13431 23450 19020 15129 9036 8964 5046

(ground obs.) < 1 mm/h 42849 39956 55566 29223 48182 73481 35385 35686 23973 57947 53587 55552 42756 45521 27527

> 10 mm/h -9.85 -10.4 -11.3 -15.8 -16.9 -17.5 -18.6 -18.0 -17.0 -13.3 -45.9 -14.5 -12.0 -9.18 -13.2

ME (mm/h) 1-10 mm/h -1.00 -1.21 -1.09 -1.82 -1.87 -2.04 -2.14 -2.08 -1.98 -1.04 -0.93 -0.84 -1.18 -0.87 -1.39

< 1 mm/h 0.08 -0.07 -0.03 -0.32 -0.21 -0.12 -0.15 -0.03 -0.24 0.30 0.18 0.17 0.06 0.16 0.00

> 10 mm/h 1.24 1.27 1.66 6.30 8.72 11.6 14.8 11.5 10.9 6.37 104 9.23 2.08 1.19 8.44

SD (mm/h) 1-10 mm/h 1.09 0.96 1.37 1.55 1.79 2.07 2.65 3.05 2.66 2.25 1.50 1.70 1.42 1.25 1.56

< 1 mm/h 0.62 0.49 0.72 0.61 0.92 1.24 1.53 2.08 1.30 1.33 0.90 0.94 0.64 0.67 0.69

> 10 mm/h 9.93 10.5 11.5 17.0 19.0 21.0 23.8 21.4 20.2 14.7 114 17.2 12.2 9.25 15.7

RMSE (mm/h) 1-10 mm/h 1.48 1.55 1.75 2.39 2.59 2.91 3.40 3.69 3.31 2.48 1.77 1.90 1.85 1.52 2.09

< 1 mm/h 0.63 0.50 0.72 0.69 0.95 1.25 1.54 2.08 1.32 1.36 0.92 0.95 0.65 0.69 0.69

> 10 mm/h 88 89 97 99 100 99 99 98 98 94 92 90 96 82 93

RMSE (%) 1-10 mm/h 74 80 85 100 104 109 136 157 158 117 85 93 78 73 86

< 1 mm/h 161 125 172 149 213 283 327 473 277 311 217 221 160 173 172

> 10 mm/h 0.16 0.40 -0.41 -0.16 -0.02 -0.06 -0.04 -0.01 0.09 -0.02 0.08 0.01 -0.09 0.47 -0.18

CC 1-10 mm/h 0.16 0.10 0.04 0.04 -0.05 -0.03 -0.02 0.01 0.00 -0.03 0.05 0.07 -0.05 0.14 0.18

< 1 mm/h 0.06 0.03 0.10 0.03 0.02 0.01 0.03 0.02 0.04 0.08 0.10 0.14 0.10 0.13 0.03

POD ≥ 0.25 mm/h 0.59 0.49 0.44 0.16 0.21 0.23 0.18 0.21 0.17 0.50 0.60 0.56 0.56 0.63 0.43

FAR ≥ 0.25 mm/h 0.84 0.91 0.87 0.82 0.80 0.80 0.85 0.76 0.81 0.77 0.77 0.83 0.86 0.83 0.90

CSI ≥ 0.25 mm/h 0.15 0.09 0.11 0.09 0.12 0.12 0.09 0.13 0.10 0.19 0.20 0.15 0.12 0.16 0.09

Fig. 43 to Fig. 46 display the time evolution of the various statistical scores.

Fig. 43 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for medium and light precipitation.


Fig. 44 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for heavy precipitation.

Fig. 45 - Time evolution of the Correlation Coefficient for the three categories of precipitation.


Fig. 46 - Time evolution of Probability of Detection, False Alarm Rate and Critical Success Index for rain rate.

We can conclude from the graphs about the H06 instantaneous scores that:

There is seasonal dependence in the Mean Error. For the Category <1 mm/h, there is slight

overestimation, except for the months April-May-June-July-August-September, where there is slight

underestimation.

For the classes 1-10 mm/h and >10 mm/h, there is always an underestimation, just as in case of H03,

which gets very large in the case of higher rain rates. This means that the H06 can neither capture

well the high intensities.

ME, SD and RMSE values are very high for the case of precipitation >10 mm/h in November 2009.

This can be due to non-expected deficiencies of the algorithms which were corrected afterwards,

because for other months, the scores are stable.

The Standard Deviation and the Root Mean Square Error follow the tendencies of the Mean Error.

However, they increase in the period April-May-June-July-August-September, where the ME for

Category <1mm decreases under zero (for Category 1-10 mm/h, the ME also decreases towards

larger negative values in this period). The reason for the increase in SD and RMSE are larger

deviations from the radar-measured rain rates. Underestimations are general in this period when

convection is quite usual.

We can depict the seasonal dependence only in the POD values, but not in the Correlation

Coefficients and the FAR, CSI values. Probability of Detection is the lowest in summer months,

contrary to the H03 product.

The Correlation Coefficient is very low in case of precipitation of intensity of <1 mm/h. However,

there is a slight improvement in the CC during the period, becoming higher than 0.1 in January and

February 2010. The Correlation Coefficient is the lowest for the summer months for the Rain Rate

Category 1-10 mm/h. There is big variability for the Category >10mm/h.

The False Alarm Rate is very high, always around 0.8.


Table 12 - Summary results of the validation of PR-ASS-1accumulated in Hungary by OMSZ

H06 v1.0 Hungary Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm


No. samples 155936 283005 294636 30297 175068 223199 276881 105681 106133 213099 250895 201748 33400 34164 18941

ME (mm) -1.35 -1.19 -1.11 -0.75 -1.45 -1.57 -1.69 -2.07 -1.77 -1.11 -1.11 -1.20 0.06 0.32 -0.30

SD (mm) 1.71 1.48 2.46 3.06 3.81 4.25 4.90 4.52 3.84 3.13 2.40 1.95 2.25 2.18 2.31

RMSE (mm) 2.18 1.90 2.70 3.15 4.08 4.53 5.19 4.97 4.23 3.32 2.64 2.29 2.25 2.21 2.33

03-h cumulate RMSE (%) 90 88 132 125 143 140 163 157 123 126 99 96 140 152 152

CC 0.31 0.30 0.19 0.27 0.27 0.21 0.20 0.07 0.07 0.23 0.38 0.33 0.16 0.27 0.10

POD 0.61 0.47 0.53 0.18 0.24 0.30 0.26 0.30 0.29 0.56 0.66 0.64 0.60 0.69 0.47

FAR 0.87 0.95 0.89 0.86 0.83 0.80 0.83 0.74 0.75 0.78 0.79 0.83 0.88 0.85 0.92

CSI 0.12 0.05 0.10 0.09 0.11 0.13 0.12 0.16 0.16 0.19 0.19 0.15 0.11 0.14 0.07

No. samples 30101 24534 41144 25224 51203 99212 55775 60864 30136 57339 49345 48531 35002 35364 20441

ME (mm) 1.05 0.13 0.31 -1.64 -2.11 -1.31 -1.72 -2.11 -1.50 0.74 0.64 1.04 0.69 0.72 0.15

SD (mm) 3.67 2.75 3.23 3.36 4.08 5.89 6.26 8.52 5.25 5.77 4.03 3.92 3.09 3.03 3.37

RMSE (mm) 3.81 2.75 3.24 3.74 4.59 6.03 6.49 8.77 5.46 5.82 4.08 4.06 3.17 3.11 3.37

06-h cumulate RMSE (%) 282 177 199 158 148 229 218 251 189 314 215 218 190 201 212

CC 0.16 0.09 0.28 0.05 0.02 0.09 0.10 0.12 0.30 0.18 0.29 0.38 0.29 0.35 0.07

POD 0.70 0.56 0.56 0.25 0.28 0.45 0.37 0.41 0.38 0.62 0.73 0.72 0.72 0.74 0.52

FAR 0.81 0.90 0.83 0.77 0.75 0.68 0.74 0.64 0.70 0.71 0.71 0.78 0.82 0.79 0.88

CSI 0.18 0.09 0.15 0.14 0.15 0.23 0.18 0.24 0.20 0.25 0.26 0.20 0.17 0.20 0.11

No. samples 26695 25112 40097 23789 47780 86184 50322 51250 26115 48872 44151 43084 30292 31150 20063

ME (mm) 2.50 1.59 1.36 -1.56 -1.59 -1.15 -1.73 -2.09 -2.22 2.42 2.21 2.27 2.01 2.96 1.16

SD (mm) 4.55 3.95 4.33 3.93 5.48 7.70 6.71 10.1 6.34 9.00 5.42 5.39 4.08 4.60 4.78

RMSE (mm) 5.19 4.26 4.53 4.23 5.71 7.79 6.92 10.4 6.72 9.32 5.85 5.85 4.55 5.47 4.92

12-h cumulate RMSE (%) 310 288 263 167 191 279 216 257 192 398 303 317 287 334 278

CC 0.28 0.21 0.37 0.07 0.19 0.07 0.12 0.18 0.26 0.34 0.34 0.41 0.27 0.34 0.14

POD 0.85 0.77 0.71 0.37 0.35 0.53 0.45 0.51 0.37 0.68 0.81 0.82 0.85 0.90 0.67

FAR 0.76 0.85 0.78 0.65 0.68 0.57 0.63 0.52 0.64 0.67 0.66 0.75 0.78 0.74 0.84

CSI 0.23 0.14 0.20 0.22 0.20 0.31 0.26 0.33 0.22 0.29 0.31 0.24 0.21 0.25 0.14

No. samples 23791 23595 33834 19286 44022 71123 47566 44613 23350 42872 36768 37770 24846 27197 16936

ME (mm) 4.94 3.25 2.64 -1.23 -1.79 -0.51 -1.46 -1.76 -2.12 4.47 4.42 4.28 2.86 5.13 1.91

SD (mm) 7.37 5.54 6.09 5.41 5.86 9.49 7.97 12.3 7.62 12.3 7.08 8.01 5.20 5.87 7.20

RMSE (mm) 8.87 6.42 6.64 5.55 6.12 9.50 8.10 12.4 7.91 13.1 8.35 9.08 5.94 7.79 7.45

24-h cumulate RMSE (%) 430 404 349 204 229 305 242 274 187 494 408 445 329 438 358

CC 0.41 0.18 0.49 0.16 0.14 0.19 0.10 0.17 0.33 0.37 0.39 0.44 0.35 0.48 0.23

POD 0.89 0.89 0.79 0.50 0.46 0.63 0.54 0.59 0.44 0.77 0.90 0.86 0.92 0.95 0.66

FAR 0.72 0.79 0.75 0.57 0.58 0.47 0.54 0.46 0.58 0.62 0.60 0.70 0.75 0.72 0.84

CSI 0.27 0.20 0.23 0.30 0.28 0.41 0.33 0.39 0.27 0.35 0.38 0.29 0.24 0.28 0.14

Fig. 47 (four panels) display the time evolution of Mean Error, Standard Deviation and Root Mean

Square Error for integration intervals 3, 6, 12 and 24 hours. Fig. 48 provides the same information for

the Correlation Coefficient.


Fig. 47 - Time evolution of Mean Error, Standard Deviation and Root Mean Square Error for the integration intervals 3, 6, 12 and 24 hours.

Fig. 48 - Time evolution of the Correlation Coefficient for the integration intervals 3, 6, 12 and 24 hours.

We can conclude from the graphs about the H06 accumulated continuous scores that:

There is seasonal dependence in the Mean Error. Except for the 3-hourly accumulations, for all the

accumulations there is general overestimation in the winter-spring-autumn seasons, while for the

months April-May-June-July-August-September, there is slight underestimation. This can be due to

convective rain rates, which are very high, and which are underestimated by the H06 product.


The 3-, 6-, 12-, and 24-hourly accumulations have similar tendencies in the Mean Error, SD and

RMSE values through the periods. As the accumulation period gets longer, these values also

increase. the biggest variability is for the 24-hourly accumulated product.

The Standard Deviation and the Root Mean Square Error are the highest in July, August for all the

accumulation periods.

The Standard Deviation and the Root Mean Square Error follow the tendencies of the Mean Error.

The SD and RMSE increase in the period where the ME decreases. The reasons for this increase are

larger deviations from the radar-measured rain rates. Underestimations are general in this period

when convection is quite usual.

The four frames of Fig. 49 display the time evolution of Probability Of Detection, False Alarm Rate and

Critical Success Index for integration intervals 3, 6, 12 and 24 hours, respectively.


Fig. 49 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success Index for integration intervals 3, 6, 12 and 24 hours (four panels).

We can conclude from the graphs about the H06 accumulated multi-categorical scores that:

We can depict the seasonal dependence clearly in the POD values, a little bit in the FAR values

(especially in the 24 hourly-accumulations), whereas the CSI values are stable.

The 3-, 6-, 12-, and 24-hourly accumulations have similar tendencies in the POD, FAR and CSI

values through the periods. As the accumulation period gets longer, these values ameliorate. The

best scores are found for the 24-hourly accumulated product.

The False Alarm Rate is quite high, always above 0.5. The Probability of Detection decreases for

the summer where there are convective cells. Thus, it is probable that convection is not very well

depicted by H06.


4.5 Validation in Italy (UniFe)

The facilities available to the University of Ferrara, and the methodology adopted for validation, are

described in section 3.4.4. The ground truth is provided by rain gauge networks.

UniFe has performed validation of PR-ASS-1 in respect of rain rate and of accumulation over 3 and 24

hours. The results are reported in Table 13 for the rate and Table 14 for the accumulated.

Table 13 - Summary results of the validation of PR-ASS-1rate in Italy by UniFe

H06 v1.0 Italy Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h

Gauge Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

Number of > 10 mm/h 3145 821 1504 1927 933 2366 1868 1126 4082 3178 2396 1728 1615 1274 890

comparisons 1-10 mm/h 94600 62019 77398 85880 16435 39687 16268 15112 46733 55093 63842 89773 81596 85927 55355

(ground obs.) < 1 mm/h 102312 68213 71788 82350 18740 38255 13903 15909 39579 50865 59332 98533 100090 104304 71646

> 10 mm/h -12.5 -13.3 -21.4 -12.3 -18.1 -15.4 -15.8 -15.6 -16.2 -15.4 -12.0 -11.2 -13.6 -12.4 -12.1

ME (mm/h) 1-10 mm/h -1.52 -1.65 -1.61 -1.46 -2.24 -2.04 -2.57 -2.60 -2.31 -1.82 -1.58 -1.37 -1.51 -1.43 -1.01

< 1 mm/h -0.02 -0.15 -0.02 0.04 -0.25 -0.04 -0.21 -0.14 -0.07 -0.04 0.01 -0.02 -0.02 -0.02 0.09

> 10 mm/h 10.9 7.18 32.4 6.90 10.3 8.56 8.07 7.44 8.21 11.21 7.94 12.47 12.74 9.57 13.9

SD (mm/h) 1-10 mm/h 2.19 1.91 2.12 2.26 2.33 2.80 2.90 2.47 2.88 2.93 2.47 2.05 2.08 1.90 2.16

< 1 mm/h 1.06 0.91 1.08 1.23 0.94 1.65 1.26 1.60 1.66 1.47 1.26 1.02 1.03 0.94 1.10

> 10 mm/h 16.6 15.1 38.8 14.1 20.8 17.6 17.8 17.3 18.2 19.1 14.4 16.7 18.7 15.7 18.4

RMSE (mm/h) 1-10 mm/h 2.67 2.53 2.66 2.69 3.24 3.46 3.88 3.58 3.69 3.45 2.93 2.47 2.57 2.37 2.38

< 1 mm/h 1.06 0.92 1.08 1.23 0.98 1.65 1.28 1.61 1.67 1.47 1.26 1.02 1.03 0.94 1.10

> 10 mm/h 80 89 90 88 98 95 97 97 95 93 85 77 86 85 76

RMSE (%) 1-10 mm/h 98 92 93 100 104 116 119 111 127 122 100 91 95 87 97

< 1 mm/h 231 195 240 268 218 369 292 373 373 350 275 219 223 209 242

> 10 mm/h -0.03 0.01 -0.17 -0.06 0.02 -0.02 0.06 0.02 0.06 -0.09 -0.04 -0.09 -0.06 -0.10 -0.14

CC 1-10 mm/h 0.24 0.17 0.16 0.21 0.07 0.07 0.05 0.01 0.07 0.13 0.20 0.29 0.18 0.25 0.29

< 1 mm/h 0.09 0.07 0.11 0.07 0.04 0.03 0.01 0.01 0.03 0.06 0.11 0.11 0.10 0.11 0.10

POD ≥ 0.25 mm/h 0.50 0.39 0.48 0.47 0.22 0.28 0.18 0.17 0.25 0.35 0.47 0.50 0.47 0.53 0.56

FAR ≥ 0.25 mm/h 0.38 0.42 0.34 0.40 0.63 0.59 0.72 0.79 0.59 0.47 0.33 0.37 0.41 0.37 0.41

CSI ≥ 0.25 mm/h 0.38 0.31 0.38 0.36 0.16 0.20 0.12 0.10 0.18 0.27 0.38 0.39 0.35 0.40 0.40

Table 14 - Summary results of the validation of PR-ASS-1accumulated in Italy by UniFe

H06 v1.0 Italy Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm

Gauge Score Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

No. samples 205174 136831 165062 180447 38282 92678 41646 40206 105765 121412 127005 193015 183115 189603 127370

ME (mm) -2.16 -2.24 -2.51 -1.89 -3.84 -3.35 -4.83 -4.04 -4.40 -3.02 -2.37 -1.81 -1.88 -1.69 -1.09

SD (mm) 6.15 4.46 8.05 5.01 6.74 7.03 8.08 6.40 9.02 7.42 5.85 5.22 5.06 2.91 4.70

RMSE (mm) 6.52 4.99 8.43 5.35 7.76 7.79 9.42 7.57 10.03 8.01 6.31 5.53 5.39 4.76 4.83

03-h cumulate RMSE (%) 130 111 113 128 109 157 134 146 151 140 120 118 123 109 1.28

CC 0.40 0.31 0.16 0.38 0.15 0.17 0.14 0.09 0.15 0.24 0.40 0.43 0.40 0.31 0.16

POD 0.53 0.43 0.53 0.53 0.27 0.38 0.29 0.24 0.33 0.44 0.55 0.55 0.52 0.58 0.59

FAR 0.38 0.40 0.32 0.40 0.61 0.56 0.63 0.76 0.55 0.42 0.31 0.35 0.39 0.36 0.41

CSI 0.40 0.34 0.43 0.39 0.19 0.26 0.19 0.14 0.24 0.34 0.44 0.42 0.39 0.44 0.42

No. samples 676668 474214 512135 608511 181294 386551 216405 232681 415630 439921 415068 652191 627867 664055 445230

ME (mm) -1.67 -2.66 -2.67 -0.23 -3.93 -3.76 -4.60 -3.94 -5.52 -3.69 -2.21 -1.56 -1.37 -1.07 0.78

SD (mm) 14.8 10.7 15.1 12.4 8.8 13.4 12.5 10.2 18.3 14.6 12.8 11.8 12.1 10.1 11.3

RMSE (mm) 14.9 11.0 15.3 12.4 9.67 13.9 13.3 10.9 19.1 15.0 13.0 11.9 12.1 10.1 11.3

24-h cumulate RMSE (%) 263 186 181 232 154 238 226 236 250 218 194 224 214 192 269

CC 0.56 0.58 0.51 0.69 0.31 0.45 0.39 0.26 0.43 0.50 0.64 0.64 0.54 0.59 0.64

POD 0.77 0.67 0.72 0.75 0.45 0.61 0.53 0.53 0.39 0.59 0.68 0.76 0.74 0.81 0.78

FAR 0.20 0.19 0.16 0.20 0.43 0.29 0.82 0.33 0.48 0.28 0.19 0.16 0.22 0.19 0.22

CSI 0.65 0.58 0.63 0.63 0.34 0.49 0.16 0.42 0.29 0.48 0.59 0.66 0.61 0.68 0.64

A preliminary analysis of the h06 performance is carried on by comparing the Probability Density

Functions (PDFs) of simulated and measured rainrates and rain amount cumulated over different

sampling times. In Fig. 50 the PDFs for simulated and measured rain patterns for January (top panels)

and July (bottom panels) 2009 are plotted in order to also highlight seasonal features for instantaneous

(left panels), 3 hourly (middle panels) and 24 hourly (right panels) cumulated rain fields.


Fig. 50 - Probability Density Function for instantaneous rain rate and accumulation periods 3 h and 24 h for the two months January and July 2009, for satellite-derived and ground measurements.

The instantaneous PDFs shows that the model is able to describe the rain rate dynamics, with a slight

underestimation, for January, of mid-high rates, while the underestimation for July is effective

especially for highest rainrates. The difference between winter and summer is also clear looking at POD

FAR and CSI values, relatively better in winter than in summer. This can be due to the well known

difficulties of regional models in predict the correct position and timing of small and medium-scale

convective systems that characterizes the summertime precipitation over Italy.

The 3 hourly cumulated PDFs show model underestimation for moderate and high rain amounts (above

10 mm) for both the considered months. The 24 hourly cumulated curves show very good matching

below 60 mm in January and underestimation for higher rain amounts, while in July there is

underestimation between 10 and 75 mm and overestimation above 75 mm.

The cumulation over 3 and 24 hours progressively increases the POD and CSI values, keeping a marked

difference between winter (generally better) and summertime values of these indices.

A deeper analysis of the instantaneous rain field is carried on with the help of Fig. 51, where the values

of the relevant statistical parameters are plotted as function of the month for the year 2009.

The Mean Error (black solid line) is always negative, indicating overall underestimation of the rainrate,

with worst performances during summer months. The Multiplicative Bias is, therefore, well below 1,

and reaching the lowest value in July.

Also RMSE and NRMSE show a weak seasonal signal, with higher values in warm months.


Fig. 51 - Time evolution of continuous statistical scores for the precipitation intensity.

The correlation coefficient is plotted in Fig. 52 as function of the months, for the instantaneous (black

line), 03 (red line) and 24 (green line) hours cumulated rain fields. The correlation coefficient, as most

of the other parameters, reaches higher values in cold months: cumulation over different time ranges

markedly increases the values.

Other parameters such as ME, RMSE and NRMSE cannot be compared for different cumulation ranges,

given the different ranges of rain amount values. Increasing the length of the cumulation window,

however, it can be seen that the seasonal sensitivity of the parameters decreases: the cumulation

mitigates the small scale convection timing/placing errors of the model.

Fig. 52 - Time evolution of the Correlation Coefficient for instantaneous rain rate and integration intervals 3 and 24 hours.


4.6 Validation in Poland (IMWM)

The facilities available to IMWM, and the methodology adopted for validation, are described in section

3.4.5. The ground truth is provided by rain gauge networks.

IMWM has performed validation of PR-ASS-1 in respect of rain rate and of accumulation over 3 hours.

The results are reported in Table 15 for the rate and Table 16 for the accumulated.

Table 15 - Summary results of the validation of PR-ASS-1rate in Poland by IMWM

H06 v1.0 Poland Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h

Gauge Class Jan Feb Mar Apr Mag Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

Number of > 10 mm/h 0 0 10 9 133 231 227 119 65 56 1 6 0 1 7

comparisons 1-10 mm/h 404 714 2323 172 2664 3653 1974 1862 907 4165 2026 1161 663 453 871

(ground obs.) < 1 mm/h 1661 3225 4192 232 2788 4247 2220 1692 1206 6290 2997 2559 2609 1840 1959

> 10 mm/h - - -13.4 -18.6 -18.4 -21.8 -24.5 -19.7 -18.2 -37.3 -10.2 -20.2 - -24.6 -14.2

ME (mm/h) 1-10 mm/h -1.05 -0.99 -1.12 -2.47 -1.78 -1.94 -2.16 -1.78 -2.01 -1.16 -0.82 -0.84 -1.11 -0.81 -1.37

< 1 mm/h -0.28 -0.09 -0.12 -0.37 -0.11 -0.21 -0.18 0.10 -0.27 -0.16 -0.02 -0.06 -0.13 -0.12 -0.20

> 10 mm/h - - 4.28 9.16 14.3 18.7 20.2 26.7 10.0 112 0.00 18.4 - 0.00 4.14

SD (mm/h) 1-10 mm/h 0.80 0.87 1.14 2.13 2.13 2.33 2.90 2.94 2.33 1.49 1.10 1.08 0.99 0.89 1.43

< 1 mm/h 0.35 0.45 0.49 0.82 1.47 1.02 1.31 2.63 0.99 0.66 0.63 0.57 0.4 0.46 0.52

> 10 mm/h - - 14.1 20.8 23.3 28.7 31.7 33.2 20.7 118 10.2 27.3 - 24.6 14.8

RMSE (mm/h) 1-10 mm/h 1.32 1.32 1.60 3.26 2.77 3.03 3.62 3.44 3.07 1.89 1.37 1.37 1.5 1.2 1.98

< 1 mm/h 0.45 0.46 0.50 0.90 1.47 1.04 1.32 2.63 1.03 0.68 0.63 0.57 0.42 0.47 0.55

> 10 mm/h - - 93 100 100 98 98 95 99 95 100 95 - 100 97

RMSE (%) 1-10 mm/h 75 68 70 91 100 103 138 162 110 80 65 71 72 64 79

< 1 mm/h 78 85 91 152 253 188 253 449 194 125 119 107 78 87 98

> 10 mm/h - - -0.61 0.69 -0.02 -0.04 -0.04 -0.09 -0.05 -0.09 - -0.15 - - -0.01

CC 1-10 mm/h -0.02 0.02 -0.01 -0.16 0.03 0.01 -0.03 -0.01 -0.01 0.06 0.12 0.17 -0.04 -0.09 0.01

< 1 mm/h 0.03 0.02 0.06 0.06 0.03 0.02 0.01 0.03 0.00 0.04 0.06 0.09 0.08 0.07 0.06

POD ≥ 0.25 mm/h 0.56 0.74 0.68 0.22 0.43 0.34 0.26 0.38 0.27 0.54 0.69 0.68 0.7 0.69 0.58

FAR ≥ 0.25 mm/h 0.85 0.79 0.72 0.89 0.60 0.68 0.79 0.59 0.73 0.54 0.63 0.77 0.81 0.80 0.78

CSI ≥ 0.25 mm/h 0.14 0.19 0.25 0.08 0.26 0.20 0.13 0.25 0.16 0.33 0.32 0.21 0.18 0.18 0.19

Table 16 - Summary results of the validation of PR-ASS-1accumulated in Poland by IMWM

H06 v1.0 Poland Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm


No. samples 1637 3215 7361 654 8358 12569 7657 5606 3427 12397 5821 3856 2559 1903 3059

ME (mm) -0.73 0.02 -0.60 -2.93 -2.02 -2.72 -3.31 -2.62 -2.40 -1.18 -0.33 -0.22 -0.26 -0.02 -0.78

SD (mm) 7.93 1.37 1.97 9.75 6.54 5.22 7.75 5.97 4.54 6.84 2.36 1.99 1.68 1.81 1.77

RMSE (mm) 7.96 1.37 2.08 10.18 6.85 5.89 8.43 6.55 5.13 6.94 2.39 2.00 1.70 1.81 1.94

03-h cumulate RMSE (%) 73 86 77 90 111 114 138 181 117 91 92 99 82 88 80

CC -0.01 0.18 0.18 0.03 0.11 0.09 0.02 0.16 0.22 0.09 0.27 0.3 0.07 0.11 0.26

POD 0.52 0.73 0.65 0.25 0.44 0.35 0.30 0.39 0.37 0.54 0.69 0.71 0.67 0.74 0.52

FAR 0.80 0.76 0.63 0.81 0.47 0.60 0.70 0.48 0.53 0.44 0.54 0.66 0.79 0.76 0.71

CSI 0.17 0.22 0.31 0.12 0.32 0.23 0.18 0.29 0.26 0.38 0.38 0.30 0.19 0.22 0.23

Analysis of rain rate

The PR-ASS-1 v.1.0 rain rate product has been validated against automatic rain gauges data. Polish

network of automatic rain gauges consists of 430 posts located all over the country, however, the

network density increases in the Southern Poland, where the flood danger is very high. Each post is

equipped with two gauges: heated and non-heated, what enables some quality control of data. For

validation purposes, readings from both gauges were compared in order to eliminate the cases of

clogged instruments. If both gauges worked properly, higher values was taken (automatic RG are known

to underestimate the real precipitation).

The measurements time resolution is 10 minutes, what allows estimating the rain rate with reasonable

quality, especially for stratiform rainfalls. The ground rain rate (in mm/h) was calculated from 10

minute cumulative values from the timeslot closest to the satellite overpass assuming that the real

precipitation rate was constant within that time span.


In order to combine satellite products with rain gauges data, the following simple method was applied.

For each satellite pixels, the automatic posts situated within that pixel were found. If more than one rain

gauge were found within one satellite pixel, the ground rain rate value was calculated as a mean of all

rain gauges measurements within that pixel.

Results

Following the methodology agreed in the H-SAF precipitation validation group, both continuous and

categorical statistics were calculated on the monthly mean basis for three precipitation categories:

- category 1: >10 mm/h,

- category 2: 1-10 mm/h,

- category 3: <1 mm/h

In Fig. 53 and Fig. 54, values of the Mean Error, and RMSE % calculated for the three categories and

for each month of the analysed period are presented. It should be pointed out here that the analysis was

performed only for situations when the ground rain rate was 0.25 mm/h

-40

-35

-30

-25

-20

-15

-10

-5

0

5

Jan

'09

Feb

'09

Mar

'09

Apr

'09

Ma

y'0

9

Jun

'09

Jul'

09

Au

g'0

9

Sep'

09

Oct

'09

Nov

'09

Dec

'09

Jan

'10

Feb

'10

Mar

'10

ME

> 10 mm/h

1-10 mm/h

< 1 mm/h

Fig. 53 - Mean error (ME) of PR-ASS-1 v.1.0 rain rate product for the period of Jan 2009 – Mar 2010 for Poland.

The PR-ASS-1 v.1.0 underestimates the measured rain rate values for the whole analysed period. The

underestimation is stronger in summer and early autumn, especially for high precipitation category (>10

mm/h). For this category, the seasonal variability of ME is the strongest.

0

50

100

150

200

250

300

350

400

450

500

Jan'

09

Feb'

09

Mar

'09

Apr

'09

May

'09

Jun'

09

Jul'0

9

Aug

'09

Sep'

09

Oct

'09

Nov

'09

Dec

'09

Jan'

10

Feb'

10

Mar

'10

RMSE %

> 10 mm/h

1-10 mm/h

< 1 mm/h

Fig. 54 - RMSE % of PR-ASS-1 v.1.0 rain rate product for the period of Jan 2009 – Mar 2010 for Poland.


Similarly to ME, the quality of the product in rain rate estimation also reveals seasonal variability: it

decreases in spring and summer and increases in late autumn and winter. However, amplitude of these

changes is different for each category i.e. the smallest for high precipitation category (> 10 mm/h) and

the biggest for light precipitation (<1 mm/h). The last one includes light rain cases that are were not

detected by PR-ASS-1 v.1.0 rain rate product.

The quality of PR-ASS-1 v.1.0 in precipitaion detection was validated using the dichotomous (yes/no)

statistics. The 0.25 mm/h threshold was used for rain/no rain differentiation. In Fig. 55 the variability of

Probability of Detection and False Alarm Ratio are presented.

0

0,2

0,4

0,6

0,8

1

1,2

1,4

1,6

1,8

Jan

'09

Feb'

09

Mar

'09

Ap

r'0

9

May

'09

Jun

'09

Jul'

09

Aug

'09

Sep

'09

Oct

'09

No

v'0

9

Dec

'09

Jan

'10

Feb'

10

Mar

'10

FAR

POD

Fig. 55 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-ASS-1 v.1.0 rain rate product using Polish RG data in the period of Jan 2009 – Mar 2010.

The quality of PR-ASS-1 rain rate product v.1.0 in precipitation detection undergoes seasonal

variability: it increases in winter and autumn and decreases in spring and summer, what would indicate

the problem with convective precipitation detection. For the most of the analysed period, the POD is

lower than FAR, however in winter and autumn this difference is very small. Only in November the

POD is bigger than FAR. So, it can be concluded that the quality of PR-ASS-1 v.1.0 rain rate product in

precipitation recognition is the best in autumn and winter.

Analysis of accumulated precipitation

The PR-OBS-6 3 hour cumulated precipitation v.1.0 product has been validated against automatic rain

gauges data. Polish network of automatic rain gauges consists of 430 posts located all over the country,

however, the network density increases in the Southern Poland, where the flood danger is very high.

Each post is equipped with two gauges: heated and non-heated, what enables some quality control of

data. For validation purposes, readings from both gauges were compared in order to eliminate the cases

of clogged instruments. If both gauges worked properly, higher values was taken (automatic RG are

known to underestimate the real precipitation).

The measurements time resolution is 10 minutes. The ground 3 hour cumulated precipitation was

calculated by aggregating 10 minute cumulative values from 3 hours before the satellite timeslot.

In order to combine satellite products with rain gauges data, the following simple method was applied.

For each satellite pixels, the automatic posts situated within that pixel were found. If more than one rain

gauge were found within one satellite pixel, the ground rain rate value was calculated as a mean of all

rain gauges measurements within that pixel.

Results

Following the methodology agreed in the H-SAF precipitation validation group, both continuous and

categorical statistics were calculated on the monthly mean basis.


In Fig. 56 and Fig. 57, values of the mean error and RMSE calculated on the base of all PR-ASS-1 3h

cumulated precipitation data are presented for each month of the analysed period. It should be pointed

out here that the analysis was performed only for situations when the accumulated rain on the ground

was 1.0 mm.

-3,5

-3

-2,5

-2

-1,5

-1

-0,5

0

0,5

Jan

'09

Feb

'09

Mar

'09

Ap

r'09

May

'09

Jun

'09

Jul'0

9

Au

g'09

Sep

'09

Oct

'09

No

v'09

De

c'09

Jan

'10

Feb

'10

Mar

'10

ME (mm)

Fig. 56 - Mean error (ME) of PR-ASS-1 3 h cumulated precipitation v.1.0 for the period of Jan 2009 – Mar 2010 for Poland.

For the whole analysed period, the PR-ASS-1 underestimates the measured 3-hour cumulated

precipitation. The underestimation is stronger in the spring and summer months. The seasonal

variability of the quality of PR-ASS-1 in estimation of 3 hour cumulated precipitation was found – the

worst results were obtained for summer and the best in winter.

0

20

40

60

80

100

120

140

160

180

200

Jan

'09

Feb

'09

Mar

'09

Ap

r'09

May

'09

Jun

'09

Jul'0

9

Au

g'09

Sep

'09

Oct

'09

No

v'09

De

c'09

Jan

'10

Feb

'10

Mar

'10

RMSE (%)

Fig. 57 - RMSE (%) of PR-ASS-1 3-hour cumulated precipitation v.1.0 for the period of Jan 2009 – Mar 2010 for Poland.

The quality of PR-ASS-1 3-hour cumulated precipitation v.1.0 product in precipitaion detection was

validated using the dichotomous (yes/no) statistics. The 1.0 mm threshold was used for rain/no rain

differentiation. In Fig. 58 the variability of Probability of Detection and False Alarm Ratio is presented.


0

0,2

0,4

0,6

0,8

1

1,2

1,4

1,6

Jan

'09

Feb

'09

Mar

'09

Ap

r'09

May

'09

Jun

'09

Jul'0

9

Au

g'09

Sep

'09

Oct

'09

No

v'09

De

c'09

Jan

'10

Feb

'10

Mar

'10

FAR

POD

Fig. 58 - Variabily of Probability of Detection (POD), False Alarm Ratio (FAR) obtained for PR-ASS-1 3-hour cumulated precipitation v.1.0 product using Polish RG data for the period of Jan 2009 – Mar 2010.

Both, POD and FAR undergo seasonal variability but they changes in different way. The POD is lower

in spring and summer, while FAR in autumn. Therefore it should be concluded that the ability of PR-

ASS-1 3-hour cumulated product of precipitation detection is the best in autumn and winter and worse

in summer.


4.7 Validation in Slovakia (SHMÚ)

The facilities available to SHMÚ, and the methodology adopted for validation, are described in section


SHMÚ has performed full validation of PR-ASS-1, i.e. the precipitation rate and the accumulated



Table 17 - Summary results of the validation of PR-ASS-1rate in Slovakia by SHMÚ

H06 v1.0 Slovakia Land 00 h Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 0.25 mm/h


Number of > 10 mm/h 0 0 1 24 109 238 216 124 23 0 0 0 0 0 0

comparisons 1-10 mm/h 496 544 2144 2039 5883 12846 7691 7888 3831 3504 1555 1745 120 322 114

(ground obs.) < 1 mm/h 12595 15947 21460 5462 16880 21574 10861 17018 11285 19679 18355 14061 9932 7679 5621

> 10 mm/h - - -10.7 -15.6 -14.6 -15.2 -15.3 -14.7 -13.6 - - - - - -

ME (mm/h) 1-10 mm/h -0.77 -0.93 -0.66 -1.81 -1.69 -1.73 -1.98 -1.38 -0.99 -1.09 -0.61 -0.76 -0.75 -0.65 -0.99

< 1 mm/h 0.09 0.02 0.14 -0.37 0.01 -0.06 -0.12 0.06 0.05 0.17 0.29 0.24 0.22 -0.02 -0.11

> 10 mm/h - - 0.00 6.07 5.20 5.71 5.85 5.16 4.42 - - - - - -

SD (mm/h) 1-10 mm/h 1.19 0.58 1.20 1.39 2.00 1.97 2.11 2.60 2.70 1.37 1.34 1.18 0.54 0.69 0.63

< 1 mm/h 0.62 0.47 0.72 0.43 0.97 1.25 1.37 1.90 1.88 0.91 0.90 0.97 0.56 0.58 0.43

> 10 mm/h - - 10.7 16.7 15.5 16.2 16.4 15.6 14.3 - - - - - -

RMSE (mm/h) 1-10 mm/h 1.42 1.10 1.37 2.28 2.62 2.62 2.89 2.94 2.88 1.75 1.47 1.40 0.93 0.94 1.18

< 1 mm/h 0.63 0.47 0.73 0.56 0.97 1.25 1.38 1.90 1.88 0.92 0.94 1.00 0.60 0.58 0.45

> 10 mm/h - - 89 98 100 98 99 99 96 - - - - - -

RMSE (%) 1-10 mm/h 79 78 76 92 114 108 115 148 163 91 104 87 70 69 87

< 1 mm/h 186 131 195 114 225 295 331 430 373 242 247 266 185 158 118

> 10 mm/h - - - -0.17 -0.07 -0.10 -0.11 0.08 -0.07 - - - - - -

CC 1-10 mm/h -0.18 -0.10 0.13 0.04 -0.09 -0.02 -0.04 -0.04 0.00 -0.01 -0.06 -0.05 -0.13 0.22 0.23

< 1 mm/h 0.04 0.03 0.08 0.08 0.06 0.02 0.00 0.03 0.10 0.02 0.09 0.08 0.00 0.08 0.09

POD ≥ 0.25 mm/h 0.52 0.53 0.55 0.15 0.39 0.31 0.22 0.30 0.33 0.48 0.60 0.54 0.68 0.41 0.35

FAR ≥ 0.25 mm/h 0.86 0.89 0.84 0.81 0.69 0.79 0.84 0.71 0.68 0.84 0.83 0.87 0.90 0.94 0.94

CSI ≥ 0.25 mm/h 0.12 0.10 0.14 0.09 0.21 0.14 0.10 0.17 0.20 0.13 0.15 0.12 0.09 0.06 0.05

Graphical deployment of selected validation parameters for PR-ASS-1rate in Slovakia is in Fig. 59.

Mean Error for H06-00 v1.0

-17

-15

-13

-11

-9

-7

-5

-3

-1

1

Jan

2009

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

2010

Feb Mar

Me

an

Err

or

[mm

/h]

1-10mm/h

<1mm/h

>10mm/h

Mean Error for H06-00 v1.0

-2,5

-2

-1,5

-1

-0,5

0

0,5

1

Jan

2009


2010

Feb Mar

Me

an

Err

or

[mm

/h]

1-10mm/h

<1mm/h

RMSE [%] for H06-00 v1.0

0%

50%

100%

150%

200%

250%

300%

350%

400%

450%

Jan

2009


2010

Feb Mar

>10mm/h

1-10mm/h

<1mm/h

Scores of dichotomous statistics for H06-00 v1.0

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

Jan

2009


2010

Feb Mar

Sc

ore

s

POD

FAR

CSI

Fig. 59 - Time evolution of Mean Error, Root Mean Square Error (%) and dichotomous scores for rain rate.


Scores of dichotomous characteristics show that in general POD is too low and FAR is too high. The

highest POD=0.68 was reached in January 2010 and the lowest FAR=0.68 in September 2009. The

highest CSI=0.21 was reached in May 2009. Seasonal variability can be observed on POD values. POD

is lower in summer months probably due to convective season.

Seasonal variability is well observed on RMSE values in case of precipitation rate RR<1 mm/h. The

best RMSE = 69 % occurred in February 2010 for rain rate 1<RR<10mm/h.

ME shows acceptable performance for rain rate RR<10mm/h but very high underestimation was found

in cases when RR>10mm/h.

Table 18 - Summary results of the validation of PR-ASS-1accumulated in Slovakia by SHMÚ

H06 v1.0 Slovakia Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm


No. samples 5169 6754 9612 6437 18597 37131 22604 24604 11884 16700 9748 8973 2447 2439 1337

ME (mm) 0.40 -0.11 0.21 -1.77 -1.08 -1.33 -1.79 -0.72 -0.14 0.19 0.92 0.84 0.49 0.01 -0.43

SD (mm) 1.87 1.40 1.98 2.00 3.27 4.01 4.59 4.54 4.10 2.96 2.88 3.12 1.31 1.64 1.23

RMSE (mm) 1.91 1.40 1.99 2.67 3.44 4.23 4.92 4.60 4.10 2.96 3.02 3.23 1.40 1.64 1.31

03-h cumulate RMSE (%) 146 105 139 89 144 170 191 204 181 178 215 223 119 128 97

CC 0.31 0.02 0.30 0.04 -0.13 0.03 -0.06 0.03 0.17 0.04 0.10 0.11 -0.05 0.08 0.00

POD 0.55 0.50 0.56 0.17 0.37 0.34 0.27 0.32 0.39 0.52 0.62 0.58 0.62 0.49 0.31

FAR 0.92 0.94 0.91 0.74 0.70 0.75 0.77 0.69 0.67 0.85 0.90 0.89 0.97 0.97 0.98

CSI 0.07 0.06 0.09 0.11 0.20 0.17 0.14 0.19 0.22 0.13 0.09 0.10 0.03 0.03 0.02

No. samples 18123 24504 24548 12685 32519 63954 39812 41279 21485 38625 29195 22437 14274 8746 5431

ME (mm) 2.25 0.53 0.77 -1.81 -0.83 -0.75 -1.09 -0.10 0.27 1.21 2.03 1.78 1.98 0.81 0.04

SD (mm) 4.45 2.06 2.79 2.26 4.08 5.39 6.17 6.58 4.81 4.27 3.98 4.58 2.14 2.53 1.72

RMSE (mm) 4.98 2.13 2.90 2.90 4.16 5.44 6.27 6.58 4.82 4.44 4.47 4.91 2.92 2.65 1.72

06-h cumulate RMSE (%) 373 146 198 93 165 205 239 246 190 251 282 288 235 199 128

CC 0.13 0.11 0.29 0.05 -0.02 0.15 0.04 0.11 0.25 0.11 0.20 0.14 0.01 -0.02 0.14

POD 0.72 0.63 0.59 0.22 0.42 0.47 0.39 0.40 0.50 0.61 0.75 0.70 0.87 0.63 0.49

FAR 0.80 0.84 0.84 0.66 0.54 0.62 0.65 0.56 0.58 0.71 0.75 0.82 0.86 0.91 0.93

CSI 0.18 0.15 0.15 0.15 0.28 0.27 0.23 0.27 0.30 0.24 0.23 0.17 0.13 0.08 0.06

No. samples 36873 63475 49760 21301 48283 96983 65595 61832 33830 75634 60677 47860 43893 18838 11979

ME (mm) 3.91 1.38 1.27 -1.73 -0.24 -0.22 -0.34 0.16 0.51 2.71 3.92 3.01 4.10 2.97 0.78

SD (mm) 5.55 2.76 3.36 2.57 5.48 6.64 7.36 7.31 5.36 6.30 5.17 5.91 3.73 3.70 2.22

RMSE (mm) 6.79 3.09 3.59 3.10 5.48 6.64 7.36 7.31 5.38 6.85 6.49 6.63 5.54 4.75 2.36

12-h cumulate RMSE (%) 395 191 246 102 192 235 284 259 185 376 352 347 372 346 171

CC 0.30 0.26 0.14 0.06 0.13 0.20 0.07 0.15 0.29 0.16 0.38 0.23 0.29 -0.02 0.02

POD 0.77 0.77 0.65 0.25 0.45 0.56 0.49 0.46 0.55 0.73 0.83 0.83 0.89 0.87 0.73

FAR 0.72 0.71 0.75 0.59 0.43 0.50 0.55 0.44 0.52 0.54 0.60 0.73 0.72 0.84 0.86

CSI 0.26 0.27 0.22 0.19 0.33 0.36 0.31 0.34 0.35 0.40 0.37 0.26 0.27 0.15 0.13

No. samples 61518 125536 90126 30119 59595 135549 102604 83210 47525 117568 100896 101377 91779 38314 14516

ME (mm) 5.50 2.70 2.30 -1.92 -0.80 -0.19 0.74 0.33 0.63 4.44 5.75 4.35 6.00 6.34 2.72

SD (mm) 6.78 3.99 4.63 3.00 5.47 6.47 8.12 7.94 5.76 9.75 6.85 7.33 5.99 5.42 3.21

RMSE (mm) 8.73 4.82 5.17 3.56 5.52 6.47 8.15 7.95 5.79 10.72 8.94 8.52 8.48 8.34 4.20

24-h cumulate RMSE (%) 412 267 322 108 157 226 295 242 188 496 410 430 453 563 318

CC 0.48 0.29 0.14 0.13 0.22 0.23 0.13 0.22 0.30 0.19 0.59 0.26 0.49 0.17 -0.05

POD 0.79 0.85 0.73 0.26 0.43 0.58 0.62 0.52 0.64 0.77 0.82 0.84 0.89 0.93 0.84

FAR 0.67 0.58 0.62 0.54 0.33 0.37 0.41 0.32 0.47 0.41 0.50 0.62 0.60 0.78 0.84

CSI 0.30 0.39 0.33 0.20 0.36 0.43 0.43 0.42 0.41 0.50 0.45 0.35 0.38 0.22 0.16

Graphical representations of selected validation parameters for PR-ASS-1accumulated in Slovakia are in

Fig. 60 (for Mean Error and Root Mean Square Error) and Fig. 61 (for dichotomous scores).


Mean Error for H06-03,06,12,24 v1.0

-2,50

-1,50

-0,50

0,50

1,50

2,50

3,50

4,50

5,50

6,50

Jan

2009

Mar May Jul Sep Nov Jan

2010

Mar

Me

an

Err

or

[mm

/h]

Period 3h

Period 6h

Period 12h

Period 24h

RMSE [%] for H06-03,06,12,24 v1.0

0%

100%

200%

300%

400%

500%

600%

Jan

2009

Mar May Jul Sep Nov Jan

2010

Mar

Period 3h

Period 6h

Period 12h

Period 24h

Fig. 60 - Time evolution of Mean Error and Root Mean Square Error (%), function of the integration interval 3, 6, 12 and 24 h.

Mean error of PR-ASS-1 accumulated demonstrates very strong seasonal variability. This variability is very

significant for long integration periods (24 hours) and reduced in case of short integration period (3

hours). While during summer we observed slight underestimation, during winter it was overestimation

of precipitation. RMSE is too high in winter period, especially for longer time integration periods (12,

24 hours).

Dichotomous statistics - POD for H06-03,06,12,24 v1.0

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90

1,00

Jan

2009


2010

Feb Mar

Sc

ore

s

POD 3h

POD 6h

POD 12h

POD 24h

Fig. 61 - Time evolution of Probability Of Detection, False Alarm Rate and Critical Success Index, function of the integration interval (3, 6, 12 and 24 h).

Dichotomous statistics parameters

POD, FAR and CSI also demonstrate

seasonal variability and confirm better

performance of PR-OBS-1accumulated for

winter period which is typical by

absence of convection in central

Europe. POD is typically higher and

FAR lower for longer cumulated time

periods. CSI reached the maximum

value of 0.5 in October 2009 for 24

hour cumulated period, POD=0.77 and

FAR=0.41 for this month. Absolute

maximum POD=0.93 was reached in

February 2010 and absolute minimum

FAR=0.32 in August 2009, both values

for the longest integration time period

24 hours.

Dichotomous statistics - FAR for H06-03,06,12,24 v1.0

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90

1,00

Jan

2009


2010

Feb Mar

Sc

ore

s

FAR 3h

FAR 6h

FAR 12h

FAR 24h

Dichotomous statistics - CSI for H06-03,06,12,24 v1.0

0,00

0,10

0,20

0,30

0,40

0,50

0,60

0,70

0,80

0,90

1,00

Jan

2009


2010

Feb Mar

Sc

ore

s

CSI 3h

CSI 6h

CSI 12h

CSI 24h


4.8 Validation in Turkey (ITU)

The facilities available to ITU (in collaboration with TSMS), and the methodology adopted for

validation, are described in section 3.4.7. The ground truth is provided by rain gauge networks.

ITU and TSMS have performed validation of PR-ASS-1 accumulated precipitation over 3 and 24 hours.

Separate statistics are reported for inner land and coastal zones in Table 19 and Table 20, respectively.

Table 19 - Summary results of the validation of PR-ASS-1accumulated in Turkey by ITU and TSMS over inner land

H06 v1.0 Turkey Land Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm


No. samples 67614 245327 142723 76476 53254 28699 29716 5803 51179 79742 114836 195630 200368 193036 94046

ME (mm) -0.36 -0.06 -0.11 -1.63 -1.68 -3.19 -3.52 -2.80 -3.34 -0.67 -0.44 -0.73 -0.40 -0.28 -0.28

SD (mm) 2.63 2.92 2.27 2.92 3.16 4.71 4.37 2.71 5.77 3.39 3.58 3.63 2.24 3.14 2.04

RMSE (mm) 2.66 2.92 2.27 3.34 3.57 5.69 5.61 3.90 6.67 3.45 3.60 3.70 2.28 3.16 2.06

03-h cumulate RMSE (%) 139 135 120 104 114 117 117 100 140 159 127 132 110 128 106

CC 0.16 0.32 0.36 0.31 0.16 0.06 0.08 -0.01 0.08 0.24 0.43 0.34 0.47 0.46 0.27

POD 0.51 0.53 0.54 0.39 0.40 0.14 0.15 0.018 0.21 0.32 0.52 0.52 0.49 0.53 0.50

FAR 0.45 0.45 0.54 0.62 0.63 0.69 0.78 0.96 0.70 0.50 0.42 0.39 0.49 0.44 0.50

CSI 0.36 0.37 0.33 0.24 0.24 0.11 0.10 0.01 0.14 0.24 0.38 0.39 0.33 0.37 0.33

No. samples 721196 845555 714936 347073 262853 183373 149410 40727 224855 384553 454379 766484 873949 840744 523407

ME (mm) 1.10 1.10 0.71 -1.47 -1.40 -3.65 -4.82 -3.16 -3.91 -0.87 0.09 -0.80 0.02 0.61 0.11

SD (mm) 8.62 8.13 5.25 6.59 7.62 6.76 9.45 4.01 14.09 6.46 9.22 8.32 5.69 7.69 4.13

RMSE (mm) 8.69 8.20 5.30 6.75 7.75 7.68 10.61 5.11 14.62 6.52 9.22 8.35 5.69 7.71 4.13

24-h cumulate RMSE (%) 144 151 136 144 209 125 159 120 237 167 134 124 128 163 107

CC 0.59 0.54 0.55 0.41 0.46 0.27 0.36 0.03 0.23 0.44 0.56 0.58 0.60 0.70 0.59

POD 0.81 0.79 0.79 0.67 0.51 0.24 0.31 0.16 0.47 0.48 0.80 0.77 0.78 0.75 0.74

FAR 0.10 0.12 0.17 0.26 0.39 0.34 0.48 0.61 0.30 0.15 0.08 0.14 0.13 0.13 0.17

CSI 0.75 0.71 0.68 0.54 0.38 0.21 0.24 0.13 0.39 0.44 0.74 0.68 0.70 0.67 0.65

Table 20 - Summary results of the validation of PR-ASS-1accumulated in Turkey by ITU and TSMS over coastal zones

H06 v1.0 Turkey Coast Validation period: 1st January 2009 - 31 March 2010 - Threshold rain / no rain: 1 mm


No. samples 38577 12482 8618 2774 1343 1077 990 200 3121 5998 6501 10654 11972 11577 5382

ME (mm) -0.28 -0.44 -0.24 -1.77 -2.67 -4.06 -4.22 -3.07 -4.78 -0.57 -0.83 -0.32 -0.4 -0.3 -0.3

SD (mm) 3.08 3.71 3.10 3.30 3.31 7.41 7.07 2.96 9.19 4.35 3.90 5.27 2.8 4.0 2.3

RMSE (mm) 3.09 3.74 3.11 3.75 4.25 8.45 8.23 4.26 10.36 4.38 3.99 5.28 2.9 4.0 2.4

03-h cumulate RMSE (%) 136 146 143 129 99 318 147 97 198 214 122 164 129 141 114

CC 0.35 0.35 0.37 0.10 0.08 0.00 0.04 0.03 0.05 0.26 0.49 0.34 0.50 0.50 0.40

POD 0.54 0.48 0.50 0.39 0.32 0.18 0.23 0.05 0.25 0.35 0.41 0.57 0.49 0.56 0.47

FAR 0.26 0.34 0.40 0.60 0.63 0.58 0.75 0.96 0.65 0.42 0.29 0.28 0.38 0.29 0.34

CSI 0.45 0.38 0.37 0.24 0.21 0.14 0.14 0.02 0.17 0.28 0.35 0.47 0.37 0.46 0.38

No. samples 34551 38816 34587 11236 7012 6020 4587 1656 11040 20460 20531 32187 39091 40123 20966

ME (mm) -0.20 -0.84 -0.04 -1.06 -2.79 -5.55 -6.37 -3.34 -7.96 -0.34 -2.20 0.67 0.30 -0.09 -0.30

SD (mm) 9.05 9.47 7.41 8.15 6.74 10.5 13.9 3.98 23.0 9.94 10.4 12.9 8.41 10.5 4.40

RMSE (mm) 9.05 9.51 7.41 8.21 7.29 11.9 15.2 5.20 24.4 9.94 10.6 12.9 8.42 10.5 4.41

24-h cumulate RMSE (%) 128 126 156 145 159 129 285 97 250 249 122 144 139 197 88

CC 0.61 0.53 0.57 0.46 0.45 0.28 0.39 0.21 0.23 0.41 0.59 0.60 0.59 0.66 0.72

POD 0.74 0.71 0.72 0.67 0.46 0.26 0.37 0.22 0.53 0.57 0.74 0.76 0.75 0.73 0.75

FAR 0.08 0.10 0.11 0.25 0.29 0.27 0.53 0.76 0.27 0.15 0.06 0.10 0.12 0.07 0.12

CSI 0.70 0.66 0.66 0.55 0.39 0.24 0.26 0.13 0.44 0.52 0.71 0.70 0.68 0.69 0.68

Fig. 62 shows the time evolution for inner land of Mean Error, Standard Deviation, Root Mean Square

Error, Probability Of Detection, False Alarm Rate and Critical Success Index for accumulation periods 3

and 24 hours. Fig. 63 shows the same for coastal zones. Fig. 64 shows the Correlation Coefficient for

inner land and coastal zone, for accumulation periods 3 and 24 hours.


Mean Error, Standart Deviation and RMS for H06_03 (Land)

-6

-4

-2

0

2

4

6

8

Jan

2009

Feb Mar

Apr

May Ju

nJu

lAgu

Sep

.O

ctNov

Dec

Jan

2010

Feb MarS

co

res (

mm

)

ME (mm)

SD (mm)

RMSE (mm)

Mean Error, Standart Deviation and RMS for H06_24 (Land)

-10

-5

0

5

10

15

20

Jan

2009

Feb Mar

Apr

May Ju

nJu

l

Agu

Sep

.Oct

Nov

Dec

Jan

2010

Feb Mar

Sc

ore

s (

mm

)

RMSE (mm)

SD (mm)

ME (mm)

Multi-categorical Scores for H06_03 (Land)

0

0.2

0.4

0.6

0.8

1

1.2

Jan

2009

Feb Mar

Apr

May Ju

nJu

l

Agu

Sep

.Oct

Nov

Dec

Jan

2010

Feb Mar

Sc

ore

s

POD

FAR

CSI

Multi-categorical Scores for H06_24 (Land)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Jan

2009

Feb Mar

Apr

May Ju

nJu

l

Agu

Sep

.Oct

Nov

Dec

Jan

2010

Feb Mar

Sc

ore

s

POD

FAR

CSI

Fig. 62 - Continuous and multi-categorical statistics for inner land (3 and 24 hourly accumulation).

Mean Error, Standart Deviation and RMS for H06_03 (Coast)

-6

-4

-2

0

2

4

6

8

10

12

Jan

2009

Feb Mar

AprM

ay Jun

Jul

AguSep

.O

ctNov

Dec

Jan

2010

Feb Mar

Sco

res (

mm

)

ME (mm)

SD (mm)

RMSE (mm)

Mean Error, Standart Deviation and RMS for H06_24 (Coast)

-10

-5

0

5

10

15

20

25

30

Jan

2009

Feb Mar

Apr

May Ju

nJu

l

Agu

Sep

.Oct

Nov

Dec

Jan

2010

Feb Mar

Sc

ore

s (

mm

)

RMSE (mm)

SD (mm)

ME (mm)

Multi-categorical Scores for H06_03 (Coast)

0

0.2

0.4

0.6

0.8

1

1.2

Jan

2009

Feb Mar

Apr

May Ju

nJu

l

Agu

Sep

.Oct

Nov

Dec

Jan

2010

Feb Mar

Sc

ore

s

POD

FAR

CSI

Multi-categorical Scores for H06_24 (Coast)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Jan

2009

Feb Mar

Apr

May Ju

nJu

l

Agu

Sep

.Oct

Nov

Dec

Jan

2010

Feb Mar

Sc

ore

s

POD

FAR

CSI

Fig. 63 - Continuous and multi-categorical statistics for coastal zones (3 and 24 hourly accumulation).

Correlation Coefficients for Accumulated (Land)

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Jan

2009

Feb Mar

Apr

May Ju

nJu

l

Agu

Sep

.Oct

Nov

Dec

Jan

2010

Feb Mar

Sc

ore

s

CC - 3 Hourly

CC - 24 Hourlya

Correlation Coefficients for Accumulated (Coast)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Jan

2009

Feb Mar

Apr

May Ju

nJu

l

Agu

Sep

.Oct

Nov

Dec

Jan

2010

Feb Mar

Sc

ore

s

CC - 3 Hourly

CC - 24 Hourlya

Fig. 64 - Correlation coefficients for inner land and coastal zones (3 and 24 hourly accumulation).


The following interpretations can be drawn from graphs concerning inner land and coastal precipitation.

When the patterns in Figures 62 and 63 are compared each other, very similar patterns between

inner land and coastal zones for multi-categorical and continuous statistics can be seen.

Maximum False Alarm Rate (FAR), minimum Probability of Detection (POD) and Critical Success

Index (CSI) can be seen in August for all hourly accumulations. FAR increases till August and then

it decreases. When POD and CSI are compared with FAR, reverse tendency can be seen for POD

and CSI as expected. Moreover, the 24-hourly accumulated product has best multi-categorical

statistics.

The best Correlation Coefficients (CC) are found for the 24-hourly accumulated product in Fig. 64

and CC are very low in summer months.

There is underestimation between April and October because Mean Error (ME) has negative value.

For other months, it is almost zero.


5. Overview of findings

5.1 Synopsis of validation results

In the various sections of Chapter 4 the validation results have been quoted separately by each Team

operating on a different geographic area associated to a proper climatic condition. This is correct, since

the precipitation field is affected by orography and local climatology. In this section a synoptic

overview is provided, of the results achieved in the different countries, and in different seasons.

In order to reduce the volume of data to be commented, and not to overlap with the detailed reports in

Chapter 4, the results are summarised by seasons of three months each. The phase is:

Spring: Summer: Autumn: Winter:

March, April and May 2009 June, July and August 2009 Sept., Oct. and Nov. 2009 Dec. 2009, Jan. and Feb. 2010

Table 21 and Table 22, split in four sections, one for each season, report the Country/Team results side-

by-side. There are three sets of columns:

one set for four Countries/Teams that compared satellite data with meteorological radar in inner land

areas: Belgium/IMR, Germany/BfG, Hungary/OMSZ and Slovakia/SHMÚ; and their average

weighed by the number of comparisons;

one set for three Countries/Teams that compared satellite data with rain gauges in inner land areas:

Italy/UniFe, Poland/IMWM and Turkey/ITU; and their average weighed by the number of

comparisons;

one column for Turkey/ITU that compared satellite data with rain gauges in coastal zones.

It is noted that not all Countries/Teams performed validation of all products, as shown in the following

map (shadowed cells: no validation).

Belgium Germany Hungary Italy Poland Slovakia Turkeyland Turkeycoast

PR-ASS-1 00 h

PR-ASS-1 03 h

PR-ASS-1 06 h

PR-ASS-1 12 h

PR-ASS-1 24 h

It is reminded, from Chapter 4 Table 07, that the User requirements are:

Table 07 - Accuracy requirements for product PR-ASS-1 [RMSE (%)]

Product threshold target optimal Additional specifications

Precipitation rate 100 50 25 To be validated for RR < 1.0, 1-10 and > 10 mm/h.

To be verified for RR in the range 1-10 mm/h

Accumulated precipitation 200 100 50 To be validated for integration over 3, 6, 12 and 24 h.

To be verified for integration over 24 h

The unit for accuracy specification is Root Mean Square Error percent (RMSE %), used since error

grows with rate. In Chapter 4 further scores were recorded: Mean Error (or bias, ME), Standard

Deviation (SD) and Correlation Coefficient (CC), Probability Of Detection (POD), False Alarm Rate

(FAR) and Critical Success Index (CSI). In this summary Chapter 5, in order to streamline the

discussion and minimize repetition with Chapter 4, we only focus on RMSE (%), ME (mm/h) and POD

/ FAR.

Tables 21 and 22 highlight the RMSE (%) rows by yellow colour, the weighed average column by blue

and the averaged RMSE (%) values (first thing to attract the attention) by green. The column for

“coast”, being single, is not averaged.


Table 21 - Comparative results of validation in several Countries/Teams split by season. PR-ASS-1 00 h (rain rate)

PR-ASS-1rate Spring IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast

Version 1.0 2009 radar radar radar radar radar gauge gauge gauge gauge gauge

> 10 mm/h 1,593 541 134 2,268 4,364 152 4,516

N. of samples 1-10 mm/h 183,246 41,187 10,066 234,499 179,713 5,159 184,872

< 1 mm/h 428,793 132,971 43,802 605,566 172,878 7,212 180,090

> 10 mm/h -16.9 -16.6 -14.7 -16.7 -16.7 -18.1 -16.7

ME (mm/h) 1-10 mm/h -1.22 -1.65 -1.49 -1.31 -1.60 -1.51 -1.60

< 1 mm/h -0.22 -0.16 0.03 -0.19 -0.02 -0.12 -0.02

> 10 mm/h 92 100 99 94 91 99 91

RMSE (%) 1-10 mm/h 133 98 101 125 97 86 97

< 1 mm/h 158 182 196 166 251 155 247

POD ≥ 0.25 mm/h 0.21 0.29 0.43 0.24 0.45 0.55 0.45

FAR ≥ 0.25 mm/h 0.73 0.83 0.77 0.75 0.40 0.67 0.41

PR-ASS-1rate Summer IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast


> 10 mm/h 6,076 3,657 578 10,311 5,360 577 5,937

N. of samples 1-10 mm/h 245,953 100,346 28,425 374,724 71,067 7,489 78,556

< 1 mm/h 232,768 144,552 49,453 426,773 68,067 8,159 76,226

> 10 mm/h -14.8 -18.0 -15.1 -15.9 -15.6 -22.4 -16.3

ME (mm/h) 1-10 mm/h -1.85 -2.08 -1.70 -1.90 -2.28 -1.96 -2.25

< 1 mm/h -0.15 -0.11 -0.03 -0.12 -0.10 -0.14 -0.10

> 10 mm/h 94 98 98 96 96 97 96

RMSE (%) 1-10 mm/h 115 130 121 119 116 127 117

< 1 mm/h 255 341 349 295 354 260 344

POD ≥ 0.25 mm/h 0.18 0.21 0.29 0.20 0.23 0.33 0.24

FAR ≥ 0.25 mm/h 0.81 0.80 0.78 0.80 0.66 0.69 0.66

PR-ASS-1rate Autumn IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast


> 10 mm/h 1,709 778 23 2510 9,656 122 9,778

N. of samples 1-10 mm/h 226,404 55,901 8,890 291,195 165,668 7,098 172,766

< 1 mm/h 435,778 135,507 49,319 620,604 149,776 10,493 160,269

> 10 mm/h -13.4 -19.2 -13.6 -15.2 -14.9 -26.9 -15.0

ME (mm/h) 1-10 mm/h -1.32 -1.23 -0.96 -1.29 -1.87 -1.17 -1.84

< 1 mm/h -0.18 0.16 0.19 -0.08 -0.03 -0.13 -0.04

> 10 mm/h 95 96 96 95 92 97 92

RMSE (%) 1-10 mm/h 89 116 124 95 115 79 113

< 1 mm/h 166 268 274 197 326 131 313

POD ≥ 0.25 mm/h 0.26 0.47 0.48 0.32 0.37 0.55 0.38

FAR ≥ 0.25 mm/h 0.68 0.78 0.79 0.71 0.45 0.59 0.46

PR-ASS-1rate Winter IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast

Version 1.0 2009/10 radar radar radar radar radar gauge gauge gauge gauge gauge

> 10 mm/h 102 90 0 192 4,617 7 4,624

N. of samples 1-10 mm/h 202,975 33,129 2,187 238,291 257,296 2,277 259,573

< 1 mm/h 788,275 143,829 31,672 963,776 302,927 7,008 309,935

> 10 mm/h -12.6 -13.3 - -12.9 -12.4 -20.8 -12.4

ME (mm/h) 1-10 mm/h -0.99 -0.94 -0.74 -0.98 -1.43 -0.91 -1.43

< 1 mm/h -0.30 0.13 0.17 -0.22 -0.02 -0.10 -0.02

> 10 mm/h 100 90 - 95 82 96 82

RMSE (%) 1-10 mm/h 85 83 83 85 91 70 91

< 1 mm/h 114 188 214 128 217 91 214

POD ≥ 0.25 mm/h 0.21 0.58 0.55 0.27 0.50 0.69 0.50

FAR ≥ 0.25 mm/h 0.65 0.84 0.90 0.69 0.38 0.79 0.39


Table 22 - Comparative results of validation in several Countries/Teams split by season. PR-ASS-1 accumulated precip.

PR-ASS-1acc Spring IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast


N. of samples 578,568 500,001 34,646 1,113,215 383,791 16,373 272,453 672,617 12,735

ME (mm) -1.26 -1.21 -0.85 -1.22 -2.35 -1.42 -0.84 -1.72 -0.83

3 h RMSE (%) 133 135 132 134 120 95 114 117 135

POD 0.37 0.41 0.39 0.39 0.50 0.53 0.47 0.49 0.46

FAR 0.52 0.87 0.77 0.68 0.39 0.56 0.58 0.47 0.47

N. of samples 935,156 117,571 69,752 1,122,479

ME (mm) -1.25 -1.16 -0.45 -1.19

6 h RMSE (%) 165 168 163 165

POD 0.50 0.37 0.44 0.48

FAR 0.45 0.78 0.67 0.50

N. of samples 1,587,263 111,666 119,344 1,818,273

ME (mm) -1.43 -0.52 0.12 -1.27

12 h RMSE (%) 202 212 198 202

POD 0.59 0.48 0.50 0.58

FAR 0.36 0.71 0.59 0.40

N. of samples 44,387 2,638,946 97,142 179,840 2,960,315 1,301,940 1,324,862 2,626,802 52,835

ME (mm) -2.37 -1.66 -0.14 0.57 -1.49 -1.71 -0.28 -0.99 -0.62

24 h RMSE (%) 149 222 266 231 223 201 152 176 154

POD 0.70 0.65 0.58 0.55 0.64 0.70 0.70 0.70 0.67

FAR 0.29 0.27 0.64 0.51 0.30 0.22 0.24 0.23 0.16

PR-ASS-1acc Summer IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast


N. of samples 612,563 605,761 84,339 1,302,663 174,530 25,832 64,218 264,580 2,267

ME (mm) -1.98 -1.71 -1.28 -1.81 -3.86 -2.87 -3.31 -3.63 -4.04

3 h RMSE (%) 149 153 185 153 149 136 115 139 224

POD 0.31 0.28 0.32 0.30 0.33 0.34 0.13 0.28 0.19

FAR 0.67 0.80 0.74 0.73 0.62 0.60 0.76 0.65 0.69

N. of samples 1,023,481 215,851 145,045 1,384,377

ME (mm) -1.88 -1.64 -0.66 -1.71

6 h RMSE (%) 178 232 226 191

POD 0.43 0.42 0.43 0.43

FAR 0.56 0.68 0.61 0.58

N. of samples 1,718,760 187,756 224,410 2,130,926

ME (mm) -1.74 -1.56 -0.15 -1.56

12 h RMSE (%) 202 256 256 212

POD 0.54 0.50 0.51 0.53

FAR 0.46 0.57 0.50 0.47

N. of samples 45,473 2,878,938 163,302 321,363 3,409,076 835,637 373,510 1,209,147 12,263

ME (mm) -3.01 -1.58 -1.13 0.24 -1.41 -4.03 -4.06 -4.04 -5.56

24 h RMSE (%) 205 225 278 252 230 234 138 204 183

POD 0.51 0.63 0.59 0.58 0.62 0.57 0.26 0.47 0.30

FAR 0.32 0.35 0.49 0.37 0.36 0.44 0.43 0.44 0.43


PR-ASS-1acc Autumn IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast


N. of samples 663,053 570,127 38,332 1,271,512 354,182 21,645 245,757 621,584 15,620

ME (mm) -1.41 -1.23 0.27 -1.28 -3.20 -1.14 -1.12 -2.31 -1.52

3 h RMSE (%) 110 113 188 114 136 95 140 136 172

POD 0.40 0.55 0.51 0.47 0.45 0.55 0.39 0.43 0.35

FAR 0.49 0.78 0.81 0.63 0.42 0.48 0.50 0.45 0.41

N. of samples 1,081,399 136,820 89,305 1,307,524

ME (mm) -1.39 0.21 1.25 -1.04

6 h RMSE (%) 123 251 246 145

POD 0.49 0.61 0.63 0.51

FAR 0.40 0.71 0.69 0.45

N. of samples 1,825,397 119,138 170,141 2,114,676

ME (mm) -1.49 1.33 2.70 -0.99

12 h RMSE (%) 125 318 329 152

POD 0.56 0.66 0.73 0.58

FAR 0.30 0.66 0.56 0.34

N. of samples 30,055 2,992,214 102,990 265,989 3,391,248 1,270,619 1,063,787 2,334,406 52,031

ME (mm) -1.08 -1.66 2.96 4.26 -1.05 -3.81 -1.10 -2.58 -2.69

24 h RMSE (%) 151 132 394 408 162 221 168 197 199

POD 0.71 0.62 0.74 0.77 0.64 0.55 0.61 0.58 0.63

FAR 0.22 0.22 0.60 0.45 0.25 0.32 0.15 0.24 0.14

PR-ASS-1acc Winter IRM BfG OMSZ SHMÚ Total UniFe IMWM ITU Total ITUcoast

Version 1.0 2009/10 radar radar radar radar radar gauge gauge gauge gauge gauge

N. of samples 827,677 269,312 13,859 1,110,848 565,733 8,318 589,034 1,163,085 34,203

ME (mm) -1.30 -0.85 0.63 -1.17 -1.79 -0.19 -0.47 -1.11 -0.34

3 h RMSE (%) 91 108 188 96 117 91 123 120 144

POD 0.31 0.64 0.57 0.39 0.55 0.70 0.51 0.53 0.54

FAR 0.40 0.84 0.92 0.51 0.37 0.72 0.44 0.41 0.32

N. of samples 1,423,592 118,897 45,457 1,587,946

ME (mm) -1.54 0.84 1.66 -1.27

6 h RMSE (%) 93 205 254 106

POD 0.38 0.73 0.74 0.42

FAR 0.29 0.79 0.85 0.34

N. of samples 2,459,129 104,526 110,591 2,674,246

ME (mm) -1.97 2.40 3.44 -1.58

12 h RMSE (%) 93 313 357 112

POD 0.44 0.85 0.86 0.47

FAR 0.20 0.76 0.74 0.24

N. of samples 48,643 3,930,494 89,813 231,470 4,300,420 1,944,113 2,481,177 4,425,290 111,401

ME (mm) -0.27 -2.68 4.14 5.33 -2.08 -1.33 -0.03 -0.60 0.27

24 h RMSE (%) 126 93 411 461 120 210 139 170 161

POD 0.83 0.50 0.90 0.87 0.53 0.77 0.77 0.77 0.75

FAR 0.36 0.12 0.72 0.64 0.16 0.19 0.13 0.16 0.10

5.2 Summary conclusions on comparative elements

In the various sections of Chapter 4 the Countries/Teams have concluded with highlighting the main

positive aspects of the product and the main failures, according to the experience on their area of

investigation. This Section does in no way bias the original conclusions in the different sections of

Chapter 4, but only refers to features stemming from observation of Tables 21 and 22.

It is noted that this validation cycle was the first performed by the H-SAF validation Units, the previous

having been performed internally to CNMCA. As for all other products, the comparison were made in

the native product grid, that in this case coincides with the grid points of the COSMO-ME model.


Variability with geographical area

Rain rate (PR-ASS-1 00 h)

It may be observed that the performances are rather consistent across the various geographical areas,

especially for heavy (> 10 mm/h) and medium (1-10 mm/h) precipitation. This in spite of the fact that,

for this first cycle of “external” validation, the comparison methodology was not yet fully standardised.

Even for inner lands and coastal areas the performance are similar rather.

Accumulated (PR-ASS-1 03. 06, 12 and 24 h)

The performance is similar in spring, but in all other seasons sharply degrades in Hungary and Slovakia.

Variability with validation tool

Rain rate (PR-ASS-1 00 h) and Accumulated (PR-ASS-1 03. 06, 12 and 24 h)

The performances resulting from validation by radar and those by rain gauges are rather similar. This is

very important because User should not mind about which tool has been used for the validation: the

information on the performance is regarded as a property of the product, not of the ground truth.

Variability with season

Rain rate (PR-ASS-1 00 h) and Accumulated (PR-ASS-1 03. 06, 12 and 24 h)

The performances are rather similar across seasons. The FAR is rather high for rain rate in all seasons,

much better for accumulated precipitation. The POD is relatively high for accumulated precipitation,

especially in summer and autumn.

Variability with precipitation type (or intensity) or integration interval

Rain rate (PR-ASS-1 00 h)

Heavy and medium precipitation have very similar performances through all seasons and geographical

areas, whereas for light precipitation there is a substantial degradation in spring and autumn, and very

substantial in summer.

Accumulated (PR-ASS-1 03. 06, 12 and 24 h)

Slowly degrading with increasing integration interval.

Overall observation on compliance with User requirements

This final comment does not replace an in-depth analysis of the compliance between user requirements

and satellite-derived product performance, that would deserve a much detailed discussion. However,

with the understanding that this is just a rough overall assessment, Table 23 offers a view on the subject.

It is reminded that the verification is due for rate in the range 1-10 mm/h and accumulated over 24 h.

Table 23 - Simplified compliance analysis for product PR-ASS-1

Between target and optimal Between threshold and target Threshold exceeded by < 50 % Threshold exceeded by ≥ 50 %

PR-ASS-1 v1.0 Spring Summer Autumn Winter

Benchmark Requirement (RMSE %) radar gauge gauge radar gauge gauge radar gauge gauge radar gauge gauge

range thresh target optimal land land coast land land coast land land coast land land coast

Rate (1-10 mm/h) 100 50 25 125 97 119 117 95 113 85 91

Cumulate (24 h) 200 100 50 223 176 154 230 204 183 162 197 199 120 170 161

Compact presentation of validation results

Since it has been noted above that the variability of results with validation tool (radar or gauge), as well

as across the various Countries, and between inner land and coastal zones, are not very pronounced, it

may be useful to synthesis all results in a presentation that averages among the various situations only

leaving differentiation with season. This is provided in Table 24.


Table 24 - Synthesis of all validation results, including yearly average

PR-ASS-1 Version 1.0 Spring 2009 Summer 2009 Autumn 2009 Winter 2009/10 Yearly average

Instantaneous precipitation

> 10 mm/h 6,784 16,248 12,288 4,816 40,136

N. of samples 1-10 mm/h 419,371 453,280 463,961 497,864 1,834,476

< 1 mm/h 785,656 502,999 780,873 1,273,711 3,343,239

> 10 mm/h -16.7 -16.0 -15.0 -12.4 -15.4

ME (mm/h) 1-10 mm/h -1.44 -1.96 -1.49 -1.21 -1.52

< 1 mm/h -0.15 -0.12 -0.07 -0.17 -0.13

> 10 mm/h 92 96 93 82 93

RMSE (%) 1-10 mm/h 113 119 102 88 105

< 1 mm/h 184 302 221 149 197

POD ≥ 0.25 mm/h 0.30 0.21 0.34 0.34 0.31

FAR ≥ 0.25 mm/h 0.65 0.78 0.64 0.59 0.65

Accumulated precipitation

N. of samples 1,798,567 1,569,510 1,908,716 2,308,136 7,584,929

ME (mm) -1.40 -2.12 -1.62 -1.13 -1.52

3 h RMSE (%) 128 151 122 109 125

POD (≥ 1 mm) 0.43 0.30 0.46 0.46 0.42

FAR (≥ 1 mm) 0.60 0.72 0.57 0.46 0.57

N. of samples 1,122,479 1,384,377 1,307,524 1,587,946 5,402,326

ME (mm) -1.19 -1.71 -1.04 -1.27 -1.31

6 h RMSE (%) 165 191 145 106 149

POD (≥ 1 mm) 0.48 0.43 0.51 0.42 0.46

FAR (≥ 1 mm) 0.50 0.58 0.45 0.34 0.46

N. of samples 1,818,273 2,130,926 2,114,676 2,674,246 8,738,121

ME (mm) -1.27 -1.56 -0.99 -1.58 -1.37

12 h RMSE (%) 202 212 152 112 165

POD (≥ 1 mm) 0.58 0.53 0.58 0.47 0.53

FAR (≥ 1 mm) 0.40 0.47 0.34 0.24 0.35

N. of samples 5,639,952 4,630,486 5,777,685 8,837,111 24,885,234

ME (mm) -1.25 -2.11 -1.68 -1.31 -1.53

24 h RMSE (%) 200 223 176 145 179

POD (≥ 1 mm) 0.67 0.58 0.62 0.65 0.63

FAR (≥ 1 mm) 0.27 0.38 0.24 0.16 0.24

Products Validation Report, 30 May 2010 - PVR-06 (Product PR-ASS-1) - Appendix

Appendix to PVR-06 (Instantaneous and accumulated precipitation at ground

computed by a NWP model)

Collection of validation experiment reports

(extracted from REP-3/07 dated 28 February 2010)

INDEX

2. Validation figures from the COSMO Consortium


2. Validation figures from the COSMO Consortium

2.1 Information source and method

Product PR-ASS-1 is basically the accumulated precipitation generated by the COSMO-ME operational

model, integrated over several intervals (3, 6, 12 and 24 hours) including 0 hours, i.e. the instantaneous

precipitation. The performances of the COSMO model are routinely monitored by the COSMO

Consortium, and several output features are systematically recorded. The H-SAF parameters are not

directly monitored, but CNMCA has used the records to extract a few significant performance

parameters, as follows:

only over continent, since only raingauge networks are used for comparison;

only accumulated precipitation, over 6, 12 and 24 h;

categorisation over three accumulated precipitation amounts: < 1 mm ; 1 to 10 mm ; > 10 mm

selected statistical scores:

- Frequency BIas (FBI)

- Probability Of Detection (POD)

- False Alarm Rate (FAR)

- Equitable Threat Score (ETS).

The statistical scores stem from the Contingency Table.

Contingency Table

Observed

yes no total

yes hits false alarms forecast yes

Forecast no misses correct negatives forecast no

total observed yes observed no total

where:

- hit: event forecast to occur, and actually occurred

- miss: event forecast not to occur, but actually occurred

- false alarm: event forecast to occur, but actually not occurred

- correct negative: event forecast not to occur, and actually not occurred

The statistical scores are defined as follows.

yesobserved

yesforecast

misseshits

alarmsfalsehitsFBI Range: 0 to infinity. Perfect score: 1

yesobserved

hits

misseshits

hitsPOD Range: 0 to 1. Perfect score: 1

yesforecast

alarmsfalse

alarmsfalsehits

alarmsfalseFAR Range: 0 to 1. Perfect score: 0

random

random

hitsalarmfalsemisseshits

hitshitsETS with

total

yesforecastyesobservedhitsrandom

ETS ranges from -1/3 to 1. 0 indicates no skill. Perfect score: 1. This score measures the fraction of

observed and/or forecast events that were correctly predicted, adjusted for hits associated with random

chance (for example, it is easier to correctly forecast rain occurrence in a wet climate than in a dry

climate). The ETS is often used in the verification of rainfall in NWP models because its "equitability"

allows scores to be compared more fairly across different regimes. Sensitive to hits. Because it

penalises both misses and false alarms in the same way, it does not distinguish the source of forecast

error.


2.2 Summary tables of accuracy

Products PR-ASS-1 have been available for H-SAF purposes since early 2008 and since then have been

regularly distributed. PR-ASS-1 actually provides five products:

precipitation rate, and accumulated precipitation over 3, 6, 12 and 24 h.

The following tables record certain accuracy scores for accumulated precipitation (AP), derived at the

CNMCA from the monitoring activity of the COSMO Consortium. It only refers to accumulation

periods of 6, 12 and 24 h over continental areas. Performances are provided for various classes of AP.

The tables are being produced for H-SAF at quarterly intervals, starting on September 2008.

Rolling statistics of comparisons of PR-ASS-1 accumulated precipitation against rain gauges.

Period: 1 September 2008 - 31 August 2009. Number of comparisons: 5000 per quarter. Continental areas.

Accuracy Class Integration over 6 h Integration over 12 h Integration over 24 h

score of AP Sep-Nov Dec-Feb Mar-May Jun-Aug Sep-Nov Dec-Feb Mar-May Jun-Aug Sep-Nov Dec-Feb Mar-May Jun-Aug

> 10 mm 0.80 0.75 0.77 0.82 0.70 0.80 0.75 0.85 0.90 1.20 0.85 0.85

FBI 1-10 mm 1.10 1.25 1.00 0.97 1.00 1.20 1.10 1.21 1.00 1.20 1.10 0.94

< 1 mm 1.20 1.30 1.10 1.12 1.10 1.20 1.20 1.58 1.10 1.10 1.00 1.04

> 10 mm 0.30 0.20 0.35 0.29 0.20 0.30 0.25 0.35 0.30 0.50 0.35 0.37

POD 1-10 mm 0.50 0.40 0.55 0.32 0.50 0.50 0.55 0.44 0.55 0.65 0.60 0.45

< 1 mm 0.60 0.70 0.60 0.38 0.60 0.75 0.70 0.53 0.75 0.80 0.75 0.54

> 10 mm 0.80 0.85 0.75 0.64 0.20 0.30 0.25 0.59 0.65 0.75 0.55 0.57

FAR 1-10 mm 0.70 0.65 0.60 0.65 0.50 0.50 0.55 0.62 0.45 0.50 0.40 0.53

< 1 mm 0.50 0.45 0.40 0.66 0.40 0.38 0.40 0.66 0.35 0.28 0.30 0.48

> 10 mm 0.10 0.10 0.15 0.19 0.15 0.15 0.20 0.23 0.20 0.20 0.25 0.23

ETS 1-10 mm 0.17 0.20 0.20 0.19 0.25 0.30 0.28 0.23 0.35 0.35 0.35 0.26

< 1 mm 0.40 0.38 0.45 0.2 0.38 0.35 0.40 0.23 0.38 0.40 0.45 0.3

Period: 1 September 2009 - 30 November 2009. Number of comparisons: 5000 per quarter. Continental areas.

Accuracy Class Integration over 6 h Integration over 12 h Integration over 24 h

score of AP Sep-Nov Dec-Feb Mar-May Jun-Aug Sep-Nov Dec-Feb Mar-May Jun-Aug Sep-Nov Dec-Feb Mar-May Jun-Aug

> 10 mm 0.84 0.97 0.92

FBI 1-10 mm 0.89 1.03 0.95

< 1 mm 0.92 1.09 0.99

> 10 mm 0.30 0.49 0.51

POD 1-10 mm 0.42 0.58 0.59

< 1 mm 0.54 0.68 0.68

> 10 mm 0.64 0.50 0.45

FAR 1-10 mm 0.53 0.44 0.38

< 1 mm 0.41 0.38 0.31

> 10 mm 0.19 0.31 0.32

ETS 1-10 mm 0.23 0.36 0.37

< 1 mm 0.35 0.42 0.43


2.3 Main factors affecting accuracy

The ability of a NWP model to predict precipitation strongly depends on the structure and the objective

of the model. A short list of factors affecting quantitative precipitation forecasting follows:

COSMO-ME is a limited-area model (LAM). It is optimised for the small-synoptic scale (few tens

of kilometres wavelengths). for multi-purpose application at national level.

Although the grid-mesh of the model could be reduced to < 3 km. it is not a convection-resolving

model (CRM). Cloud microphysical parameters are carried forward in the model to an extent

limited to fulfil the multi-purpose national requirements.

In any event. a CRM would provide precipitation representation less biased than a LAM. but its use

in operational weather prediction would experience the intrinsic limits of predictability of the

atmosphere at the scale of convection.

The model handles precipitation better as concerns accumulated precipitation. less well as concerns

instantaneous precipitation. The information on precipitation is inferred through the energy and

water budget. that requires sufficiently large time-space scale to install and stabilise. For H-SAF.

accumulated precipitation is computed for 3. 6. 12 and 24 time intervals. and precipitation rate also

is provided. but verification for quality control is applied only for the 6. 12 and 24 hour intervals.

Of course. the quality of precipitation data is linked to the actual daily accuracy of the forecast. that

changes with the availability and quality of the observations for analysis and assimilation on the

specific date.

product validation report - servizio...

Documents