the ipwg inter-comparison effortipwg/meetings/bologna-2016/bologna2016_orals/5 … · a national...

The IPWGInter-comparison EffortChris Kidd, Beth Ebert, John Janowiak,Joe Turk, Daniel Vila, Matt Sapiano, Shoichi Shige, Estelle de Coning, Ralph Ferraro, Vincenzo Levizzani, Phil Arkin

The Whys and WhereforesIPWG objectives:

#2) Establish standards for validation and independent verification of precipitation measurements

#3) Foster the exchange of data on inter-comparisons of operational precipitation measurements from satellites

• Need to provide regional-scale validation of precipitation products.

• Building upon work of the GPCP Algorithm Inter-comparison Programme (AIP1-3) and the WetNet Precipitation Inter-comparison Projects (PIP1-3).

• On-going inter-comparisons using common graphical, descriptive and statistical analysis of daily, 0.25° resolution products.

• Algorithm developers make products available for comparisons done in local ‘validation’ centres.

BESTEFFORTBASIS8th IPWGand5th IWSSMworkshop,3-7October2016Bologna,Italy

GV-IPWG synergies

Criteria GV program IPWG

Type of validation Priority on physical, also statistical

Has focused on descriptive and statistical

Source of validation data Arranged for and collected by principle investigators

Doesn't request. IPWG participants free to contribute

Source of observational data Specific satellite-based products

participants provide products directly to validation groups

Types of Validation data Gauge, radars and specialist instrumentation, diverse in specific locations

Conventional gauge and/or radar networks, usually part of a national network

Types of observational data single-sensor, instantaneous, full-resolution datasets

Blended satellite sensor products, time/area averaged.

AfterTurk&Arkin,BAMS2008

Bothapproachesarecomplementary

GV=GroundValidation

8th IPWGand5th IWSSMworkshop,3-7October2016Bologna,Italy

IPWG Inter-comparison regionsNearreal-timeinter-comparisonofmodel&satelliteestimatesvsradar/gauge

IPWG- http://www.isac.cnr.it/~ipwg/8th IPWGand5th IWSSMworkshop,3-7October2016Bologna,Italy

Accesing IPWG intercomparisons

http://www.isac.cnr.it/~ipwg/


MainIPWGInter-comparison

Websites

Regional Web pages - Japan

Dailyimagesandstatisticsandsummaryplots!Probablythebestrunwebpageatpresent!


Regional Web pages - US

Dailyimagesandstatistics.(night-timecolour scheme?!)


Regional Web pages - Europe

Limiteddailyimagesandstatistics.

BUT– NEW!– instantaneouscomparisons


Processing systemInitialsetup:SettingofdatesCleaningoutold/decayeddata

Acquiringdata:SearchingexistingdataListingmissingdataCreationof.netrcfileftpdatasources

Remappingofdata:…toregionalgridor5kmPSGprojection…

Resultsgeneration:StatisticalanalysisGraphicaloutput

Webpages:GenerateHTMLfilesCopyingtoserver


Processing checksforeachday(d0-d0-31)

dn=dn+1

setd0=today

foreachday(d0-d0-31)

foreachproduct(p1-pn)

if(productforday)!exist

addto.netrcfile

foreachdatasource(s0-sn)

ftpdatasource(4k)

N

Y

foreachproduct&day

remaptoPSGusingLUTs&standardiseformat

standardisefilename

foreachproduct&day

Generatestatistics

Generateplots

foreachproduct&day

generateHTMLfiles


Processing checksforeachday(d0-d0-31)

dn=dn+1

setd0=today

foreachday(d0-d0-31)

foreachproduct(p1-pn)

if(productforday)!exist

addto.netrcfile

foreachdatasource(s0-sn)

ftpdatasource(4k)

YN

foreachproduct&day

remaptoPSGusingLUTs&standardiseformat

standardisefilename

foreachproduct&day

Generatestatistics

Generateplots

foreachproduct&day

generateHTMLfiles

Setuplistofpastdates/daysUsuallyokay:

sometimesneedstweaking

Checksforaproductsresults:Okayifnoresults,butnotifbaddata

Preparesproductsintocommon

formatUsuallyokay…

Generatesoutputs:

Okayifthereisrain…

GeneratesrawHTML:Occasionalissues

withserver

FTPrunsseveraltimes:

4Kbufferlimitonmacros

AutomatedsystemstheyareNOT!8th IPWGand5th IWSSMworkshop,3-7October2016Bologna,Italy

What’s new?Growing interest in the inter-comparison of precipitation retrievals at the full-spatial/instantaneous resolution, but:• Requires much more processing;• Data often in different formats, resolutions, sampling;• More comparisons – orbital (~2-3/sat) vs daily (1)Nevertheless• Europe – same region as daily estimates and US –

mapping done for eastern US (better quality radar)• About 32,000 comparisons/year/region for GPM alone• Assumes a ‘uniform’ 15 km footprint• EU radar (15 min at 52°N), US radar (2 min at 37.5°N).


MainIPWGInter-comparison

Websites

European Instantaneous Inter-comparisons


Regional analysis: Western EuropeAMSR2 GMI SSMIS

ATMSDPR-NS MHS-V04MHS-V03

0.00.10.20.30.40.50.60.70.80.9Correlation

EUMETSAT Conference, 26-30 September 2016Goddard Space

Flight Center

Regional analysis: United States

AMSR2 GMI SSMIS

ATMSDPR-NS MHS-V04MHS-V03

0.00.10.20.30.40.50.60.70.80.9Correlation


Flight Center

Inter-satellite retrieval comparisons

GMI-AMSR2 338cases 6,919,090fovsGMI-SSMIS 1673cases 11,718,314fovsGMI-MHS 1732cases 4,466,330fovsGMI-ATMS 444cases 1,156,309fovs

(basedupon<=5kmfov matchups)

GPM orbit crosses all

constellation satellite orbits thus allowing inter-satellite comparisons to be made. 60s,nadircrossingmatchups(2015)


Flight Center

Instantaneous matchupsGMI AMSR2

2015

-04-17

050

3UTC

NAtla

ntic

GMI ATMS

2015

-06-07

160

9UTC

Papu

aNew

Guine

a

GMI MHS

2015

-05-12

230

8UTC

PacificOcean

GMI SSMIS

2015

-01-23

103

4UTC

Indian

Ocean

GMI SSMIS20

15-03-13

071

4UTC

NorthAtla

ntic

GMI AMSR2

2015

-02-09

235

3UTC

NPacific(Ha

waii)


Flight Center

00.11.010100GMIrainrate(mmh-1)

GMIrainrate(mmh-1)00.11.010100



100

10

1.0

0.1

0

AMSR

2

106

105

104

103

102

101

#ob

servations

Instantaneous retrievals GMI vs sensor100

10

1.0

0.1

0

SSMIS

100

10

1.0

0.1

0

ATMS

100

10

1.0

0.1

0

MHS

-V03


100

10

1.0

0.1

0

MHS

-V04

<=60smatch<=5kmco-location

Intercomparison findingsStatistical metrics:• are at best imprecise measures of performance, and are

very different between daily/instantaneous – which are the best ones to use?

• rain/no-rain boundary is visually important, but has little effect on some statistics (e.g. correlation)

Resolution: • previously 0.25/daily, but each instantaneous product has

different resolutionsNumber of products: • there are many more instantaneous products than daily

products (per overpass vv per day).


Conclusions• First and foremost, the IPWG inter-comparisons are carried

out on a ‘best effort’ basis;

• Many challenges face ‘inter-comparisons’ – particularly from ‘not-just’ rain products (i.e. POPs, error bars, high-resolution/instantaneous);

• Great care is needed in the interpretation of inter-comparisons, yet alone the statistics;

• Observational capability is not the same as retrieval ability (e.g. SSMIS ≠ GPROF);

• The inter-comparisons are very reliant upon the contributors (or lack of – modellers take note!).


HELP!Need for guidelines/best practices on:

• Resolutions:- Are the retrievals Gaussian weighted or ’flat’?- What is the ‘true’ retrieval resolution (incl. channel weights)?- Do we need a ‘standard’ retrieval resolution (yes!) ?

• Over-sampling at edge-of-scan (conicals/imagers)

• Co-located channel fovs (e.g. SSMI vs SSMIS/etc)

• Deterioration of GV data with resampling (e.g. MRMS EqArea-LatLon-EqArea)

• True impact of parallax?


Advice….• Developers: • always look at your results – instantaneous

retrievals, not just averages!• For instantaneous estimates, map the log10 of

the rainrate – it can be very revealing!• Users:• Statistics are useful, but rarely convey the true

usefuless of the product

3-hour

day

5-day

Month

VAR vs. HQ (mm/hr) Feb. 2002 30°N-S

Huffman 2/10

Fine-scale data allows users to decide the averaging strategy

Space-timedependency

At full resolution the ‘accuracy’ of estimated rain is low; averaging over time and space improves the picture

IPWG#5,Hamburg,11-15October2010

Statistical chasm….

Perfo

rmance

Temporal/spatialscaleGoodphysicalcoincidence Goodsamplingpoorsampling

The future of IPWG validation regions

Suggested analysis levels:• real time analysis: 'correctness' of near real-time

products: should these be easily available (cics?)• 'historical' analysis: 'authorised' release of quality-

controlled products • high-resolution analysis: similar set-up to daily/0.25

degree products (including instantaneous)• 'training' data sets: example data sets from a

comprehensive range of precipitation products –common naming convention and format

4th InternationalPrecipitationWorkingGroup(IPWG)Workshop13-17October2008Beijing,China

Personal recommendations

Evaluation of combined techniques:• individual component products for combined algorithm• actual combination techniques

Standardisation of precipitation product format• simple 2D arrays with accompanying text files• common format –variable size• quantitative resolution – 0.1 mm/day (WMO standard)• error fields can be accompanying files• Naming conventions standardised

4th InternationalPrecipitationWorkingGroup(IPWG)Workshop13-17October2008Beijing,China

TheIPWGgauntlet:….togenerateglobal1km,1minuterainfallestimates….

Multi-product comparisons

Satellite(andmodel)precipitationestimatesaregenerallysimilar,but

differindetail

EarthObservationfortheWaterCycle,ESRIN,20-23October2015GoddardSpaceFlightCenter

the ipwg inter-comparison effortipwg/meetings/bologna-2016/bologna2016_orals/5 … · a national...

Documents