update on the collaborative radar acquisition field test (craft): planning for the next steps kelvin...
TRANSCRIPT
Update on the Collaborative Update on the Collaborative Radar Acquisition Field Radar Acquisition Field
Test (CRAFT): Test (CRAFT): Planning for the Next StepsPlanning for the Next Steps
Kelvin K. DroegemeierKelvin K. DroegemeierUniversity of OklahomaUniversity of Oklahoma
Presented toPresented toNEXRAD Technical Advisory CommitteeNEXRAD Technical Advisory Committee
3 December 20013 December 2001
NCDC
s fn
The Collaborative Radar The Collaborative Radar Acquisition Field Test Acquisition Field Test
(CRAFT)(CRAFT) A Collaborative project to demonstrate the real A Collaborative project to demonstrate the real
time compression and Internet-based time compression and Internet-based transmission of WSR-88D base (Level II) datatransmission of WSR-88D base (Level II) data
Motivated by Motivated by – Storm-scale NWP research at CAPS (now for WRF)Storm-scale NWP research at CAPS (now for WRF)– Decision support system development (NSSL)Decision support system development (NSSL)– Desire to improve long-term archive rate (NCDC)Desire to improve long-term archive rate (NCDC)
Philosophy: Leverage existing infrastructures Philosophy: Leverage existing infrastructures (Internet), public-domain software (Unidata (Internet), public-domain software (Unidata LDM), and low-cost services (56K phone lines)LDM), and low-cost services (56K phone lines)
CRAFT was established by the OU Center CRAFT was established by the OU Center for Analysis and Prediction of Storms for Analysis and Prediction of Storms (CAPS) in collaboration with the (CAPS) in collaboration with the – UCAR Unidata Program OfficeUCAR Unidata Program Office– NOAA WSR-88D Operational Support Facility NOAA WSR-88D Operational Support Facility
(now Radar Operations Center)(now Radar Operations Center)– University of WashingtonUniversity of Washington– National Severe Storms LaboratoryNational Severe Storms Laboratory
Now coordinated by CAPS, NSSL, ROC, Now coordinated by CAPS, NSSL, ROC, Unidata, and NCDC Unidata, and NCDC
s fn
ParticipantsParticipants
s fn
Current Technical StrategyCurrent Technical Strategy
Repeater Hub
RIDDS Linux PC + LDM + Compression
($1000)
LDM Server
Dedicated 56K line or ($2000 - $6000/year)
or NWS LANOptional Router
($2000)
WSR-88D
At the radar site
Users
Internet or Abilene
BZIP2 off-the-shelf compression algorithmBZIP2 off-the-shelf compression algorithm– maximum 16:1 in clear air; 8:1 in the worst case; maximum 16:1 in clear air; 8:1 in the worst case;
average of about 12:1average of about 12:1 56K lines found to be inadequate in some cases 56K lines found to be inadequate in some cases
(squall lines) – maybe 10% of time(squall lines) – maybe 10% of time– Exceed available bandwidth by 10-20%Exceed available bandwidth by 10-20%– Can create latencies of 10s of minutesCan create latencies of 10s of minutes– Up to 26% improvement in compression possible in Up to 26% improvement in compression possible in
BZIP2, but requires changing code at user endBZIP2, but requires changing code at user end– A pre-processing algorithm (NSSL) provides 10% A pre-processing algorithm (NSSL) provides 10%
improvement – no changes in BZIP requiredimprovement – no changes in BZIP required– 56K line overrun may not be an issue – used for initial 56K line overrun may not be an issue – used for initial
testing because of cost and securitytesting because of cost and security– New scan strategies and availability of cheaper, faster New scan strategies and availability of cheaper, faster
links (even cable modems) may render 56K lines mootlinks (even cable modems) may render 56K lines moot
s fn
Technical HighlightsTechnical Highlights
Radars Now Delivering Real Radars Now Delivering Real Time Level II Data to NCDC/OUTime Level II Data to NCDC/OU
s fn
Cedar City
TulsaLas Vegas
Flagstaff
Tucson
Amarillo
Lubbock
Oklahoma CityFort Smith
Dallas/Fort Worth
WilmingtonYuma Phoenix
Seattle
Huntsville
Norfolk
Charleston
Morehead City
Columbia
Raleigh
Denver
2121
Courtesy Steve DelGreco (NCDC)
Old WSR-88D Level II Data Processing at Old WSR-88D Level II Data Processing at NCDCNCDC
WSR-88DWSR-88D
Abilene T3Abilene T3
Compressed Level II DataCompressed Level II DataFrom 21 RadarsFrom 21 Radars
LDM HDSS
MassMass Storage Storage
At NCDC TodayAt NCDC Today
Modified from Steve DelGreco (NCDC)
via FTPvia FTP
Processing/Processing/RecompressionRecompression
s fn
WSR-88D Level II Archival at WSR-88D Level II Archival at NCDCNCDC% Rcpts % Rcpts #Sites#Sites
Level II NWSLevel II NWS 64.964.9 120120Level II DoDLevel II DoD 35.735.7 2626Level II FAALevel II FAA 9.59.5 1212
Total 88D NetworkTotal 88D Network 55.755.7158158
s fn
WSR-88D Level II Archival at WSR-88D Level II Archival at NCDCNCDC
% Rcpts % Rcpts #Sites#Sites
Level II NWSLevel II NWS 64.964.9 120120Level II DoDLevel II DoD 35.735.7 2626Level II FAALevel II FAA 9.59.5 1212
Total 88D NetworkTotal 88D Network 55.755.7158158
CRAFT EFFICIENCY IS ABOVE 95%CRAFT EFFICIENCY IS ABOVE 95%
University of Iowa requested 19.4 gigabytes University of Iowa requested 19.4 gigabytes (uncompressed) of Level II data(uncompressed) of Level II data
The NCDC was able to retrieve it from the The NCDC was able to retrieve it from the HDSS, compress it, and stage it to FTP in HDSS, compress it, and stage it to FTP in 2 2 hours and 5 minuteshours and 5 minutes
Data was then moved over the Internet in a Data was then moved over the Internet in a matter of minutesmatter of minutes
The old 8 mm tape system would have The old 8 mm tape system would have required required – 58 tapes58 tapes– 9 workstations each running for 26 hours9 workstations each running for 26 hours– A total of A total of 232 hours232 hours
Improvement is more than a factor of 100 in Improvement is more than a factor of 100 in time, not including personnel coststime, not including personnel costs
s fn
A Notable NCDC Success A Notable NCDC Success StoryStory
s fn
0
20
40
60
80
100
120
0 5 10 15 20 25 30
Radar Data LatencySEAAMACMBFLGFTWSRPTULLBBFSMOKCTCS
Hour
PreliminaryPreliminary Latency DataLatency Data
ESDIM is fundingESDIM is funding– Analysis of latency, bandwidth, etcAnalysis of latency, bandwidth, etc– Network simulation (reliability, Network simulation (reliability,
redundancy)redundancy) Additional analysis being conducted in Additional analysis being conducted in
collaboration with the private sectorcollaboration with the private sector
s fn
Networking R&DNetworking R&D
Additions by End of CY 2001Additions by End of CY 2001
s fn
Cedar City
TulsaLas Vegas
Flagstaff
Tucson
Amarillo
Lubbock
Oklahoma CityFort Smith
Dallas/Fort Worth
WilmingtonYuma Phoenix
Seattle
Huntsville
Norfolk
Charleston
Morehead City
Columbia
Raleigh
Denver
Rapid City
Melbourne
Buffalo
Chicago
Pittsburgh
Cleveland
Cincinnati
Detroit
N. Indiana
21+ 21+ 1111
Cheyenne
Goodland
s fn
Cedar City
TulsaLas Vegas
Flagstaff
Tucson
Amarillo
Lubbock
Oklahoma CityFort Smith
Dallas/Fort Worth
WilmingtonYuma Phoenix
Seattle
Huntsville
Norfolk
Charleston
Morehead City
Columbia
Raleigh
Denver
Rapid City
Melbourne
Buffalo
Chicago
Pittsburgh
Cleveland
Cincinnati
Detroit
N. IndianaCheyenne
Goodland
Dodge CityWichita Springfield
Little Rock
Shreveport
Kansas City
Additions by Spring, 2002Additions by Spring, 2002
21+ 21+ 11 + 11 + 66
IHOP Field ProgramIHOP Field ProgramPlus CAPS FAAPlus CAPS FAAResearchResearch
s fn
Cedar City
TulsaLas Vegas
Flagstaff
Tucson
Amarillo
Lubbock
Oklahoma CityFort Smith
Dallas/Fort Worth
WilmingtonYuma Phoenix
Seattle
Huntsville
Norfolk
Charleston
Morehead City
Columbia
Raleigh
Denver
Rapid City
Melbourne
Buffalo
ChicagoPittsburgh
Cleveland
Cincinnati
Detroit
N. Indiana
Dodge CityWichita Springfield
Little Rock
Shreveport
Kansas City
Cheyenne
Goodland
Enid
Altus
Military RadarsMilitary Radars
21+21+11+11+66+2+2
s fn
Cedar City
TulsaLas Vegas
Flagstaff
Tucson
Amarillo
Lubbock
Oklahoma CityFort Smith
Dallas/Fort Worth
WilmingtonYuma Phoenix
Seattle
Huntsville
Norfolk
Charleston
Morehead City
Columbia
Raleigh
Denver
Rapid City
Melbourne
Buffalo
ChicagoPittsburgh
Cleveland
Cincinnati
Detroit
N. Indiana
Dodge CityWichita Springfield
Little Rock
Shreveport
Kansas City
Cheyenne
Goodland
Enid
Altus
FAA-MIT/LL Additions FY 2002FAA-MIT/LL Additions FY 2002
Central Ill
Milwaukee
Quad Cities
St. Louis
Paducah
Louisville
Charleston
Baltimore
Central PAPhiladelphia
New York City
AlbanyBoston
Binghamton
4040+14=+14=5454
s fn
Current Technical StrategyCurrent Technical Strategy
Repeater Hub
RIDDS Linux PC + LDM + Compression
($1000)
Server
Dedicated 56K line or ($2000 - $6000/year)
or NWS LANOptional Router
($2000)
WSR-88D
At the radar site
Users
Internet or Abilene
s fn
Interim BDDSInterim BDDS
Repeater Hub
BDDS Linux PC + LDM + Compression
($1000)
Server
Dedicated 56K lines or or NWS LAN
Optional Router ($2000)
WSR-88D
At the radar site
Users
Internet or Abilene
s fn
Final BDDSFinal BDDS
Repeater Hub
BDDS+LDM/Compression
ServerDedicated 56K lines or or
NWS LAN
WSR-88D
At the radar site
Users
Internet or Abilene
How Does This How Does This Experiment Experiment Become an Become an Operational Operational
System?System?
s fn
Abilene NetworkAbilene NetworkJanuary 1999 January 1999
Cleve
land
New York
Atlanta
Indianapolis
Kansas City
Houston
Denver
Los Angeles
Sacramento
Seattle
Abilene Router Node
Abilene Access Node
Operational January 1999
Planned 1999
One Possible StrategyOne Possible Strategy
s fn
NOAA Facility
via phone linesor commodity
Internet
Silver SpringSilver Spring
s fn
via phone linesor commodity
Internet
Abilene NetworkNOAA Facility
NOAA Facility NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility
NormanNorman
BoulderBoulder
Silver SpringSilver Spring SeattleSeattle
MiamiMiami
AtlantaAtlanta
One Possible StrategyOne Possible Strategy
s fn
via phone linesor commodity
Internet
Abilene NetworkNOAA Facility
NOAA Facility NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility
NormanNorman
BoulderBoulder
Silver SpringSilver Spring SeattleSeattle
MiamiMiami
AtlantaAtlanta
““Super Sites”: Each LDM on Abilene Super Sites”: Each LDM on Abilene ““bus” Carries all 88D data -- redundancybus” Carries all 88D data -- redundancy
One Possible StrategyOne Possible Strategy
s fn
Abilene NetworkNOAA Facility
NOAA Facility NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility
s fn
Abilene Network
Commodity InternetCommodity Internet
Commodity InternetCommodity InternetCommodity InternetCommodity Internet
Commodity InternetCommodity Internet
Com
mod
ity
Inte
rnet
Com
mod
ity
Inte
rnetC
omm
odity Internet
Com
modity In
ternet
NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility
One Possible StrategyOne Possible Strategy
s fn
Abilene NetworkNOAA Facility
NOAA Facility NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility
Private Sector
Academia
s fn
Abilene NetworkNOAA Facility
NOAA Facility NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
s fn
Abilene NetworkNOAA Facility
NOAA Facility NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
s fn
Abilene NetworkNOAA Facility
NOAA Facility NOAA Facility
NOAA Facility NOAA Facility
NOAA Facility
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Private Sector
Academia
Meeting all of the NeedsMeeting all of the Needs
88D 88D 88D 88D 88D
Abilene-based network of (aggregating & relaying) LDMs, each carrying all 88D data, with redundant interconnections
Private 3
Academia 2Private 3
Academia 1
Private 4
Private 1
NOAA 3
NOAA 2
NOAA 1
Courtesy D. Fulker, UCAR
Meeting all of the NeedsMeeting all of the Needs
88D 88D 88D 88D 88D
Abilene-based network of (aggregating & relaying) LDMs, each carrying all 88D data, with redundant interconnections
Private 3
Academia 2Private 3
Academia 1
Private 4
Private 1
NOAA 3
NOAA 2
NOAA 1
Courtesy D. Fulker, UCAR
– NOAA runs its operational ingest system but links to NOAA runs its operational ingest system but links to private sector and academic systems, each providing private sector and academic systems, each providing redundancy for the otherredundancy for the other
– Entire community benefits – builds upon the Entire community benefits – builds upon the collaborative spirit of CRAFTcollaborative spirit of CRAFT
– Configuration completely scalableConfiguration completely scalable– Leverages existing infrastructure (LDM sites, Abilene, Leverages existing infrastructure (LDM sites, Abilene,
WFO connection to commodity Internet)WFO connection to commodity Internet)– Significant capacity for future growth (dual-pol, Significant capacity for future growth (dual-pol,
phased array)phased array)
Benefits of This ConceptBenefits of This Concept
s fn
s fn
Weber (2000)
Longer Term: Addition of FAA Longer Term: Addition of FAA RadarsRadars