Cyberinfrastructure to Support Real-time, End-to-End Local

Forecasting

Mohan Ramamurthy

Tom Baltzer, Doug Lindholm, and Ben Domenico

Unidata/UCAR

AGU Fall Meeting

December 16, 2004

Local NWP: A Growing Activity• Mesoscale forecast models

are being run by universities, in real time, at dozens of sites around the country, often in collaboration with local NWS offices– Tremendous value– Leading to the notion of “distributed” NWP

• Yet only a few (OU and U of Wash) are actually assimilating local observations – which is one of the fundamental reasons forsuch models!

•Applied Modeling Inc. (Vietnam) MM5 •Atmospheric and Environmental Research MM5 •Colorado State University RAMS •Florida Division of Forestry MM5 •Geophysical Institute of Peru MM5 •Hong Kong University of Science and Technology MM5 •IMTA/SMN, Mexico MM5 •India's NCMRWF MM5 •Iowa State University MM5 •Jackson State University MM5 •Korea Meteorological Administration MM5 •Maui High Performance Computing Center MM5 •MESO, Inc. MM5 •Mexico / CCA-UNAM MM5 •NASA/MSFC Global Hydrology and Climate Center, Huntsville, AL MM5 •National Observatory of AthensMM5 •Naval Postgraduate School MM5 •Naval Research Laboratory COAMPS •National Taiwan Normal University MM5 •NOAA Air Resources Laboratory RAMS •NOAA Forecast Systems Laboratory LAPS, MM5, RAMS •NCAR/MMM MM5 •North Carolina State University MASS •Environmental Modeling Center of MCNC MM5 MM5 •NSSL MM5 •NWS-BGM MM5 •NWS-BUF (COMET) MM5 •NWS-CTP (Penn State) MM5 •NWS-LBB RAMS •Ohio State University MM5 •Penn State University MM5 •Penn State University MM5 Tropical Prediction System •RED IBERICA MM5 (Consortium of Iberic modelers) MM5 (click on Aplicaciones) •Saint Louis University MASS •State University of New York - Stony Brook MM5 •Taiwan Civil Aeronautics AdministrationMM5 •Texas A\&M UniversityMM5 •Technical University of MadridMM5 •United States Air Force, Air Force Weather Agency MM5 •University of L'Aquila MM5 •University of Alaska MM5 •University of Arizona / NWS-TUS MM5 •University of British Columbia UW-NMS/MC2 •University of California, Santa Barbara MM5 •Universidad de Chile, Department of Geophysics MM5 •University of Hawaii MM5 •University of Hawaii RSM •University of Hawaii MM5 •University of Illinois MM5, workstation Eta, RSM, and WRF •University of Maryland MM5 •University of Northern Iowa Eta •University of Oklahoma/CAPS ARPS •University of Utah MM5 •University of Washington MM5 36km, 12km, 4km •University of Wisconsin-Madison UW-NMS •University of Wisconsin-Madison MM5 •University of Wisconsin-Milwaukee MM5

Science Drivers for Local Modeling• Many weather phenomena that affect society and

commerce occur on the mesoscale. E.g., squall-lines, snowbands, hurricanes; downslope windstorms, lake-effect snowfall, etc.

• Need high-resolution local modelling to accurately resolve and predict these phenomena;

• Utilize dense local observations (e.g., Mesonets);• Resolve local topography• Collaboration with local NWS forecast offices;

Show examplesShow examples

Technology Trends Enabling A New Generation of

Local NWP Activities

Commodity microprocessors & inexpensive but powerful workstations/clusters

High-bandwidth networks (e.g., Internet 2)

Transparent data access and delivery

Community Models (MM5, WRF)Local observatories (e.g., mesonets)Community codes for data

assimilation (e.g., 3DVAR, ADAS)

Analysis/Assimilation

Quality ControlRetrieval of Unobserved Quantities

Creation of Gridded Fields

Prediction

PCs to Teraflop Systems

Product Generation,

Visualization, Dissemination

End Users

NWSPrivate Companies

Students

Numerical Weather Prediction: Key Steps

Observations & Previous Model Forecast

Mobile MesonetsSurface Observations

Upper-Air BalloonsCommercial Aircraft

Geostationary and Polar Orbiting SatelliteRadar Data

Wind ProfilersGPS/Met instruments

Unidata Technologies [That can be Used] in Local Modeling

Local Data Manager – data transportData streams: IDD and CONDUIT – Relaying

and accessing dataDecoders – Data conversionNetCDF libraries and tools – Data infrastructureOPeNDAP – Remote data access (Collaborator)THREDDS – Cataloging dataGEMPAK and IDV - VisualizationGIS Integration tools (in future)

Real-time Data Distribution

Source

LDM

Source

Source

LDM LDM

LDMLDM

LDM LDM

LDM

LDM

Internet

Radar

Model

Satellite

There are over 150 university sites in North and South America, Europe, and Asia that receive real-time data using the Unidata Local Data Manager; Plus there are over 300+ LDM sites in NWS, NOAA, NASA, KMA, Taiwan, and Spain that are not part of the “open” IDD.

LDM in Action

LDM is providing a variety of real-time meteorological observations and model output from operational prediction systems for local NWP initialization

WSR 88-D DataWSR 88-D Data SuomiNet

MeteorologicaMeteorologicall

AssimilationAssimilationSystemSystem

User running local User running local analysis and display toolsanalysis and display tools

Regional Model Regional Model Hosted on local Hosted on local

hardwarehardware

Decoders

National National Forecast Forecast

Model OutputModel Output

Today’s Local NWP Process at Many Universities

Assimilated DataFor Initial ConditionsReal-time Real-time

Weather DataWeather Data

DecodersDecoders

MeteorologicaMeteorologicall

AssimilationAssimilationSystemSystem

User running local User running local analysis and display toolsanalysis and display tools

Regional Model Regional Model Hosted on local Hosted on local

hardwarehardware

Decoders

National National Forecast Forecast

Model OutputModel Output

Today’s Local NWP Process at Many Universities

Assimilated DataFor Initial ConditionsReal-time Real-time

Weather DataWeather Data

DecodersDecoders

There is no Data Sharing (other than with local NWS offices)

OPeNDAP Servers

Unidata Motherlode Server Unidata LEAD Testbed

There are many OPeNDAP servers for operational and historical data, but none outside of Unidata & LEAD for real-time local NWP output

Remote Data Access and Catalogs

Developed for real-time WRF predictions from University of Illinois.

Courtesy: Brian Jewett

Integrated Data Viewer

• Unidata’s newest scientific analysis and visualization tool

• Provides 2, 3 and 4-D displays of geoscientific data

• Stand-alone or networked application, providing client-server data access via multiple protocols

• Java-based tools: Runs on Windows, Macs and Unix machines

Remote Visualization of Local NWP Output

Developed for real-time WRF predictions from University of Illinois.

Courtesy: Brian Jewett

GEMPAK Example

• Some sites convert their forecast output into a format compatible with GEMPAK analysis and visualization tool

• Enables integration of local model output with other operational data sets

User applications: e.g., McIDAS, IDV,

LAS, IDL, MatLab...

DLESEDigital Library for

Earth-System Education

HydrologyData, e.g.

GeophysicalData, e.g.

Satellite Images, e.g.Satellite

Images, e.g.Satellite

Images, e.g.Satellite

Imagery...

OpenDAP, ADDE, &

FTP protocols

IDD

DLinterchange

protocol

IDD

Discovery

IDD IDD IDD

User applications: e.g., McIDAS, IDV,

LAS, IDL, MatLab...

DLESEDigital Library for

Earth-System Education

HydrologyData, e.g.

GeophysicalData, e.g.

Satellite Images, e.g.Satellite

Images, e.g.Satellite

Images, e.g.Satellite

Imagery...

OpenDAP, ADDE, &

FTP protocols

IDD

DLinterchange

protocol

IDD

Discovery

IDD IDD IDD

PeoplePeople

DocumentsDocuments DataData

Catalog

Generation Tools

Analysis andVisualization Tools

Data Services

Discovery andPublication Tools

Discovery and Publication Services

Dat

a C

atal

ogS

ervi

ces

PeoplePeople

DocumentsDocuments DataData

Catalog

Generation Tools

Analysis andVisualization Tools

Data Services

Discovery andPublication Tools

Discovery and Publication Services

Dat

a C

atal

ogS

ervi

ces

Thematic Real-time Environmental Distributed Data Services (THREDDS)

Combines “push” with several forms of “pull” and digital library discovery

To make it possible to publish, locate, analyze, visualize, and integrate a variety of environmental data

Connecting People with Documents and Data

THREDDSMiddleware

LEAD: A Large-ITR Effort• Linked Environments for

Atmospheric Discovery– Identify, Access,

Assimilate, Predict, Manage, Mine, and Visualize a broad array of meteorological data and model output, independent of format and physical location

– A range of Grid and Web Services will be developed for dynamic, on-demand, end-to-end weather prediction

– Institutions: U. Oklahoma, Unidata, U. Alabama, U. Illinois, U. Indiana, Millersville U., Howard U. and Colorado State U.

Web Services• They are self-contained, self-describing, modular

applications that can be published, located, and invoked across the Web.

• Web Services are emerging as tools for creating next generation distributed systems that are expected to facilitate program-to-program interaction without the user-to-program interaction.

• Besides recognizing the heterogeneity as a fundamental ingredient, these web services, independent of platform and environment, can be packaged and published and they can communicate with other systems using the common protocols.

User running local User running local analysis and display toolsanalysis and display tools

Data ServiceData Service

Decoder Decoder ServiceService

Assimilation Assimilation ServiceService Regional Regional

Model Model ServiceService

User Orchestrates User Orchestrates Web Services to Web Services to Create Regional Create Regional

ForecastForecastProduct Product

Generation & Generation & Data Mining Data Mining

ServiceService

LEAD Vision