Grid resources for NWP models at Grid resources for NWP models at national level in Korea national level in Korea
Korean Meteorological Administration
2004. 1. 292004. 1. 29
Super Computer CenterSuper Computer Center
Korea Meteorological Administration Korea Meteorological Administration
APANAPAN
ContentsContents
What the operational weather forecasting isWhat the operational weather forecasting is
Why the operational centers need Grid Technology.Why the operational centers need Grid Technology.
What Grid activities are in the Meteorological field.What Grid activities are in the Meteorological field.
What we have and will do with IT infrastructure.What we have and will do with IT infrastructure.
Korean Meteorological Administration
APANAPAN
What the operational weather What the operational weather forecasting is…forecasting is…
Korean Meteorological Administration
APANAPAN
observationobservation
Technology for the weather Technology for the weather forecastingforecasting
collectioncollection Diss.Diss.Anal.& WFAnal.& WF
NumericalWeatherPrediction
Infrastructure for Earth SystemInfrastructure for Earth System
Korean Meteorological Administration
APANAPAN
Nature
Korean Meteorological
Administration
DATAPRODUCERS
GlobalObserving
System
GOS
DATACONVEYORS
GlobalTelecommunication
System
GTS
DATA USERS
Global DataProcessing
System
GDPS
World Meteorological Organization (187members)World Meteorological Organization (187members)
0
500
1000
1500
2000
2500
3000
3500
4000
GB
/Day
NWP Messages NEXRAD Prod NEXRAD Base
APANAPAN
Current GTS will be replaced with FWIS until 2010Current GTS will be replaced with FWIS until 2010
Korean Meteorological Administration
APANAPAN
연구배경
• Global Scale Study => climatology with Atmos., Ocean, land surface, etc Global Scale Study => climatology with Atmos., Ocean, land surface, etc
• Global Observation and sharing informationGlobal Observation and sharing information• Co-work with international society is importantCo-work with international society is important
Some Characteristics of Earth ScienceSome Characteristics of Earth Science
global science with huge observation data and high performance computing
• Dynamic Equation SetDynamic Equation Set • Numerical RepresentationNumerical Representation
• Super ComputingSuper Computing
Korean Meteorological Administration
APANAPAN
Why we need GridWhy we need Grid
Korean Meteorological Administration
APANAPAN
Archived data volume and the Moore law Archived data volume and the Moore law
19
91
19
92
19
93
19
94
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
Te
rao
cte
ts
volume loi de Moore
Source: CAS2k3 , MetroFranceSource: CAS2k3 , MetroFrance
Korean Meteorological Administration
APANAPAN
Source:CAS2K3 DKRZSource:CAS2K3 DKRZ
Daily transfer volumeDaily transfer volume
Korean Meteorological Administration
APANAPAN
0
200
400
600
800
1000
[TB]
Sep
01
Dez 0
1
Mrz
02
Ju
n 0
2
Sep
02
Dez 0
2
Mrz
03
Ju
n 0
3
archive capacity
original
duplicates
0
200
400
600
800
1000
[TB]
Sep
01
Dez 0
1
Mrz
02
Ju
n 0
2
Sep
02
Dez 0
2
Mrz
03
Ju
n 0
3
archive capacity
original
duplicates
0
500
1000
1500
2000
2500
3000
[GB]
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
daily transfer volume
fetch
store
0
500
1000
1500
2000
2500
3000
[GB]
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
daily transfer volume
fetch
store
Korean Meteorological Administration
APANAPAN
Source:CAS2K3 DKRZSource:CAS2K3 DKRZ
Data Size in Operational Center ( DWD )Data Size in Operational Center ( DWD )
Korean Meteorological Administration
APANAPAN
Forget Modeling: How About Case Forget Modeling: How About Case Study Research?Study Research?
Exhaustive study of one or more selected observed weather events, Exhaustive study of one or more selected observed weather events,
sometimes involving the use of models and always involving lots of data!sometimes involving the use of models and always involving lots of data!
We spend We spend 70%70% of our time dealing with of our time dealing with data logisticsdata logistics, i.e., searching, , i.e., searching,
formatting, acquiring, reading, reducing, merging onto common grids and formatting, acquiring, reading, reducing, merging onto common grids and
projections, visualizing, etc…projections, visualizing, etc…
- Radar, satellite, surface nets, GPS, upper-air balloon, commercial - Radar, satellite, surface nets, GPS, upper-air balloon, commercial
aircraft, wind profiler, etc…aircraft, wind profiler, etc…
And only And only ~30%~30% of our time is spent actually studying the event! of our time is spent actually studying the event!
Source: LEAD, Prof. KelvinSource: LEAD, Prof. Kelvin
Korean Meteorological Administration
APANAPAN
WGNE Table of Operational NWP Centres WGNE Table of Operational NWP Centres (November 2002; ref. Kamal Puri)(November 2002; ref. Kamal Puri)
Source : CAS2k3 M.Naughton BOMSource : CAS2k3 M.Naughton BOM
Korean Meteorological Administration
APANAPAN
What Grid activities are in the What Grid activities are in the Meteorological fieldMeteorological field
Korean Meteorological Administration
APANAPAN
The Grid and Meteorology:The Grid and Meteorology:OpportunitiesOpportunities
Inter-personal collaborationInter-personal collaboration
- E.g., Access Grid, CHEF- E.g., Access Grid, CHEF
On-demand access to simulation modelsOn-demand access to simulation models
- E.g., Espresso- E.g., Espresso
Access to, and integration of, data sourcesAccess to, and integration of, data sources
- E.g., Earth System Grid- E.g., Earth System Grid
Dynamic, virtual computing resourcesDynamic, virtual computing resources
- “Metacomputing”- “Metacomputing”
Integration of all of the aboveIntegration of all of the above
- Collaborative, computationally intensive analysis of large quantities of - Collaborative, computationally intensive analysis of large quantities of
online data online data Source : Ian Foster, 16Source : Ian Foster, 16thth APAN Meeting APAN Meeting
Korean Meteorological Administration
APANAPAN
European GridEuropean Grid
www.eurogrid.orgwww.eurogrid.orgKorean Meteorological
Administration
Computational grid exampleComputational grid exampleAPANAPAN
Korean Meteorological
Administration
Computational grid example(cont.)Computational grid example(cont.)APANAPAN
Korean Meteorological
Administration
APANAPAN
Earth System GridEarth System Grid2
Earth System Grid ANL, LBNL, LLNL, NCAR, ORNL, USC/ ISI
Why Climate Changeis a Grid Problem
Climate modeling Data sources: high data rates
– 2MB/s (100 GF/s)– Multiple, high-resolution, multi-component runs: 100
MB/s (@5 TF/s) => 3 PB/yr
User community: large and diverse– O(100+) laboratories, centers, universities– Highly multidisciplinary– International connections
Data analyses: sophisticated, intensive– Multiplicity of tools used for an analyses– Intercomparison of data in different formats
Korean Meteorological Administration
Data grid exampleData grid exampleAPANAPAN
http://lead.ou.eduhttp://lead.ou.edu
Korean Meteorological Administration
APANAPAN
Korean Meteorological
Administration
APANAPAN LEAD ConceptLEAD Concept
Korean Meteorological Administration
APANAPAN LEAD Data CloudLEAD Data Cloud
What we haveWhat we have
Korean Meteorological Administration
APANAPAN
High Speed Network - KORENHigh Speed Network - KOREN
Korean Meteorological Administration
APANAPAN
High Speed Network - KREONETHigh Speed Network - KREONET
Korean Meteorological Administration
APANAPAN
High Speed Network – International LinkHigh Speed Network – International Link
Korean Meteorological Administration
APANAPAN
Korean Meteorological
Administration
APANAPAN KMA Intra NetworkKMA Intra Network
Filer2
Filer1
BackupSystem
Front-EndSystem
GigabitSwitch2
GigabitSwitch1
SX-SX-5/16A5/16A
SX-5/12ASX-5/12A
XYLANF/W
KORENKOREN KMA KMA BackboneBackbone
Fast.EFast.E
MRIMRI
Giga.EGiga.E
G-NEtG-NEt
KMA HPC and NetworkKMA HPC and Network
KREONET
Korean Meteorological Administration
APANAPAN
What we will doWhat we will do
Korean Meteorological Administration
APANAPAN
NEXT KMA HPC SystemNEXT KMA HPC System54 times faster than the current system54 times faster than the current systemInitial stage : 6 times 4Q 2004Initial stage : 6 times 4Q 2004Final stage : 54 times 4Q 2005Final stage : 54 times 4Q 2005More than 80M$ for next 5 yearsMore than 80M$ for next 5 years
Plan for the National Data Center : from 2004Plan for the National Data Center : from 2004 To collect Obs. DataTo collect Obs. Data To disseminate Weather Information and ProductTo disseminate Weather Information and Product
Digital Weather ForecastDigital Weather Forecast Fine Mesh Weather Grid InformationFine Mesh Weather Grid Information Real time DisseminationReal time Dissemination
Korean Meteorological Administration
APANAPAN
KMA Met Satellite at 2008KMA Met Satellite at 2008
Representative of Group on Earth Observation Representative of Group on Earth Observation
V-GISC V-GISC
Future WMO Information SystemFuture WMO Information System
E-Science for Earth SystemE-Science for Earth System
APEC Climate Network APEC Climate Network
Korean Meteorological Administration
APANAPAN
Test for the collaboration in the field of Met. with High Speed Network Test for the collaboration in the field of Met. with High Speed Network
targettargetData Grid Data Grid
For sharing For sharing
Huge dataHuge data
Computational Computational
gridgrid
Access GridAccess Grid
VirtualVirtual
LabLab
Virtual Storage for Huge Virtual Storage for Huge Met. DataMet. Data
User Interface based on User Interface based on 3D, VR3D, VR
Parallel Development of modules Parallel Development of modules
Computational grid Computational grid
Re-locatable and adoptive Re-locatable and adoptive grid modelgrid model
Access grid for parallel Access grid for parallel development of model codesdevelopment of model codes
Korean Meteorological Administration
APANAPAN
연구 방법
Computational grid for Meteo. Application(Distributed Computing)Computational grid for Meteo. Application(Distributed Computing)
Using High Speed Network ( KOREN, KREONET )
Real Time Allocation of computing resource and data resource, sharing model codes
re-locatable grid with friendly UI for high resolution model모 듈 3
Model
모듈 4
Group A
Module 1
Group CModule 3
Group BModule 2
Group DModule 4
DataRes.
ComputingRes.
ComputingRes.
DataRes.
user
모듈 1 모 듈 2module2
Module 1
Module 3 Module 4
Korean Meteorological Administration
APANAPAN
연구 방법
System development for the sharing huge Met. dataSystem development for the sharing huge Met. data
Development of User Interface
virtual storage based on Grid Technology
technique development for Remote Display based on 3D,VR
Storage A
Storage B
Storage C
GridFTP
GridFTP
GridFTPVirtual Storage
Request
Manager
Data
Catalog
Manager
ReplicaCatalogManager
User
Display
Korean Meteorological Administration
APANAPAN
연구 배경
conferenceOff-Line
Media
Group A
Module1Module2Module3
…
meeting E-Mail Phone/Fax
Group B
Module1Module2Module3
…
Group C
Module1Module2Module3
…
Group D
Module1Module2Module3
…
One-Side
Media
Current research status Current research status
– – individual off-line individual off-line
Korean Meteorological Administration
APANAPAN
Display system 3D, VR
연구 방법
Access Grid for Collaboration
Access grid for parallel- development
Parallel computing based on high performance computing
Discussion Forum
Video Conference
Exchange E-Message
Co-operational Project
Local A
Local B
Local C
Access Grid
Computing Grid
Data Grid
Korean Meteorological Administration
APANAPAN
For the High quality weather For the High quality weather informationinformation
- More regional and global Observation Data - More regional and global Observation Data
- Data assimilation- Data assimilation
- High Performance Computing- High Performance Computing
- Ensemble Prediction for regional and global scale- Ensemble Prediction for regional and global scale
- Enhance the research activity- Enhance the research activity
- Training and Education- Training and Education
Korean Meteorological Administration
APANAPAN
Thank you!!Thank you!!
Korean Meteorological Administration
APANAPAN
Data Exchange withData Exchange with
National Information Center for Earthquakes and Disasters,National Information Center for Earthquakes and Disasters,
National Research Institute for Earth Science and Disaster Prevention National Research Institute for Earth Science and Disaster Prevention
Earthquake and ZunamiEarthquake and ZunamiEarthquake and ZunamiEarthquake and Zunami
Korean Meteorological Administration
APANAPAN