Computing and Computational Science at NASA in Support of

Earth System Science

Dr. Jack A. Kaye*Associate Director for Research

Earth Science DivisionNASA Science Mission Directorate

8 September 2011* With assistance from Tsengdar Lee, Mike Little, Martha Maiden, Steve Berrick, David Considine

2

NASA’s Research and Analysis Foci

NASA missions and observations are generating tremendous amount of data.• NASA funds projects to process and analyze data• Performs analysis, extracts information, and

generate science resultsModels are essential tools used in this process.

NASA focuses on:• Observation-Driven Modeling

– Data assimilation for prediction problems– Model-data intercomparisons– Observation System Simulation Experiments

• Model-Synthesized Observations– Observational data reanalysis

3

High-End Computing (HEC) Portfolio

Mission Objective: Develop and deliver productive high-end computing systems and services to support NASA’s mission needs; operate and manage these resources for the benefit of Agency users, customers, and stakeholders.

Customers: Any NASA funded R&A projects are entitled to request for HEC resource as described in ROSES summary of solicitation

Key Science and Engineering Products:– Model simulation output & analysis products– Data reanalysis products

System Components: Compute: NASA continues to build and

operate two of the world’s largestsupercomputers

Storage: large online and tape storage for model dataNetworks: High speed network peered with academia networks Services: Supercomputing runtime environment, data analysis & visualization, application development & performance optimization, code porting & tuning, large-scale data storage & sharing, local & wide area networking, 24x7 help desk & user training

Discover Supercomputer

GSFC/NCCS

Pleiades, Columbia Supercomputershyperwall2ARC/HECC

Two Computing centers: - High-End Computing Capability (HECC) - NASA Center for Climate Simulations (NCCS)

4

High-Resolution Climate Simulations with Goddard Earth Observing System, Version 5 (GEOS-5) Cubed-Sphere Model

GEOS-5 scales to 13,824 cores Running with GOCART interactive chemistry at 3.5- to 10-km globally for the first time

Tuning physics to better resolve tropical deep convection and hurricane structure5-day forecasts run at 10-km globally within a 3-hour operational window

3.5-km global simulations complete 3-day forecasts in 24-hours

3.5-kmGEOS-5

2009 Atlantic Hurricane Bill

Bill Putman, Max Suarez, NASA Goddard Space Flight Center; Shian-Jiann Lin, NOAA Geophysical Fluid Dynamics Laboratory

5

Tools for Getting the Data You Need

Visualize and Analyze Data Before DownloadingOnline tools, such as Giovanni, offer you the ability to visualize, analyze, and compare data from multiple sensors on multiple satellites without having to download the data first. After the analysis, you can select and download only the pertinent data. Giovanni provides portals tailored to multiple science discipline communities.

NASA's Earth Science Data Systems Program is comprised of core and community elements to serve the needs of a broad and diverse community. The Advancing Collaborative Connections in Earth System Science (ACCESS) Program focuses on community elements by leveraging technologies to improve the interconnectedness between data and users. Successful community elements can be infused into the core creating a vibrant, flexible, and continuously evolving infrastructure (science.nasa.gov/earth-science/earth-science-data/).

Finding the Right DataReverb allows you to explore and search thousands of data products across NASA’s distributed Earth science data centers using advanced search criteria that focus on multiple data aspects (reverb.echo.nasa.gov).

Other search interfaces are tailored to your discipline-specific research community and give you additional advanced capabilities optimized for your community’s tools and practices.

A-Train Data Depot uses Giovanni to visualize multiple parameters from multiple satellites

online (disc.gsfc.nasa.gov/atdd).

Accessing Data Your WayAdvanced science-supported data services including quality screening, subsetting, reprojection, and reformatting reduce data preparation time.

Standard Web services such as OPeNDAP, Google Earth, OGC Web Map Service, and Web Coverage Service allow data to be used immediately with your tools.

Get Notified About New DataData casting and service casting are new, lightweight notification technologies that allow you to be notified of data or services meeting your specific criteria for quality or other characteristics.

6

Key elements:

Monitoring

Modeling

Forecasting

Local to Global

NASA Earth Exchange Data - Modeling Platform

Funded by NASA’s Earth Science Research & Analysis, Applied Sciences, and Technology Programs

7

Challenges

1. Software Development Practice• Earth system scientists are not trained in software engineering.

This results in poorly organized and constructed model codes.• Computational scientists are not trained in Earth system

sciences. This results in poorly executed codes and significant model errors.

2. Validation and Verification• Model V&V done manually results in insufficient test and

evaluation.• Models and observations are producing tremendous amount of

data at different time and spatial scale. This result in significant challenge comparing the two.

3. Tools and Environment• Productivity significantly reduced due to the inadequate tools

and development environment

8

Education and Workforce Development

We need educated people who can bridge the science, computing, and computational disciplines.1. We are thinking of short-term intensive education programs to

train Earth and Space Scientists in parallel computing and software engineering.

2. Computing and modeling centers are looking for interns to work on specific NASA problems – see http://intern.nasa.gov/

3. We hope the academia will share in developing approaches to help develop the cross-disciplinary workforce needed for the future.

9

Research Opportunities at NASA

Research Opportunities in Space and Earth Sciences (ROSES) solicitation the CASC community may consider:1. Data systems, data management, access, and data processing

• Making Earth System data records for Use in Research Environments (MEaSUREs)

• Advancing Collaborative Connections for Earth System Science (ACCESS)

2. Advanced Information System Technology3. Modeling and Data Assimilation Research:

• Earth science Modeling and Assimilation Program• Atmospheric Composition: Modeling and Analysis• Heliophysics, Astrophysics Theory Programs

4. Computational Modeling Algorithms and Cyberinfrastructure Program

5. Supplemental Education Awards for ROSES InvestigatorsROSES Solicitation Web site:

http://nspires.nasaprs.com/external/solicitations/solicitations.do?method=open&stack=push

10

Backup

11

NCCS’s NVIDIA GPGPUs Supporting Climate Science

GEOS-5 column physics (Matt Thompson)• Porting of radiation code, the largest of

GEOS-5 column physics codes, is well underway.

• Porting of cloud microphysics and the turbulence code will begin shortly.

GEOS-5 dynamics code (Bill Putman)• An 85x speedup was measured on ~1/3

of the dynamics kernel of the fvCubed Sphere.

Discover NVIDIA GPGPU system (36.4 TFLOPs Peak):

• 32 general-purpose 12-core Westmere nodes, each with 48 GB memory.

• 64 “Tesla” M2070 GPGPUs, each with 488 CUDA cores and 6 GB memory (28,672 Total GPGPU Streaming Cores).

Integrated into Discover software stack and file systems for quick access and code porting

• PGI CUDA FORTRAN compiler is available for porting codes GPGPUs.

For more information:• http://www.nccs.nasa.gov/gpu_front.html• http://sciences.gsfc.nasa.gov/600/highlights/

stories/gpus.html

Diagram of the JCSDA-developed Community Radiative Transfer Model (CRTM) used by GEOS-5.2 Data Assimilation System.

GMAO researchers continued porting portions of the GEOS-5 General Circulation Model for use with General Purpose Graphical Processing Units (GPGPUs), considered well-suited to “column physics,” as NCCS staff integrated the 64 NVIDIA GPGPUs, attached to 32 Westmere computational nodes, delivered in March 2011.

NVIDIA M2070 “Tesla” GPGPU

12

Understanding Global Landscapes at High Resolution

For monitoring crop growth, deforestation,

and impacts of natural disasters

For the first time in Landsat history of nearly 30 years, we can now process and create quantitative information about

changes in global landscapes, in a matter of hours.

Near realtime quantitative updates about global landscapes

Building virtual teams

13

High-End Computing Capability

HECC Project is located within the NASA Advanced Supercomputing (NAS) Division at Ames, and managed under the Agency’s Strategic Capabilities Assets Program (SCAP). It supports the modeling, simulation, analysis and decision support activities for all four NASA Mission Directorates and NESC.

• Balanced Environment– 3 PB disk filesystem; 30 PB tape

archive– Archiving 500TB – 1PB every month– High-bandwidth WAN to other

Centers, external peering– Large-scale and concurrent

visualization• Resources enable broad mission

impact– More than 400 science &

engineering projects– More than 1,200 user accounts– Typically 400-500 jobs running 24x7– Demand for cycles extremely high

• Computing Systems– Pleiades: 101,120-core SGI Altix ICE

(Xeon quad), 1.3 PFlops– Columbia: 4608-processor SGI Altix

(Itanium2)– hyperwall2: 1,024-processor, 128-

node GPU cluster– Multiple secure front ends, metadata

servers, object storage servers

NAS support staff help users productively utilizeNASA’s supercomputing environment to rapidlysolve large computational problems

NAS software expertsutilize tools to parallelize& optimize codes, dramaticallyincreasing simulation performancewhile decreasing turn-around time

Scientists, engineers plancomputational analyses, selectingbest codes to address NASA’s complex mission challenges

Outcome: Dramatically enhanced understanding & insight, accelerated science & engineering, and increased mission safety & performance

NAS viz experts applyadvanced data analysis & rendering techniquesto help users explore& understand large,complex computationalresults

1414

14

NASA Center for Computational Sciences (NCCS) supports SMD modeling and analysis activities:

Atmospheric modeling – for climate and weather researchOcean modeling – for climate, chemistry, and biologyLand surface modeling – for agriculture, land use, and water resource managementSpace & solar modeling – for fundamental astrophysics, space weather, and gravitational wave studiesCoupled models and systems of models – in support of collaborative science effortsObserving system studies – to enhance the use and design of space instruments

NASA Center for Climate Simulations (NCCS)

15

NCCS – Computational Science Projects

August 1, 2010 to July 31, 2011

Astrophysics (14 projects)• Researchers performed for the first time

axisymmetric and global 3D simulations of the magnetorotational instability-driven accretion onto magnetized stars.• The most detailed, physically motivated model X-ray

luminosity functions for X-ray binaries in nearby and low-to-high-redshift galaxies are starting to match observations.

• Efforts to simulate formation of planetary systems include the first theoretical models to produce planets with masses ranging from the mass of Earth to 10 times the mass of Jupiter.

Heliophysics (5 projects)

Planetary Science (10 projects)

• 3D, multifluid MHD studies of the interaction of the solar wind with Mars described new physical processes such as density plumes and asymmetries due to ion decoupling.

• Simulations are being used to assess the effects of the active solar wind on Mercury’s magnetosphere during the MESSENGER spacecraft’s orbit.

Earth Science (95 projects)• The GEOS-5 model completed a 10-

kilometer “nature run” covering 2 years and can finish 3-day, 3.5-kilometer global weather forecasts within 24 hours.

• For the next IPCC assessment, coupled atmosphere-ocean models performed historical simulations reaching back to the year 1000.

• NCCS enables real-time forecasts and provides data services for NASA field campaigns including DISCOVER-AQ (air quality) and HS3 (hurricanes).

• New analyses show the benefits of greater adoption of American and European vehicle emission standards and the potential effects of sea level rise on New York City.

• MHD simulations of the solar wind are increasing understanding of Voyager observations in the outer heliosphere and the causes of space weather.

16

NCCS Encourages Large Scale Computational Science Research with 1000+ Cores

• The NCCS advanced Discover SCU7’s use for SMD’s large-scale science investigators by extending access to researchers who intend to scale codes to 1000 or more cores.

• Researchers’ requests for large-scale research runs have included the following applications:• GEOS-5 fvCubed Sphere• NASA Unified Weather Research

and Forecasting model (NU-WRF)• Enzo adaptive-mesh refinement

code for cosmological simulations

3.5-kmGEOS-5

fvCubed Sphere

2009 Atlantic Hurricane Bill

Primordial star simulation using Enzo

Simulated radar reflectivity 2004 Atlantic

Hurricane Charley 2-km NU-WRF

17

Introducing NASA Earth Exchange Collaborative Earth Science (http://nex.arc.nasa.gov)

To encourage exploration and collaboration in the Earth science community to create new ways to understand and improve life here on Earth.

To encourage exploration and collaboration in the Earth science community to create new ways to understand and improve life here on Earth.