Computational Science and Infrastructure for Supercomputing in JapanKiyoyuki TerakuraHokkaido University2006/11/20Supercomputing Korea 2006 KIAS

Supercomputer projects in JapanTop500 List November 2006

9. Global Scientific Information and Computing Center, Tokyo Inst. Tech. TSUBAME Grid Cluster Sun Fire x4,600 cluster, Opteron 2.4/2.6 GHz and ClearSpeed Accelerator, Infiniband NEC/Sun # of proc.: 11,088, Year: 2006, Rmax: 47 Tflops, Rpeak: 82 Tflops main memory: 21 TB

14. The Earth Simulator Center Earth Simulator NEC # of proc.: 5,120, Year: 2002, Rmax: 36 Tflops, Rpeak: 41 Tflops main memory: 10 TB13 systems in TOP10030 systems in TOP500

Earth simulator http://www.es.jamstec.go.jp/esc/eng/AP: arithmetic vector processorPN: processor node

Earth simulator

Atmospheric general circulation modelAn atmospheric general circulation model called AFES (AGCM for Earth Simulator) was developed and optimized for the architecture of the Earth Simulator (ES). AFES is based on the CCSR/NIES AGCM and is a global three dimensional hydrostatic model using the spectral transform method. We achieved a high sustained performance by the execution of AFES with T1279L96 resolution on the ES. The performance of 26.58 Tflops was achieved the execution of the main time step loop using all 5120 processors (640 nodes) of the ES. This performance corresponds to 64.9% of the theoretical peak performance 40.96 Tflops. AFES's this performance was recognized as the fastest computation at Super Computing 2002, Baltimore, MD, U.S.A., November 2002, and AFES won Gordon Bell Prize for Peak Performance. Atmosphere and Ocean Simulation Research Group (AOSG), Earth Simulator Center, JAMSTEC

Atmospheric general circulation modelAtmosphere and Ocean Simulation Research Group (AOSG), Earth Simulator Center, JAMSTEC

Another example of large simulationSoftware: MSSG(Multi Scale Simulator for Geoenvironment) holistic simulation for coupled atmosphere-ocean model (software developed by Multiscale Simulation Research Group)

resolution 10 km x 10 km over the whole earth 2.78 km x 2.78 km in the region near Japan Time integration over 5 days

Computer resource 768CPU (96 Nodes) memory: about 1TB CPU time: about 7.5 hourshttp://www.es.jamstec.go.jp/esc/eng/GC/index.html Simulation of Typhoon ETAU in 2003

Supercomputers deployed inInformation Initiative Centers in 7 universitiesTohoku Univ.SX-7(2Tflops)Nagoya Univ.PrimePower(12.5Tflops)Univ. of TokyoSR11000/K2(19Tflops)Kyoto Univ.PrimePower(8.8Tflops)Osaka Univ.SX-8(5.3Tflops)Kyushu Univ.P5(3.25Tflops)Hokkaido Univ.SR11000/K1(5.4Tflops)Available to researchers in all universities and national institutes

Some other supercomputer centersInstitute for Materials Research, Tohoku Univ.High Energy Accelerator Research OrganizationCenter for Computational Sciences, Tsukuba Univ.National Institute for Materials ScienceInstitute for Solid State Physics, Tokyo Univ.Advance Center for Computing and Communication, RIKENGlobal Scientific Information and Computing Center, TITECEarth simulator centerJapan Aerospace Exploration AgencyNational Institute for Fusion ScienceInstitute for Molecular ScienceJapan Atomic Energy Agency

Next generation supercomputer vs. earth simulatorEarth simulator Start of the project: 1997 Start of operation: 2002 number of processors: 5,120 peak performance: 40 TflopsNext generation supercomputer (NGSC) Start of the project: 2006 Start of operation: 2011 number of processors: > 100, 000 peak performance: 10 Pflops total budget : about 1 billion USD

Target of next-generation supercomputerfrom RIKEN Next-Generation Supercomputer R&D Center

Structure of the NGSC projectCouncil for industrial applications of supercomputing (about 160 companies)Project Leader: T. Watanabe

Important aspects NGSC projectContinuity There was no connection between consecutive developments of supercomputers in Japan, though almost every 10 years new supercomputers have been developed. We have entered a new era in the sense that Japanese Government promised to develop world top level supercomputer consecutively.General purpose Earth simulator was developed for a particular research field, i.e., earth science. The next generation supercomputer is, however, for general purpose. It is expected that the technology development associated with the next generation supercomputer will be utilized for the development of lower rank supercomputers in several computer Centers.Contribution to industry It is expected that the next generation supercomputer will benefit the industries in their research and development.

Supercomputing as a national key technologySupercomputers are extremely expensive: In order to make such a big investment for developing supercomputers, importance of supercomputing has to be well recognized.Some by-products Computational sciences are getting more important. Education program will be introduced for training skilled people for software engineering for operating systems and applications. Council for Science and Technology Policy has set supercomputing as one of the national key technologies.

Grand challenge applications in NGSCTwo grand challenges in application have been set up.

Next Generation Integrated Nanoscience Simulation Software Institute for Molecular Science Institute for Solid State Physics, University of Tokyo Institute for Materials Research, Tohoku University Natl Institute of Advanced Industrial Science and Technology

Research and Development of Next-Generation Integrated Life Simulation Software RIKEN http://www.nsc.riken.jp/p8-eng.html

Chemist members of nano-scienceWG1-1About 70 researchers

Physicist members of nano-science

NTTM. BoeroNECWG1-2About 100 researchers

DFTNano biological systemsDDSfunctions and materials for information technologyDDSenergyMODPDFEMMDRISM13

Related projects: GRAPE-DR 2004 - Leader: Kei Hiraki (Univ. Tokyo) http://grape-dr.adm.s.u-tokyo.ac.jp/project-en.html1. Research Purpose GRAPE-DR project aims to construct an information system infrastructure which combines the high-speed Internet, high-speed computer systems, and high-capacity storage systems to serve the need in the field of scientific researches. Specifically (1) distributed data sharing for experimental science, (2) high-speed computation for various large-scale numerical simulations, and (3) distributed database processing for scientific data.

2. Target By 2008, we construct an information system infrastructure for science with 2PFLOPS computation power and network throughput over 40Gbps. We also provide libraries, applications and database system with mining facilities for various fields of academic research.

GRAPE-DR (2)http://grape-dr.adm.s.u-tokyo.ac.jp/project-en.htmlHardware design

GRAPE-DR (3)Recent achievement They have succeeded in the development of a processor chip (PC) with 512 processor elements with performance of 512 Gflops. This is the world highest performance of one chip. news release: 2006/11/06Cf: final target: PC with 1024 processor elements with performance of 1Tflops.

Related project: PACS-CS 2005 - 2007Leader: Akira Ukawa (Tsukuba Univ.) http://www.ccs.tsukuba.ac.jp/PACS-CS/

(Parallel Array Computer System for Computational Sciences ) The PACS-CS Project aims to advance frontiers of computational sciences through development of a massively parallel cluster PACS-CS and its concentrated usage on the problems that require large-scale simulations. One of the emphases of the Project is placed on the development of first-principles quantum simulations in materials and life sciences capable to treat O(10,000) atoms, thereby allowing an exploration of the connection between the spatial structure and function characteristic of nano materials such as carbon nano-tubes and large bio molecules such as proteins. Another emphasis is advancement of full QCD simulations in particle physics and astrophysics research which forms the core of our fundamental understanding of the creation and history of our Universe. The design targets are a peak performance of 14.3 Tflops with 2,560 nodes connected by a 3-dimensional Hyper-Crossbar Network. The target for the start of operation is July 2006.

Related projectBy March 2008, Tokyo Univ. : 150 Tflops Kyoto Univ. : 66 Tflops Tsukuba Univ.: 80 Tflops total 296 Tflops The above universities have set up joint agreement for cooperative operation of these systems.

Summary: Hardware developmentProjects involving hardware development

PACS-CS 14.6 Tflops, July/2006 joint project among Tokyo, Kyoto and Tsukuba Univ. 300 Tflops, March/2008 GRAPE-DR 2 Pflops, fiscal 2008 (maybe March/2009?) next generation supercomputer (NGSC) project 10 Pflops, March/2011

Other application projects (1) Grant-in-Aid for Scientific Research in Priority Areas: Ministry of Education, Culture, Sports, Science and Technology (MEXT)

Development of New Quantum Simulators and Quantum Design: 2005 2008 Hisazumi Akai (Osaka University) http://ann.phys.sci.osaka-u.ac.jp/~tokutei/ Molecular Theory for Real Systems: 2006 2009 Shigeyoshi Sakaki (Kyoto University) http://www.riron.moleng.kyoto-u.ac.jp/

F. AryasetiawanD.W. AgericoR. MuhidaV.A. Dinh WG2Members in Quantum Simulator and Quantum Design

WG3Members in Molecular Theory for Real Systems

Other application projects (2)CREST (Core Research for Evolutional Science and Technology): supported by JST (Japan Science and Technology Agency)

New High-Perfomance Information Processing Technology Supporting Information-Oriented Society Aiming at the Creation of New High-Speed, Large-Capacity Computing Technology Based on Quantum Effects, Molecular Functions, Parallel Processing etc. : 2001 2003 +5 Hidehiko Tanaka (Institute of Information Security)

The innovation of Simulation Technology and the Construction of Foundations for Its Practical Use: 2002 2004 +5 Norihisa Doi (Chyo University) http://www.simulation.jst.go.jp/

High Performance Computing for Multi-Scale and Multi-Physics Phenomena: 2005 2007 + 5 Genki Yagawa (Toyo University) http://www.multi.jst.go.jp/en/index.html

Virtual Lab in Nanotechnology Area: 2004 - 2007For each area, 3 to 5 groups are funded each year.

Exmaples of projects in Dois areaDevelopment of Multi-scale and Multi-physics Simulation of Heart forDisease Care and Drug Discovery: Toshiaki Hisada (Univ. Tokyo)Normal beatAbnormal beathttp://www.sml.k.u-tokyo.ac.jp/

Exmaples of projects in Yagawas areaSimulations and Dynamics for Nanoscale and Biological Systems Kimihiko Hirao (Univ. Tokyo)UTChem is high-performance software for carrying out ab initio quantum chemistry calculations to calculate electronic wavefunctions, and chemical properties of polyatomic molecules.http://utchem.qcl.t.u-tokyo.ac.jp/

Other application projects (3)Precursory Research for Embryonic Science and Technolgoy, SAKIGAKE: supported by JST

The innovation of Simulation Technology and the Construction of Foundations for Its Practical Use: 2002 2004 +3 Norihisa Doi (Chyo University) http://www.simulation.jst.go.jp/

Funded to individual researcher

Computer centers for common useISSP, IMS, IMR, Tsukuba Univ.WG1NanoscienceOf NGSC projectWG2Quantum simulatorand Quantum designWGMolecular theoryfor real systemsWGJSTCRESTSAKIGAKENano-virtual LabWG5ITBLWG6Materials scienceto be formedWG7Council for industrial applications of supercomputingA plan forConsortium for Computational Materials science

Missions for computational science communityDevelopment of linear scale O(N) algorithms If the calculation scales N3, next generation supercomputer can treat only 6 or 7 times bigger system than earth simulator.Highly parallel computation So far in my community of materials science, the maximum parallelization has been up to about 1,000 processors, while the NGSC will have more than 100,000 processors.Multi-scale simulations Most problems in real life have aspects of multi-scale.Overcoming of fundamental difficulties For instance, in the field of nano-bio, vdW interaction is crucially important, while present DFT treatment fails in treating it.