application and research of national supercomputer center ... web/nscc-tj application and...

Application and Research of National

Supercomputer Center in Tianjin

Dr. MENG Xiang-Fei

mengxf@nscc-tj.gov.cn

Outline

Introduction to National Supercomputer Center in

Tianjin (NSCC-TJ)

Applications on TH-1A

HPC Clouding & Big Data

International Collaboration

Outline

Introduction to National Supercomputer Center in

Tianjin (NSCC-TJ)

Applications on TH-1A

HPC Clouding & Big Data

International Collaboration

Chinese government: MOST, MOF, MOII, …….

Local government: Tianjin Binhai New Area

To accelerate the economy, science innovation and industry

of China

To provide high performance computing service to whole

China and even to all over the world

Overview of NSCC-TJ

Quad cpu blade

FT-1000

X5670

Chips

Twin GPU blade Compute node

rack (16 x cn)

Cabinet (4 x rack)

On-line storage

TH-Net

(4CPU+2GPU)

Host Computing Power: TH-1A

System Resource

Different Platforms: HPC, CC, Big-Data, ….

Overview of NSCC-TJ

LASG, CAS

HKPU HKU

Nan Kai

University

Zhejiang

University

USTC

Tsinghua

University

Peking

University

Tianjin

University

Northeastern

University

DUT

Jilin

University

National Animation Industry Park

718th CSIC

CATRC

Tianjin FAW

Automobile

BGI

China Oilfield Services

Co. Limited

SINOPEC

GRI

CNPC

BGP

China First Heavy Industries

Great Wall

Motors

SAMSUNG

Tianjin Institute of

Pharmaceutical Research

NUDT, CAS

and NSCC-TJ

IoM, CAS

IMR, CAS

IoP, CAS

IoB, CAS

SIMM, CAS

Nanjing Hydraulic

Research Institute

AoMMS

State Oceanic

Administration

Cooperation

with Institutes

Cooperation

with Universities

Industrial

Collaboration

Xi’an Jiaotong

University

CUP

Fudan

University

Shanghai

Jiaotong University

Overview of NSCC-TJ

Joint

Lab

Union

Joint

Innovation

center

NVIDIA

BGI

Communication Univ.

BGP

SC Center

TH Engineering Design

Modern Ports Adm.

HPC Dev. & Appl.

NAOC

LASG

Overview of NSCC-TJ

Outline

Introduction to National Supercomputer Center in

Tianjin (NSCC-TJ)

Applications on TH-1A

HPC Clouding & Big Data

International Collaboration

Engineering Design

Bio-informatics Animation & Movie

Oil Exploration

Remote Data

Metrology & Climate Aero and Space Craft

Design

Environment Science

Applications of NSCC-TJ

Seismic exploration is the practical application of seismic methods to measure the physical properties of rocks,

and in particular, to detect the measurable physical differences between rocks that contain ore deposits or

hydrocarbons and those without.

Seismic data processing, which is aimed to image subsurface geological structures with field data, requires a lot of

numerical calculations, especially with the development of reverse time migration and prestack full waveform

inversion, the demanding for high-performance computer growth rapidly[1].

1. Seismic exploration

Reverse time migration utilizes the full wave

equation to extrapolate the seismic wavefield, which

makes it the most accurate imaging procedure, it

contains two part:

1) Extraploation of source wavefield and receiver

wavefield: huge computation cost, massive memory

demand for the intermediary results, frequent I/O

operations;

2) Extract image with imaging condition: frequent I/O

operations to processing huge amount of intermediary

data.

v(x,y,z) subsurface velocity

[1] Zhu et al., Supercomputing with prestack migration and prestack inversion[C], SPG/SEG Shenzhen 2011 International geophysics

conference symposium. Shenzhen, 2011:1019-1025

3D RTM profile

RTM Application on TH-1A

Block A Block B Block B

Optimization

System BGP Computing

Center

TH-1A 7100 Nodes

TH-1A 2000 Nodes

Block Square: 1050Km2 Depth: 5 Km Square: 2600Km2 Depth: 5Km

Data 70000 shots，700GB 217900 Shots, 2.2TB

Time 30 days 16 hours 65 hours 36 hours

1 shot

513 shots 158 shots

5 shots

5.266

3.245

1.193

0.66

0

2

4

6

Apr.2010 Nov.2010 Jan.2012 Dec.2012

Co

st (

CP

U*H

/Sh

ot)

CPU*H per shot on TH-1A with RTM

1. Seismic exploration: RTM

Full-waveform inversion (FWI) is an automated method for refining a velocity

model by iteratively attempting to match modelled data with recorded data .

Full-waveform inversion

Challenges

Intensive computation: FWI needs to iteratively solve the two-way

wave equation, the computation cost of each iteration is equal to

the cost of reverse time migration, which is unaffordable for the

computing center of oil company;

Massive memory cost: To solve the two-way wave equation in

frequency domain needs to solve the massive sparse linear

equations, which need a huge amount of memory;

1. Seismic exploration: FWI

Initial Mordel Forward Modeling

Comparison

Final Velocity Model

Conform

)||||min( 2

Ni

obs

ii ddE

Solve the equation to modify the initial model with conjugate gradient method

Not Conform

Initial velocity model for 3D marine data full waveform inversion

Research on acceleration of

full waveform inversion

Carry out massive parallel

optimization of 3d pre-stack full

waveform inversion jointly;

Laplace transform before model

initiation;

Program Efficiency accelerate 5X,

inversion of 3d marine data just need

3days.

Final velocity model of 3D marine data full waveform inversion

1. Seismic exploration: FWI

Gather generation is a data sorting procedure which rearrange the reverse time

migration image according to the coordinate of image point and offset.

Challenges

Huge data volume: Total data volume about 100 TB for one

field;

Complex distribution: generation of one image point gather

involves about 2000~3000 partial image file, totally 4 TB;

Time consuming of gather generation equals to reverse time

migration!!

FFT

1. Seismic exploration: GG

Gather generation

Time consuming of each node after optimization Time consuming of each node before optimization

Optimization of gather generation

Utilizing professional tools to find efficiency bottleneck;

Specific optimization method to specific bottleneck;

After optimization, data rearrange efficiency improved 3x, Total efficiency

improved 5x.

1. Seismic exploration: GG

Genomics Analysis

and Processing

New Drug Design

Prediction of protein

structure and dynamics

behavior

2.Life Science

2. Life Science: Genomics research

Human whole-genome shotgun (WGS)

With the rapid development of Genome sequencing technology，

human genome sequencing confronts technical challenges on

computation, such as:

Data accumulates at an exponential rate（beyond Moore‘s

Law），the amount of genome data is up to TB or even PB

Sequencing and splicing is more complex，CPU architecture is

unable to complete these task

Data intensive computing require higher storage and

communication systems

Carrying out technical cooperation with BGI and building Hadoop

genome data platform. Stored bioinformatics data is more than 2PB

on TH-1A.

Based on Hadoop platform, we construct genome analysis Gaea

cloud platform.

Gaea cloud platforms have faster data analysis speed, stronger

scalability, higher computing resources, etc.

Gaea on TH-1A only costs 15 hours to complete 64 X all human

WGS data analysis process.

Research and development of independent innovation

of new drugs for treating epilepsy

Through the CADD and supercomputer simulation, combined

mutagenesis and electrophysiology experiment, built ztz240 with KCNQ2 complex structural model.

Carried out lots of 200-300ns Molecular dynamics simulation, obtained accurate optimization ztz240 combination mode.

A KCNQ2 MD process with 512core on TH-1A , only need seven days.

Screened 200,000 compounds using ztz240 combination mode, without any chemical modification, obtained lead compounds for drug treatment of epilepsy which has a good activity ability in animal.

The research results have been applied for patent, and received by the international famous journal "Cell Research“.

Epilepsy Epilepsy with new drug

2. Life Science: New drug R&D

Compound lib

KCNQ2/ztz240

Structure

KCNQ2

Structure

Kv1.2

crystal structure

Virtual screening

Electrophysiological study

Antiepileptic testing

Reform of structure and optimization

35 candidate

molecules

9 active compound

2 Antiepileptic

candidate molecules

MD simulation

Modeling

LigBuilder

LigBuilder is a general-purposed program package written for structure -based drug design procedure. Based on the 3D structure of the target protein, it can automatically build ligand molecules within the binding pocket.

New Structure

Automatic Generating

Seed Exploring Mode

User Defined Seeds

Binding Site Detection & Drugability Analysis

Growing Mode

Linking Mode

Binding Affinity

Synthesizability

Lock-Key Match

PhyChem Property

Evaluation

Candidates

Fragment Library

GA Process

Energy Optimize

Design

Exploring Mode

2. Life Science:

Hao Chen et al., A novel binding pocket of cyclin-dependent kinase 2, Proteins, 2009,74,122-132 Wei, D., X. Jiang, et al.Discovery of multitarget inhibitors by combining molecular docking with common pharmacophore matching. Journal of Medicinal Chemistry. 2008, 51(24): 7882-7888. Wu, Y., C. He, et al. Dynamic modeling of human 5-lipoxygenase–inhibitor interactions helps to discover novel inhibitors. Journal of Medicinal Chemistry. 2012,55,2597-2605

Red: Catalyst domain Green: New binding site with acceptable CavityScore

2. Life Science: LigBuilder

According to viscosity, compressibility and 3D motion, the non-linear partial differential

equation is complex as well as massive computation.

To boost cost-performance and save cost of experiments, we consider the

supercomputer to carry out large-scale numerical calculation to solve the non-linear

partial differential equation.

Now, we are carrying out verification and optimization on the self-developed CFD

software with users, including reliability, accuracy and efficiency.

Structure of flow around Bicylinder

High angles of attack on delta wing

3.Aerodynamics Simulation

Current driven by radio frequency wave and heating plasma is two methods of smooth running of

Tokamak. Nonlinear interaction between RF wave and plasma becomes very important.

Gyrokinetic Toroidal Code (GTC) is a massively parallel particle-in-cell code for first-principles,

integrated simulations of the burning plasma experiments.

Collaborated with Peking Univ. , USTC, Zhejiang Univ., NSCC-TJ developed the GPU version of GTC

code.

Performance of GPU-GTC based on Tianhe-1A • Large scale benchmark: up to 4096 computing nodes are used

• High efficiency: up to 35 billion electrons per second are processed

• Independent development: develop heterogeneous GPU-GTC code, which can get 2X speedup comparing to the

original CPU version.

• Accuracy & Good scalability: simulation is much similar to real results, which verify that accurate simulation of

lower hybrid wave in GTC 。

4. large-scale simulation of magnetic confinement fusion

This work has been selected as top 10 application software

on Titan and accepted to presentation by ISC13.

As constrained by many factors, such as environmental factor(temperature, humidity),

physical factor(diffusion, volatilization) and biological factor, we can't achieve the goal of

researching and forecasting natural environmental movement and evolution just through

field observation；

Owing to the high-speed of high-performance computer and efficient task allocation

strategy, the researches of ocean climate formation and variation, global climate change,

ocean environmental resources change and ecological evolution are greatly advanced.。

5.Environment Science

Challenges

Program calculation increased with the grid

division of physical model

Cores number to Synchronize and

communicate in parallel program exceeds ten

thousand: > 10,000 Cores

More precise model resolution: < 0.1°

7.Astronomy Research

There are various characteristics of objects in Astronomy Research, including:

• Extremely physical parameters, such as density, viscosity and pressure；

• Extremely Spatial scale from planet to universe；

• Long evolutionary time；

As limitation of theory and experiments, numerical calculations, especially high-

performance computing (parallel computing), is becoming more and more important means

for non-linear astronomy research；

Numerical simulation can study larger spatial evolution of celestial bodies and finer

structure in space;

We are able to carry out longer celestial dynamics simulation within limited time;

For the application, extend the simulation from astrophysics to the massive data analysis

and processing.

Engineering Simulation and Design

Outline

Introduction to National Supercomputer Center in

Tianjin (NSCC-TJ)

Applications on TH-1A

HPC Clouding & Big Data

International Collaboration

Yes, but we need an important module!

Combination of HPC, Clouding and Big

Data？

Possibility！

HPC, Cloud Computing and Big Data

Massive Data

Storage

Data Mining and

Search Engine

Cloud Computing

service oriented

Industry

Data Collection Data Service

Wonderful!

To carry out the R&D about the

high reliability and efficiency of

massive storage based-on TH-1A,

even PB and EB；

To carry out parallel analysis and

data processing for Big Data

based-on TH-1A, aiming at

promoting industrial development

with Big Data technique.

HPC, Cloud Computing and Big Data

Virtualization

3rd-level storage

(+500P)

2nd-level storage

1st-level storage

(Memory based Storage)

(Lustres: 12P)

10 Gigabit Ethernet

4P 4P 4P

High-Speed Intranet of TH-1A

Clouding & Big

Data Platform

Seismic

Data

Processing

Bioinformati

cs R&D

Astronomy

R&D

Engineering

Design

Platform

Intellectual

Traffic

Control

HPC, Cloud Computing and Big Data

Outline

Introduction to National Supercomputer Center in

Tianjin (NSCC-TJ)

Applications on TH-1A

HPC Clouding & Big Data

International Collaboration

International Collaboration

Project Title: Strategic collaboration with China on super-computing based on Tianhe-1A

Abstract:

SCC-Computing Project is held by two unions, one is five institutes of EU, the other

comprises the NSCC-TJ and the Tianjin University. This project, which would last two years,

based-on the Tianhe-1A supercomputer leads to large-scale applied-research and applied-

testing so that we could explore the trend of supercomputer development and application

technology in the future so as to provide suggestion for the supercomputing.

SCC-Computing between EU-China

Thanks