August 1-4, 2012APCTP Mini-Workshop on Nuclear Physics and Nuclear AstrophysicsDogye Campus, Kangwon National University

1

Kihyeon Cho 2

Contents

Introduction Theory-experiment-simulationSimulation Project Geant4Supercomputing portingSummary

3

4

Kihyeon Cho

e-Sci-encepara-digm

Theory

Simu-la-tion

Experi-ment

• Theory-Simula-tion• PGS• LQCD• Geant4

To probe the Standard Model and search for New Physics

CDFBelle/Belle II

e-Science paradigm

Theory-Experiment-Simulation

K.Cho and H.W.Kim, JKPS (2009)

• Theory-Exp. • FKPPL• B physics

• e-HEP env. - Remote Cont. - DP, DH - EVO

Kihyeon Cho

International Collaborations

6

Belle*/Belle II* @ KEK, Japan

CDF*@FNAL, USA

e-Science @KISTI

IN2P3@France

France-Korea Particle Physics Laboratory

* Official Collaborations

• The Leader of FKPPL - CDF Project (Kihyeon Cho)• The Chair of Belle II Data Handling working Group (Kihyeon

Cho)

Kihyeon Cho7⇒ Feed-back between experiments (Belle & CDF) and theories inside

KISTI

1. Theory-Experiment

Kihyeon Cho 8

Kihyeon Cho 9

Kihyeon Cho 10

Kihyeon Cho

e-HEP environment

1. Data production• CDF Remote Control Room @KISTI

2. Data processing• Pacific CAF(CDF Analysis

Farm) ⇒ North America CAF @KISTI

3. Data Analysis Collabora-tion

• EVO servers @KISTI

4. Belle II Data Handling Sys-tem • Working Group Chair (K. Cho)

To study high energy physics any-time anywhere even if we are not on-site (accelerator laboratory)

2. Experiment-Computing Simu-lation

Kihyeon Cho

CDF Remote Control Room

CDF Operation Center KISTI Remote Control Room

We take shifts at KISTI even if we are not at Fermilab.

0

100

200

300

400

500

600

700

2008 2009 2010 2011

CDF Remote Control RoomTotal used hours (2011.10.1)

Total hours8

2325 24

0

5

10

15

20

25

30

2008 2009 2010 2011

CDF Remote Control Room(2011.10.1)

days

Kihyeon Cho 13

Kihyeon Cho

Theory model – Simulations MadGraph-pythia-PGS LQCD Geant4 ..

14

Ex) A study of Higgs model using cyberinfrastructur\ @KISTI

3. Theory-Simulation

Kihyeon Cho 1515

- To study high energy physics anytime and anywhere

- Fusion research of experiment-computing–theory

Publication

B physics using the concepts of e-Sci-ence

⇒ We are applying both concepts to high energy physics. It is a great success!

commentCollider physics using e-Science paradigm of experiment-comput-ing-theory

B physics

Leading Belle II Data Handling Working Group which consists of more than 30 persons from 12 countries

e-Science paradigm

Theory-Experiment-Simulation

Kihyeon Cho

e-Sci-encepara-digm

Theory(Nuclear physics)

Simu-la-tion

(Geant4,et)

Experi-ment(RISP

Acceler-ator)

For Nuclear Physics and Nuclear As-trophysics- r-p process- Shell models…

To apply domestic accelerator experiment

Theory-Experiment-Simulation

Astro-Nuclear Simulation by RISP ac-celerator’s user group

17

18

Geant4

동아일보(2012.7.5)

2012 년 7 월 4 일 오후 4 시 ( 한국시각 )

21C 물리학계 최고의 사건 ! – Higgs 입자 발견

Kihyeon Cho

Project Outline

Title: Development of Geant4 for accelerator and medical physicsPI: Kihyeon ChoSponsor: Creative project by KISTIBudget: ~ half million dollarPeriod: Jan. 1, 2012~ Dec. 31, 2012

(Possibly extended)Goal: Porting on Supercomputing Simulation for HEP and Medical physics using Geant4 Profiling and Large-scale test, evaluation and improvement Geant4 multi-proecssing R&D

Web page https://hep.kisti.re.kr/geant4

19

Kihyeon Cho

Man Power

KISTI, Korea Kihyeon Cho, Soonwook Hwang, Taegil Bae,

Youngjin Kim, Sangsu RyuYonsei University, Korea Kyungho Kim, Youngjoon KwonNational Cancer Center Se Byeong Lee, Sungkoo Cho, Jaeik ShinFermilab, USA Soon Yung Jun, Daniel Elvira, James

Kowalkowski, Philippe Canal, Krzysztof GenserCMU, USA Dongwook JangUCSF, USA Jungwook Shin, Bruce Faddegon

20

Kihyeon Cho

Global Community(FKPPL)

HEP - Accelerator R&D- Detector, …

Medical Physics

- Proton Accelerator, ..

Geant4 R&D

Grid Farm Supercom-puter

Geant4 Collaboration- code development

UC San Fran-cisco

- Proton Accel-erator GEANT4 Application

- TOPAS

R&D

Infra-struc-ture

HEP Medical Physics

Fermilab- Next genera-tion simulation package devel-opment

Appli-cation

CERN IN2P3 KEK

Interna-tional

Collabora-tion

21

Kihyeon Cho

High Energy Physics vs. Medical Physics

High Energy Physics Medical PhysicsInput data N/A 50MB (ex, CT data)Farm Grid Farm/ Supercom-

puterSupercomputer

Porting Grid Farm SupercomputerApplication Package HEP Application TOPAS, GATECommunity HEP Application (Accel-

erator and Detector)Proton Accelerator Cen-ter

International Collabora-tion

Fermilab, USA UCSF/SLAC/MGH, USA

Contents of interna-tional Collaboration

Detector simulation package using Multi-thread/parallel comput-ing

Proton Accelerator us-ing Geant4 application

22

Kihyeon Cho

Work on Progress

Geant4 R&D Supercomputing Porting

SUN Cluster Porting - Geant4-ST, MPI(ParGeant4), Geant4-MT IBM AIX OS – Geant4-MT=> Performance measurement and analysis

Parallel Computing Using Many core Collaboration with Fermilab

Accelerator physics application Accelerator and Detector R&D, etc. Nuclear Astrophysics Theory models (MadGraph, ..)

Medical Application Proton accelerator, therapy, etc.

23

Kihyeon Cho 24

Simulation on secondary beam

Object Geant4 Simulation for secondary beams and dE/dxMethod To analyze secondary beam depending on incident beam energy, target ma-

terial, interaction model, …

Output Secondary particles Energy dissipation…

Kihyeon Cho 25

Preliminary results

26

1. Geant4 및 HEP 소프트웨어 설치2. 슈퍼컴퓨팅 초보사용자 신청서 접수3. 전략과제 신청서 접수

Kihyeon Cho 27

Kihyeon Cho 28

Kihyeon Cho 29

Kihyeon Cho 30

Kihyeon Cho 31

Kihyeon Cho 32

Kihyeon Cho 33

Kihyeon Cho

KISTI Supercomputing Porting

Sun Linux Cluster Geant4 ST Geant4 MPI(ParGeant4) Geant4 MT

IBM AIX OS Geant4-MT Medical Application

34

Tachyon(SUN)

Phase 2

Manufacture SUN Blade 6048

Architecture Cluster

Process model Intel (Nehalem)

# of Nodes 3,200 nodes

# of CPU cores 25,600(8 per node)

Rpeak(Tflops)

300

324

Total Memory 76TB

Disk Storage 1.2PB

Tape Storage 2PB

Interconnection Network IB 4X QDR

Kihyeon Cho 35

Supercomputing Porting

Done HEP software

Cernlib, CLHEP, ROOT Belle library Geant4

Geant4 ST (Single Thread) Geant4 MPI Interface ParGeant4

To do list Geant4 MT (Multi Thread) Nuclear Astro-physics Channel Theory models..

Kihyeon Cho

|-- ROOT| |-- root -> root_5.22.00.patches/| |-- root_5.22.00.patches| |-- root_v5.14.00| `-- root_v5.14.00.tar.gz|-- TOP-C| `-- 2.5.2|-- belle| |-- belle| |-- belle.install.tar.gz| |-- cern -> ../cern/| |-- i386| `-- local|-- cern| |-- 2001| |-- 2005| |-- 2006| |-- 2007| `-- root -> ../ROOT/root|-- etc| |-- bashrc_general| `-- cshrc_general|-- marshalgen| `-- 1.0`-- public|-- bin

/applic/applications/hep |-- CLHEP| `-- 2.1.2.2|-- Geant4| |-- 9.4.p03| |-- 9.5| `-- 9.5.p01|-- Geant4-data| |-- G4ABLA3.0| |-- G4EMLOW6.19| |-- G4EMLOW6.23| |-- G4NDL0.2| |-- G4NDL3.14| |-- G4NDL4.0| |-- G4NEUTRONXS1.0| |-- G4NEUTRONXS1.1| |-- G4PII1.2| |-- G4PII1.3| |-- PhotonEvaporation2.1| |-- PhotonEvaporation2.2| |-- RadioactiveDecay3.3| |-- RadioactiveDecay3.4| |-- RealSurface1.0| |-- geant4-data-conf.csh| `-- geant4-data-conf.sh

KISTI 슈퍼컴 Geant4/ HEP Directory 구조

Kihyeon Cho

[## Geant4 환경변수 설정 @tachyon2c ~]$ source /applic/applications/hep/Geant4/9.5.p01/bin/geant4.(c)sh[@tachyon2c ~]$ source /applic/applications/hep/Geant4/9.5.p01/share/Geant4-9.5.1/geant4make/geant4make.(c)sh ## 테스트 디렉토리 생성 및 예제 파일 복사 @tachyon2c ~]$ mkdir test[@tachyon2c ~]$ cp -r /applic/applications/hep/Geant4/9.5.p01/share/Geant4-9.5.1/examples/extended/analysis/A01 test[@tachyon2c ~] cd test

[## 컴파일 및 실행 @tachyon2c ~]$ make[@tachyon2c ~]$ $G4WORK_DIR/bin/Linux-g++/A01app ...................................... HadCalLayerPhysical[-1] HadCalLayerLogical 1 Lead HadCalScintiPhysical[0] HadCalScintiLogical 0 Scintillator /Had-Calorimeter Idle> /vis/open OGL # for using OpenGL Idle> /vis/drawVolume # example geometry drawn

Idle>/vis/scene/add/trajectories # to make visualization, add trajectories and hits Idle>/vis/scene/add/hits Idle>/run/beamOn 1 # run an event

Login node 실행 예제

Kihyeon Cho

Nuclear Physics and nuclear Astrophysics

Nuclear physics and Astrophysics theory needs 10,000 cores and 1PB storage. Shell model r-p process Etc.

To buy clusterTo use KISTI supercomputer con-stantly Processing on MoU between RISP of IBS and KISTI super-

computing Center => It take times

38

Kihyeon Cho

초보사용자

http://helpdesk.ksc.re.kr

39

Kihyeon Cho

전략과제

http://www.ksc.re.kr

40

Kihyeon Cho 41

Kihyeon Cho

1. Geant4 User Forum Website (https://hep.kisti.re.kr/geant4)

2. Geant4 Indico Website (https://hep.kisti.re.kr/indico)

3. User support mail account (hep@kisti.re.kr)

Geant4 사용자 지원

Kihyeon Cho 43

Summary

As one of e-Science paradigm of ex-periment-Theory-Simulation, Simula-tion (so-called Geant4) project has been launched this year at KISTI.

Simulation project supports Nuclear Physics and Astro-nuclear Physics.

User support for supercomputer hep@kisti.re.kr https://hep.kisti.re.kr/geant4

HEP@KISTI