report on belle ii data handling group kihyeon cho high energy physics team kisti (korea institute...
TRANSCRIPT
![Page 1: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/1.jpg)
Report on Belle II Data Handling Group
Kihyeon ChoHigh Energy Physics Team
KISTI (Korea Institute of Science and Technology Information)
June 16~18, 2010Belle II Computing workshop, Krakow, Poland
1
![Page 2: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/2.jpg)
Kihyeon Cho
Contents
PrologueWho?Belle II time scheduleData Handling OutlineStatus and planTo do listDiscussionEpilogue
2
![Page 3: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/3.jpg)
Kihyeon Cho
Prologue
KISTI To maintain the central AMGA server and take care of the
metadata replication. To work on the replication of metadata.
Krakow is responsible for the job submission tool and the assignment of files to jobs based on the file-level metadata. To take care of the recovery mechanism for failed jobs.
We agreed to continue with the parallel development of AMGA clients for the moment and evaluate them
when we do the large scale data handling test. We will merge the systems:
command script -> c++
portal tool -> python
3
What has been discussed on Dec. 15, 2009
![Page 4: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/4.jpg)
Kihyeon Cho
Who?
KISTI GroupHEP Team – JungHyun Kim(JK), Kihyeon Cho(KC),
YoungJin Kim, …AMGA Team – Soonwook Hwang, Sunil Ahn(SA), HanGi Kim,
Tae-sang Huh, …NSDC Team – Beob Kyun Kim(BK), Seoyoung Roh,
Jonghu Lee, Haengjin Jang,…
Melbourne GroupTom Fifield, Martin Sevior, …
Krokow GroupMaciej Sitarz, Mitosz Zdybat, Rafat Grzymkowski, Henryk Polka, …
KEK, Karlsruhe, etc…
4
![Page 5: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/5.jpg)
Kihyeon Cho 5
Belle II Time schedule
Hara San
![Page 6: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/6.jpg)
Kihyeon Cho 6
![Page 7: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/7.jpg)
Kihyeon Cho
Data Handling Outlines
7
KISTI
KEK NSDC, Melbourne, Krokow, …
![Page 8: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/8.jpg)
Kihyeon Cho
Goal of Data Handling System
8
• To improve the scalability and performance• To run based on grid farm => AMGA
To construct the DH system for Belle II
Data Handling system
Meta catalog system
(2009)
Data Cache=> Ntuple
(2010)
![Page 9: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/9.jpg)
Kihyeon Cho
Status and plan
To construct Belle II Data Handling System To test Large Scale Data Handling
Belle Data at NSDC farm at KISTIBelle II Data (Random data) => doingBelle Data at KEK => will do
Data Handling and Job Management for Belle II Grid
Interactive Grid ConsoleData Handling -> FTS
9
![Page 10: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/10.jpg)
Kihyeon Cho
Status and plan of KISTI group
Data/Job handling prototype => doneJA, SA, March 2010
Large Scale Data Testing => doneJK, BK, KC March 2010Depending on NSDC
Data Caching R&D => doneJK, June 2010
Configuration for AMGA system in DH system (Optimization of large scale data) => doing
SA, JK, Dec 2010
RB & DH system based on Grid system => doingJK, SA, Dec 2010
10
http://b2comp.kek.jp/~twiki/bin/view/Computing/ProjectPlanning
![Page 11: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/11.jpg)
Kihyeon Cho 11
Status and plan of NSDC @ KISTI
=> See Dr. Jonghu Lee’s talk
![Page 12: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/12.jpg)
Kihyeon Cho
Plan and Status of Krokow Group
12
=> See Milosz Zdybal’s talk
![Page 13: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/13.jpg)
Kihyeon Cho
Large Scale Data Handling TestThe whole process:
Data Transfer => Extraction => Scalability => Replication => LFC => Grid farm
KISTI takes care for up to LFC. Then, Melbourne gets LFC file and does Grid test.
JungHyun is rebuilding from metadata attributes to logical files. Therefore, Junghyun gave logical file name to Tom.
The procedure for metadata is as follows:
Metadata => LFC => SE => Physical Data
For this processing, we have installed Grid SE
KISTIMelbourne
13
Melbourne / NSDCKISTI
=> See Dr. JungHyun Kim’s talk13
![Page 14: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/14.jpg)
Kihyeon Cho
Belle II DH group activities
Host Belle II DH group meetingApril 27, 2010May 16, 2010KISTI internal DH group meeting – April 8, 2010
Report to Belle computing meetingJanuary 19, 2010March 02, 2010April 27, 2010June 3, 2010
1423年 4月 21日
![Page 15: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/15.jpg)
Kihyeon Cho
Belle II Data Handling Group meeting
15
First and Third Thursday 5:00 PM (KEK Time)
![Page 16: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/16.jpg)
Kihyeon Cho
KISTI group meeting
Belle II DH KISTI meetingDate: April 8, 2010 1:30-3:30 PMParticipant:
Kihyeon Cho ([email protected]) – HEP team Junghyun Kim ([email protected]) – HEP team Soonwook Hwang ([email protected]) - AMGA team Tae-Sang Huh ([email protected]) –AMGA team Sunil Ahn ([email protected]) – AMGA team Beob Kyun Kim ([email protected]) – NSDC team Jinseung Yu ([email protected]) – NSDC team Seoyoung Roh ([email protected]) – NSDC team
Minutes of meetingThe role of HEP team, AMGA team and NSDC team
23年 4月 21日
16
![Page 17: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/17.jpg)
Kihyeon Cho
Products
Conference talks1. The Advanced Data Searching System with AMGA at the
Belle II ExperimentCCP2009, Gaushung, Taiwan, Dec., 2009Submitted to CPC (Computational Physics Communication)
2. Data Cache System at Belle II experimentCCP2010, Trondheim, Norway, June 2010
3. Belle II Data Management systemCHEP 2010, Taipei, Taiwan, Nov. 2010
Draft1. Design of the advanced metadata service system with
AMGA for the Belle II experimentTo be submitted to JKPS (Journal of the Korean Physical Society)
17
![Page 18: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/18.jpg)
Kihyeon Cho 18
![Page 19: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/19.jpg)
Kihyeon Cho
Belle II DH Group – Daeduk net (2010.4.6)
19
http://www.hellodd.com
![Page 20: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/20.jpg)
Kihyeon Cho
To do list
Configuration for AMGA system in DH system (Optimization of large scale data) => doing
Dec 2010RB & DH system based on Grid system => doing
Dec 2010To test Large Scale Data Handling
Belle II Data (Random data) => doingBelle Data at KEK => will do
Data Handling and Job Management for Belle II GridInteractive Grid ConsoleData Handling -> FTS
20
![Page 21: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/21.jpg)
Kihyeon Cho
Discussions (suggestions)
1. Belle II DH Group meeting• 1st and 3rd Thursday at 5:00 PM
2. The 3rd Belle II Computing Workshop (?)• Date: December 2010• Place: KISTI, Daejeon, Korea• Hosted by HEP team @ KISTI
21
![Page 22: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/22.jpg)
Kihyeon Cho
• For more in detail, please see today’s talks 1. Test of Belle II Data Handling System
(JungHyun Kim)- EVO2. Technical aspect of AMGA system (Soonwook
Hwang) – EVO3. The Data Cache System at the Belle II
Experiment (Kihyeon Cho) – CCP2010 practice talk
4. Introduction to Ganga (HanGi Kim) - EVO
22
Epilogue
![Page 23: Report on Belle II Data Handling Group Kihyeon Cho High Energy Physics Team KISTI (Korea Institute of Science and Technology Information) June 16~18, 2010](https://reader033.vdocuments.mx/reader033/viewer/2022051115/56649f155503460f94c2a51f/html5/thumbnails/23.jpg)
Kihyeon Cho 23