real-time data reduction integrated into instrument...
TRANSCRIPT
![Page 1: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/1.jpg)
![Page 2: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/2.jpg)
Outline
Setting The Context
Use-Case
Solution
Implementation’s Examples
Evolution
![Page 3: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/3.jpg)
Setting The Context
C++
Java
CORBA
Server
GUI
Transport Layer
![Page 4: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/4.jpg)
Use-Case
Electronic log
PlotScreen generator
WEB Spy
![Page 5: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/5.jpg)
Use-Case
Interaction with data reduction/analysis
Instrument Status
What To Do Next
Multi-process
Multi-environment
Synchronization
Crash management
![Page 6: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/6.jpg)
Possible Solutions
Monolithic Microservices
Every single Scientific method is included in NOMAD
Difficult to maintain
No freedom and flexibility for scientists
![Page 7: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/7.jpg)
What Is NAPPLI
Lightweight application server
Multiplatform (Linux, Mac, Windows)
Provides client API in C++ and Java
Manages the entire application lifecycle
Start/Stop nicely
Implements different communication patterns
Request/response Publisher/Subscriber Return value at the end
![Page 8: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/8.jpg)
NAPPLI Basics
Start
Subscribe Protocol Buffers
![Page 9: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/9.jpg)
MATLAB Synchronous Server
LOOP
Control Science Computer
NAPPLI REMOTE MATLAB
MATLAB ENGINE
1:start RemoteMatlab 2:start
3:request
6:response
7:stop RemoteMatlab
12:success
8:stop
4:script
5:image
9:stop
11: 10:
![Page 10: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/10.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 11: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/11.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 12: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/12.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 13: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/13.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 14: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/14.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 15: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/15.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 16: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/16.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 17: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/17.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 18: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/18.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 19: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/19.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 20: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/20.jpg)
Q Space Transformation
SCAN Raw Data
Q Space
![Page 21: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/21.jpg)
Coincidence Asynchronous Server
Control Science Computer
NAPPLI NPP
ENGINE
1:start RemoteNPP 2:start
3:publish data
5:publish results
6:stop RemoteNPP
9:success
7:stop
8:
4:publish data
X
![Page 22: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/22.jpg)
Coincidence Experiment Setup
![Page 23: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/23.jpg)
Detector Layout
4 x NaI back-catcher
8 x BGO rear side shield
8 x BGO side shield
4 x Ge crystals 24
correlated detectors
![Page 24: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/24.jpg)
Coincidence Asynchronous Server Average event rate ~ 1 MHz
Event-mode file ~ 2 GB in less than 5 min.
Detector - Crystal Raw Rate (kHz) Clean Rate (kHz)
1 XXX xxx
1 1-XXX 1-xxx
2 2-XXX 2-xxx
3 3-XXX 3-xxx
4 4-XXX 4-xxx
------ ------ ------ Coincidence Raw Rate (kHz) Clean Rate (kHz)
Single XXX xxx
Fold 2 YYY yyy
. … …
Fold n ZZZ zzz
![Page 25: Real-Time Data Reduction Integrated into Instrument ...accelconf.web.cern.ch/AccelConf/ICALEPCS2015/talks/thhb3o02_tal… · Monolithic Microservices Every single Scientific method](https://reader035.vdocuments.mx/reader035/viewer/2022071110/5fe4ec9fabaee709496e4411/html5/thumbnails/25.jpg)
Conclusion
Manage and organize the execution of different applications of the instrument control software
Easily distribute and run new/existing scientific computations over different computers
Flexible in term of platform and application’s interaction
Coming soon: decision taking within NOMAD workflow based on data analysis
NAPPLI
http://forge.ill.fr/projects/nappli