computer and automation research institute hungarian academy of sciences presentation and analysis...
TRANSCRIPT
![Page 1: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/1.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of Sciences
Presentation and Analysis of Grid Performance Data
Norbert Podhorszki and Peter Kacsuk
Laboratory of Parallel and Distributed SystemsMTA SZTAKI
Hungary
![Page 2: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/2.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesContents
● Relational Grid Information and Monitoring System(R-GMA)
● R-GMA Browser– Browsing status information
● Pulse– Analysis and presentation of performance information of grid
services and components
● GRM/PROVE– Performance monitoring of applications
![Page 3: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/3.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA
● R-GMA is the Relational Monitoring and Information System within the EU-DataGrid project
● Gives the illusion of a virtual relational database of the information available about grid services, resources and jobs
● SQL is the information publishing and query language
![Page 4: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/4.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of Sciences
R-GMA
Sensor Code
ProducerAPI
ApplicationCode
ConsumerAPI
Producer Servlet
RegistryAPI
Registry
SchemaAPI
Schema
“Definition of tables”
Consumer Servlet
RegistryAPI
![Page 5: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/5.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA
![Page 6: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/6.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of Sciences
R-GMA
![Page 7: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/7.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPulse
● On-line analysis and visualisation tool for
● monitoring data of resources and services and applications.
![Page 8: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/8.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPulse
● Data source components preprocessing components
view components their connections are defined in XML
● Separates data model from visualisation modules
● based on Model-View-Controller paradigm
![Page 9: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/9.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPulse
● Data model: meta-data (name, type, unit, range) and interface to the data source.
● Controller: preprocessing, analysis components
● View: displaying components
![Page 10: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/10.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load
![Page 11: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/11.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load
![Page 12: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/12.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesExample: CPU Load
![Page 13: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/13.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesData model
● Data channel
– Name
– Data type
– Unit
– Range of valid data
LoadChannelMetaData(
name=”load1”,
type=float,
unit=””,
range=[0.0-100.0+0.01] )
![Page 14: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/14.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesInput components
● Input component (sensor): an interface to some kind of external data source (file, service, an executable, etc.)
● A sensor– Reads data from the external source– Translates data into a channel based representation– Publishes complex information in several channels– Provides one piece of information at a time
● Buffered sensor– Buffering capability to a sensor
![Page 15: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/15.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPresentation components
● To show the results to the user
● Tabular view, Histogram plotter
● Presentation components subscribe to the sensor's notification service ->they are notified about the status changes in the sensor ->display updates are controlled by the sensors (incoming data)
![Page 16: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/16.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesAnalysis components
● Essentially:
– Take data from other components (sensors)
– Do something with the data
– Forward it to other components (plotter)
● E.g.
– Channel selector (for histogram plotter)
– Merge data from several sensors
– Compute statistical values on data
![Page 17: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/17.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesConfiguration of Pulse
● XML configuration to define
– Components
– Connections between the components
![Page 18: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/18.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesPulse: conclusion
● Pulse can be used to build an analysis chain to process dynamic information and present results in a graphical form
● Flexible data representation
● Already it can be used as a tool for the R-GMA grid monitoring and information system
![Page 19: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/19.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesApplication monitoring
● Performance monitoring (not job status!) of
● Parallel applications (MPI, PVM, P-GRADE)
● Large volume of trace information
● On-line transfer of trace to the user
![Page 20: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/20.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesGRM at EuroPar’2001
Main MonitorMM
Application Process
Application Process
Site 1
Local Host
Host 1 Host 2 Host 1
Local MonitorLM
Local MonitorLM
Site 2
Local MonitorLM
Site MonitorSM
Site MonitorSM
Appl. Process
Appl. Process
shm shm shm
![Page 21: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/21.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesTools
● GRM– Instrumentation library and trace collector
● Mercury monitor– Transfer trace data from the application processes
to GRM
● R-GMA– Information system for finding the application and
the monitor components
![Page 22: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/22.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesMercury monitor
in the GridLab project
![Page 23: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/23.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesGRM+Mercury+R-GMA
![Page 24: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/24.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesMercury monitor
● To deliver trace data from the application to the user efficiently.
– Uses TCP Socket communication
– Data in XDR format and could be optimised for TCP transmission
– Two sw. hops between application and GRM: local and main monitors
– One hw. hop: host of main monitor
![Page 25: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/25.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA
● To find the application by the user/GRM
– Where is it running? -> machineX.siteY
– What is its global job id? -> GID
● To find the monitor to be connected
– What is the address of Mercury monitor running at siteY?
● To find the application by the monitor
– What processes (PIDs) belong to application GID?
![Page 26: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/26.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesGRM
● Instrumentation library
– Connect and publish trace to Mercury monitor
● GRM and PROVE
– Look for application asking R-GMA
– Connect to Mercury monitor and query for trace
– Visualise trace information with PROVE
![Page 27: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/27.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesCurrent status
![Page 28: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/28.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesFuture work
● Intrusion: replace socket communication between LM and application process for shared-memory buffering (as in original GRM)
● Scalability: data reduction at LMs and MM.
![Page 29: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/29.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesConclusion
● Different monitoring information can be presented with different techniques
– Web browser is used to search among available information and static (or slowly changing) behaviour of the grid
– Pulse can be used to build specific analysis chains to process dynamic information about the services and resources and to create graphical plots
– GRM is a tool for performance monitoring of parallel applications, connected to Mercury and R-GMA to work for grid.
![Page 30: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/30.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of Sciences
Thank you
![Page 31: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/31.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA Schema Browser
![Page 32: Computer and Automation Research Institute Hungarian Academy of Sciences Presentation and Analysis of Grid Performance Data Norbert Podhorszki and Peter](https://reader036.vdocuments.mx/reader036/viewer/2022062718/56649e7e5503460f94b81fd0/html5/thumbnails/32.jpg)
Computer and Automation Research InstituteComputer and Automation Research Institute
Hungarian Academy of SciencesHungarian Academy of SciencesR-GMA Schema Browser