computer and automation research institute hungarian academy of sciences sztaki’s work in datagrid...

Computer and Automation Research InstituteComputer and Automation Research Institute

Hungarian Academy of SciencesHungarian Academy of Sciences

SZTAKI’s work in DataGrid WP3SZTAKI’s work in DataGrid WP32002 September2002 September

Norbert Podhorszki<[email protected]>

Laboratory of Parallel and Distributed Systems

MTA SZTAKI

www.lpds.sztaki.hu



2 September 2002, SZTAKI report DataGrid WP3 2

ContentsContents

GRM and PROVE

Pulse




GRM and PROVEGRM and PROVE

• Monitoring and visualisation of parallel programs

• Goal: Connect GRM to R-GMA– Modify instrumentation library to act as a Producer

– Modify Main Monitor of GRM to act as a Consumer

– Prove can be untouched

– C API to R-GMA is needed




GRM Main

Site

User’s Host

Host 1 Host 2

ApplicationProcess

Appl.Process

Appl.Process

R-GMA

PROVE

Connection to R-GMAConnection to R-GMA




R-GMA R-GMA

Sensor Code

ProducerAPI

Application Code

ConsumerAPI

ProducerServlet

RegistryAPI

Registry Servlet

SchemaAPI

Schema Servlet

“Event Dictionary”

Consumer Servlet

RegistryAPI

Main Monitor

Instrumented code




GRM – RGMA How to useGRM – RGMA How to use• Instrument and compile your application with GRM.

• grm_instr.h instrumentation header file

• libgrmon.a library to be linked to the application

• grm_Start should give a name (e.g. MyApp) unique for the whole grid

• Submit/execute the application

• Start GRM main monitor giving the unique application name

• grm –o trace_file MyApp

• Main monitor prints its listening socket port address (e.g. 6100)

• After execution start PROVE and read trace file




GRM Instrumentation libraryGRM Instrumentation library• RGMA C API

• Two CircularBufferProducer to publish trace data

• Two schemas (relational tables)

• GRMHeader (String appName, String event)

• appName: unique identifier for this application

• event: GRM event string as is

• contains important header events (START)

• for P-GRADE also: GROUP, CHANNEL, TEMPLATE etc.

• small buffer in servlet (10 elements)

• GRMTrace (String appName, String event)

• all other trace events (BEGIN-END, SEND-RECEIVE etc.)

• local buffer (100) + large remote buffer (1000)




GRM Main monitorGRM Main monitor• RGMA C API

• First, look for application: create a Consumer with select statement

• “SELECT * FROM GRMHeader WHERE appName = “...”

• events flushed into trace file

• R-GMA gives back all producers that produces such data.

• If found then create a streaming Consumer for trace data

• “SELECT * FROM GRMTrace WHERE appName = “...”

• remote buffer in ConsumerServlet for streaming (100)




Buffers in trace Buffers in trace deliverydelivery

Application process

Producer API Local Buffer

Application process


ProducerServlet

PServlet API Remote Buffer

ConsumerServlet

CServlet API Remote Buffer

GRM – Consumer

Consumer API

ProducerServlet


Trace file




Small PVM program workedSmall PVM program worked




GRM – RGMA Problems: GRMGRM – RGMA Problems: GRM• CircularBufferProducer can loose data (overwrite buffer before it is read)

• PROVE should be ready for this!

• Buffer sizes can help a bit. In theory, no limit on buffer sizes in R-GMA

• Could RGMA report to Consumer if there was an overwrite?

• GRM main monitor

• exit and restart later?

• Tried only with a single, simple application. Parallel programs?

• Tried to use with P-GRADE but could not. Apps aborted always.

• Simple PVM program worked.

• Nested blocks are not allowed in PROVE (from P-GRADE) but might be a natural requirement for general monitoring.




GRM – RGMA Problems: RGMAGRM – RGMA Problems: RGMA• Application exits

Producer exits ProducerServlet exits, too

all data not delivered to ConsumerServlet are

destroyed

• what about GRRP connection between servlets to keep all data in stream.

• Timeout• application process uses local buffer to send events in bucks.• no “near on-line” behaviour if events are generated rarely.• timeout is defined but not implemented in RGMA C api

• Throughput performance?• is the “quick-fixed bug in libwww” in the C api correctly solved?• measurements needed




GRM – RGMA improvementsGRM – RGMA improvements• Streaming

• single event per pop operation, i.e. communication between Consumer and ConsumerServlet

• One pop for the whole buffer in ConsumerServlet could improve performance

• Idea 1: when streaming, cons.execute() could give back the content of the buffer. This does not hurt the api.

• Idea 2: introduce new function: popAll()

• What about the streaming between the servlets? One-by-one or in blocks.




Improvement?Improvement?

Application process


Application process


ProducerServlet


ConsumerServlet


GRM – Consumer

Consumer API

ProducerServlet


Trace file

ConsumerServlet


Create a ProducerServletConnection for each process anddo not use Mediator?




Improvement?Improvement?

Application process


Application process


ProducerServlet


GRM – Consumer

Consumer API + something

ProducerServlet


Trace file

Connect to ProducerServlets directly?Needs to write a consumer which implements features of ConsumerServlet.

This is not clean RGMA anymore!!!




PulsePulse




PulsePulse

• Analysis and visualisation tool for

• monitoring data of resources and services and applications.

• Java code

• data source components preprocessing components

view components their connections are defined in XML




PulsePulse

• Separates data model from visualisation modules

• based on Model-View-Controller paradigm

• Data model: meta-data (name, type, unit, range) and interface to the data source.

• Controller: preprocessing, analysis components

• View: displaying components




PulsePulse




The PulseThe Pulse




PulsePulse




R-GMA Schema BrowserR-GMA Schema Browser• Basic connection to R-GMA

– RGMASimpleConsumer sensor component in Pulsewhich maps R-GMA data model to the data model of Pulse

– Both single query and streaming supported

• Browser

– get list of schemas (tables) from SchemaServlet

– simple streaming query on a selected table

• (select * from table)

– compose query based on selected table(s)

• Only tabular view currently.




R-GMA Schema BrowserR-GMA Schema Browser




R-GMA Simple QueryR-GMA Simple Query




Pulse: FuturePulse: Future

• More visualisation modules for different kinds of monitoring data

• Analysis: modules for data analysis can be included in Pulse

• Interactivity in Pulse




Automatic Search for BottlenecksAutomatic Search for Bottlenecks• APART project outcome: Apart Specification Language

• ASL in Java: JavaPSL from T. Fahringer, TU Vienna

• Talked about to use JavaPSL library and performance bottleneck search engine in Pulse.

• Need

– to define performance bottlenecks for Grid resources

• Theoretical work, see Zs. Németh’s presentation

– to collect necessary data through RGMA,

– to implement in Pulse what is needed to connect all of it.

computer and automation research institute hungarian academy of sciences sztaki’s work in datagrid...

Documents