e-infrastructure shared between europe and latin america usage of r-gma and grid application...

E-infrastructure shared between Europe and Latin America

Usage of R-GMAand

Grid Application Monitoring

Valeria Ardizzone

INFN-Catania

1° EELA Grid School

Itacuruçà Island,

State of Rio De Janeiro, Brazil


Outline

• Usage of R-GMA command line interface

• Usage of R-GMA Web Browser like a Consumer

• Usage of R-GMA cli tool into jobs

• Usage of R-GMA API to create Producer and Consumer

• Usage of Producer and Consumer within application

• References


R-GMA command line tool


• Before you start the R-GMA command line tool make sure you have a proxy certificate :

• Run the command and you should receive the following message on startup:

R-GMA Command Line Tool

> voms-proxy-init --voms gilda

Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN/CN=Valeria Ardizzone/[email protected]

Enter GRID pass phrase:

Creating temporary proxy ............................................................ Done

Contacting voms.ct.infn.it:15001[/C=IT/O=GILDA/OU=Host/L=INFN

Catania/CN=voms.ct.infn.it/[email protected]] "gilda" Done

Creating proxy .................................... Done

Your proxy is valid until Fri Jun 16 08:20:14 2006

> rgma

Welcome to the R-GMA virtual database for Virtual Organisations.

================================================================

Your local R-GMA server is:

https://egris198.eela.ufrj.br:8443/R-GMA

You are connected to the following R-GMA Registry services:

https://egris198.eela.ufrj.br:8443/R-GMA/RegistryServlet

You are connected to the following R-GMA Schema service:

https://egris198.eela.ufrj.br:8443/R-GMA/SchemaServlet

Type "help" for a list of commands.

rgma>


• Commands are entered by typing at the > rgma

prompt and hitting ‘enter’ to execute the command.

• A history of the commands executed can be accessed

using the Up and Down arrow keys.

• To search a command from history use CTRL-R and

type the first few letters of the command to recall.

• Command autocompletion is supported (use Tab

when you have partly entered a command).

Entering Command


General Commands

• help and help <command> Display general help information or specific for a command.

• exit or qExit from R-GMA command line interface.

• show …...

• set …...

• clear historyClear the current session history of commands executed.

• write history <filename> Write the session command history to a file.

• write results <filename>Write query results to a file.

• read <filename>Show all information about the structure of a table


Batch mode

The command line tool can be used in batch mode in three ways:

• rgma –c <command> [ -c <command> …]

Executes the command and exit. • rgma –f <filename>

Executes commands in file sequentially and exit. • Commands embedded in a shell script:

#!/bin/sh

$RGMA_HOME/bin/rgma <<EOF

set query latest

select Service_Name, Status FROM ServiceStatus WHERE Status != 0

EOF


Table Commands

• Show tables

Display the name of all tables existing in the Schema• Describe <tablename>

Show all information about the structure of a table• create table

Create a table in the R-GMA virtual database schema.• drop table <tablename>

Delete a table in the R-GMA virtual database schema.


Exercise 1: create a table

rgma> create TABLE ITATutTable(COD_Test INT PRIMARY KEY, Application VARCHAR(20), Status VARCHAR(10), PercStatus INT, Owner VARCHAR(25))

rgma> show tables

rgma> describe ITATutTable

…….| NetworkUDPPacketLoss | | NetworkFileTransferThroughput | | ITATutTable |

+-----------------+------------+-------------------------+----------------+------------------+| Column name | Type | Primary key | Can be NULL | +-----------------+-------------+------------------------+----------------+------------------+| COD_Test | INTEGER | Yes | No | | Application | VARCHAR(20) | No | Yes | | Status | VARCHAR(10) | No | Yes | | PercStatus | INTEGER | No | Yes | | Owner | VARCHAR(25) | No | Yes | | MeasurementDate | DATE | No | No | | MeasurementTime | TIME | No | No | +-----------------+------------+-------------------------+----------------+------------------+


Producer Properties

• Using the command line tool you may work with one producer at a time.

• The current producer type can be displayed using:rgma>show producer

• Set the latest retention period for tuples published by the producer

rgma>set producer latestretentionperiod|lrp <time value> [<units>]

• Set the history retention period for the producer. If the producer does not support history queries this command has no effect.

rgma>set producer historyretentionperiod|hrp <time value> [<units>]

• Producer which handles the INSERT statement. The SQL INSERT statement may be used to add data to the system:

rgma> INSERT INTO <Tablename> VALUES (’a’, ’b’, ’c’, ’d’)


Exercise 2: create a Producer

rgma> set producer latestProducer type : continuous latest

rgma> set producer latestretentionperiod 50 secondsSet producer LRP to 50 seconds

rgma> set producer historyretentionperiod 2 minutesSet producer HRP to 2 minutes

rgma>describe ITATutTable

rgma> insert INTO ITATutTable values(001,'TestProducer','Start',10,'Valeria');

Inserted 1 row into ITATutTable


Consumer Properties (1)

The behaviour of Consumer varies according to the type of query being executed. In R-GMA there are three basic types of query:

LATEST Queries: only the most recent tuple for each primary key

HISTORY Queries: all historical tuples for each primary key

CONTINUOUS Queries: returns tuples continuously as they are inserted.

• The type of query can be changed using the SET QUERY command as follow:

rgma> SET QUERY LATEST | CONTINUOUS | HISTORY

• The current query type can be displayed using

rgma> SHOW QUERY



• The maximum age of tuples to return can also be controlled. To limit the age of latest or historical tuples use the MAXAGE property.

rgma> SET MAXAGE <time> seconds|minutes|hours|days

• The current maximum tuple age can be displayed using

rgma> SHOW MAXAGE

• To disable the maximum age, set it to none:

rgma> SET MAXAGE none


• The final property affecting queries is timeout.

– For a latest or history query the timeout exists to prevent a problem (e.g. network failure) from stopping the query from completing.

– For a continuous query, timeout indicates how long the query will continue to return new tuples. Default timeout is 1 minute and it can be changed using

rgma>SET TIMEOUT <time> seconds|minutes|hours|days

• The current timeout can be displayed using

rgma>SHOW TIMEOUT



• Querying data uses the standard SQL SELECT statement:

rgma> SELECT * FROM <table_name>

• Set the output format for results. 'table' formats the results in a table, tsv outputs tab-separated results and csv outputs comma-separated results:

rgma> SET output table|tsv|csv


output csv:

1,TestProducer,Start,10,Valeria,2006-12-07,22:07:36,

set output tsv:1 TestProducer Start 10 Valeria 2006-12-07 22:07:36


Exercise 3: create a Consumer

rgma> set query latestSet query type to latest

rgma> set maxage 1 minutesSet max age to 1 minutes

rgma> set timeout 50 secondsSet timeout to 50 seconds

rgma> set output tsvSet output format to 'tsv‘

rgma>select Application,Status FROM ITATutTable

TestProducer Start

TestProducer Step1

TestProducer Step2


Exercise 4: Producer & Consumer

Continuos Producer and Consumer: (NOTE. Open 2 rgma client tool, one for Consumer the other for Producer.)

• Consumer’s client:rgma> set query continuous rgma> set timeout 50 secondsrgma> set maxage 30rgma> set output csvrgma> select * from ITATutTable

rgma> • Producer’s client:rgma> set producer continuous rgma> insert INTO ITATutTable values(004,'TestProducer','Step3',30,'Valeria');rgma> insert INTO ITATutTable values(005,'TestProducer','Step4',40,'Valeria');rgma> insert INTO ITATutTable values(006,'TestProducer','Step5',50,'Valeria');rgma> insert INTO ITATutTable values(007,'TestProducer','Step6',60,'Valeria');rgma> write history Prod_comm.rgma



Continuos Producer and Consumer:

(NOTE. Open 1 rgma client tool only for Consumer. Open one shell for Producer.)

• Consumer’s client:

rgma> set query continuous

rgma> set timeout 60 seconds

rgma> set output csv

rgma> select * from ITATutTable

• Producer’s shell:rgma -c “set producer continuous”

rgma -c “insert INTO ITATutTable values(004,‘A','Step1',100,'Valeria');”

rgma -c “insert INTO ITATutTable values(005,‘B','Step2',100,'Valeria');”



Continuos Producer and Consumer:

(NOTE. Open 1 rgma client tool only for Consumer. Open one shell for Producer.)

• Consumer’s client:


rgma> set timeout 60 seconds

rgma> write results Results.rgma


• Producer’s shell:

rgma -f Prod_comm.rgma


R-GMA Web Browser



12

3


1

2

5

4

3



12

4

3


Jobs using R-GMA commands


script.sh

#!/bin/sh

echo "I'm doing application A...now I finished."


insert INTO ITATutTable values(001,'A','Step1',100,'Valeria');

EOF

echo "I'm doing application B...now I finished."


insert INTO ITATutTable values(002,'B','Step2',100,'Valeria');

EOF

echo "Bye bye"


my.JDL file

Type = "Job";

JobType = "Normal";

Executable=“/bin/sh";

Arguments = “script.sh";

StdOutput="stdout.log";

StdError="stderr.log";

InputSandbox={“script.sh"};

OutputSandbox={"stdout.log","stderr.log"};


Exercise 7

>edg-job-list-match myJDL.jdl

>edg-job-submit –o jobID myJDL.jdl

>edg-job-status –i jobID

>edg-job-get-output –i jobID

Continuos Consumer Properties:

>rgma


rgma> show query

rgma> set timeout 3 minutes



R-GMA API to create Producer and Consumer


User Producer and Consumer

API available for Java, C, C++ and Python

Users may by-pass API if they wish, but API is the easiest way to use R-GMA services


Use Case for Monitoring Application

• Ingredients:• An application, called A, that does something and

produce data;• A producer object used by A application to publish its

relevant data;• A script to prepare the environment and launch the A

application;• A JDL;• A rgma client or R-GMA Web Browser to act like a

Consumer.


My Producer (Java): PP (1)

. . . . . . . . . .

ProducerProperties props = null;

if (producerType.equals("CONTINUOUS"))

{ props = new ProducerProperties(Storage.MEMORY, 0); }

else if (producerType.equals("LATEST"))

{ props = new ProducerProperties(Storage.DATABASE, ProducerProperties.LATEST); }

else if (producerType.equals("HISTORY"))

{ props = new ProducerProperties(Storage.DATABASE, ProducerProperties.HISTORY); }

else

{ System.err.println("Invalid producer type (" + producerType + ").");

System.exit(1);

}

Producer Properties

Type: PRIMARY

Storage type: MEMORY

Termination Interval: 20 (minutes)

Query type: CONTINUOUS

Latest Retention Period: 60 (minutes)

History Retention Period: 60 (minutes)


. . . . . . . . . .

PrimaryProducer pp = null;

ResourceEndpoint endpoint = null;

Try

{ ProducerFactory factory = new ProducerFactoryStub();

TimeInterval ti = new TimeInterval(20, Units.MINUTES);

pp = factory.createPrimaryProducer(ti, props);

String predicate = "WHERE Owner = '" + owner + "'";

TimeInterval historyRP = new TimeInterval(60, Units.MINUTES);

TimeInterval latestRP = new TimeInterval(60, Units. MINUTES);

pp.declareTable(tableName, predicate,historyRP,latestRP);

...

My Producer (Java): PP (2)


My Producer (Java)(3)

. . . . . . . . . .

public void insert(String status, int percstat){

String insert = "INSERT INTO ITATutTable (COD_Test, Application, Status, PercStatus,Owner) VALUES (" + cod +",'" + appname +"','" + status + "'," + percstat + ",'" + owner + "')";

try {

pp.insert(insert);

} catch

. . . . . . . . . .public void close()

{

try {

pp.close();

}


Application: A

public class A { public static void main(String[] args){

long startTime = System.currentTimeMillis() / 1000;

if (args.length != 3) { System.err.println("Usage: java A <Cod_Test> <Owner> <producerType>"); System.exit(1); }

PP mytest = new PP(new Integer(args[0]).intValue(),args[1],args[2]); for( int i = 0, j = 0; i <= 10; i++, j+=10) mytest.insert("Step"+i,j);

mytest.close();

long endTime= System.currentTimeMillis() / 1000; System.out.println("resourceId=" + mytest.getResourceID() ); System.out.println("startTime=" + startTime); System.out.println("endTime=" + (System.currentTimeMillis() / 1000)); }}


Script: startAPP.sh (1)

if [ -z "$RGMA_HOME" ]

then

export RGMA_HOME=$EDG_LOCATION

fi

export LD_LIBRARY_PATH=$RGMA_HOME/lib:$EDG_LOCATION/externals/lib

#Java Environment

if [ "$X509_USER_PROXY" != "" ]

then

securityProp="-DX509_USER_PROXY=$X509_USER_PROXY"

else

securityProp="-DTRUSTFILE=$TRUSTFILE"

fi

CLASSPATH="$CLASSPATH:$RGMA_HOME/share/java/glite-rgma-api-java.jar:$RGMA_HOME/share/java/glite-rgma-stubs-servlet-java.jar:$GLITE_LOCATION/share/java/glite-security-trustmanager.jar:$GLITE_LOCATION/share/java/glite-security-util-java.jar:$GLITE_LOCATION/externals/share/java/bcprov-jdk14-122.jar:$GLITE_LOCATION/externals/share/java/log4j.jar:."

export CLASSPATH


Script: startAPP.sh (2)

COMM_EXE=A.class

chmod +x $COMM_EXE

# Arguments

cod=$1

owner=$2

queryType="$3"

if [ "$JAVA_HOME" != "" ]

then

COMM_JAVA=$JAVA_HOME/bin/java

else

COMM_JAVA=java

fi

$COMM_JAVA -classpath $CLASSPATH:. -DRGMA_HOME=$RGMA_HOME $securityProp A $cod $owner $queryType

if [ $? -ne 0 ]

then

echo "Program $cod reported failure." 1>&2

exit 1

fi


JDL with User Producer Application

Type = "Job";

JobType = "Normal";

Executable=“/bin/sh";

Arguments = "startAPP.sh 1001 Valeria CONTINUOUS";

StdOutput="stdout.log";

StdError="stderr.log";

InputSandbox={"startAPP.sh","pp.class“,”A.class”};

OutputSandbox={"stdout.log","stderr.log"};


More information

• R-GMA overview page.

http://www.r-gma.org

• R-GMA documentation in EGEE

http://hepunx.rl.ac.uk/egee/jra1-uk/

• R-GMA in E-GRIS Sitehttps://egris198.eela.ufrj.br/R-GMA

• R-GMA API in E-GRIS Wiki Site


Questions…

e-infrastructure shared between europe and latin america usage of r-gma and grid application...

Documents