E-infrastructure shared between Europe and Latin America
Usage of R-GMAand
Grid Application Monitoring
Valeria Ardizzone
INFN-Catania
1° EELA Grid School
Itacuruçà Island,
State of Rio De Janeiro, Brazil
E-infrastructure shared between Europe and Latin America
Outline
• Usage of R-GMA command line interface
• Usage of R-GMA Web Browser like a Consumer
• Usage of R-GMA cli tool into jobs
• Usage of R-GMA API to create Producer and Consumer
• Usage of Producer and Consumer within application
• References
E-infrastructure shared between Europe and Latin America
R-GMA command line tool
E-infrastructure shared between Europe and Latin America
• Before you start the R-GMA command line tool make sure you have a proxy certificate :
• Run the command and you should receive the following message on startup:
R-GMA Command Line Tool
> voms-proxy-init --voms gilda
Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN/CN=Valeria Ardizzone/[email protected]
Enter GRID pass phrase:
Creating temporary proxy ............................................................ Done
Contacting voms.ct.infn.it:15001[/C=IT/O=GILDA/OU=Host/L=INFN
Catania/CN=voms.ct.infn.it/[email protected]] "gilda" Done
Creating proxy .................................... Done
Your proxy is valid until Fri Jun 16 08:20:14 2006
> rgma
Welcome to the R-GMA virtual database for Virtual Organisations.
================================================================
Your local R-GMA server is:
https://egris198.eela.ufrj.br:8443/R-GMA
You are connected to the following R-GMA Registry services:
https://egris198.eela.ufrj.br:8443/R-GMA/RegistryServlet
You are connected to the following R-GMA Schema service:
https://egris198.eela.ufrj.br:8443/R-GMA/SchemaServlet
Type "help" for a list of commands.
rgma>
E-infrastructure shared between Europe and Latin America
• Commands are entered by typing at the > rgma
prompt and hitting ‘enter’ to execute the command.
• A history of the commands executed can be accessed
using the Up and Down arrow keys.
• To search a command from history use CTRL-R and
type the first few letters of the command to recall.
• Command autocompletion is supported (use Tab
when you have partly entered a command).
Entering Command
E-infrastructure shared between Europe and Latin America
General Commands
• help and help <command> Display general help information or specific for a command.
• exit or qExit from R-GMA command line interface.
• show …...
• set …...
• clear historyClear the current session history of commands executed.
• write history <filename> Write the session command history to a file.
• write results <filename>Write query results to a file.
• read <filename>Show all information about the structure of a table
E-infrastructure shared between Europe and Latin America
Batch mode
The command line tool can be used in batch mode in three ways:
• rgma –c <command> [ -c <command> …]
Executes the command and exit. • rgma –f <filename>
Executes commands in file sequentially and exit. • Commands embedded in a shell script:
#!/bin/sh
$RGMA_HOME/bin/rgma <<EOF
set query latest
select Service_Name, Status FROM ServiceStatus WHERE Status != 0
EOF
E-infrastructure shared between Europe and Latin America
Table Commands
• Show tables
Display the name of all tables existing in the Schema• Describe <tablename>
Show all information about the structure of a table• create table
Create a table in the R-GMA virtual database schema.• drop table <tablename>
Delete a table in the R-GMA virtual database schema.
E-infrastructure shared between Europe and Latin America
Exercise 1: create a table
rgma> create TABLE ITATutTable(COD_Test INT PRIMARY KEY, Application VARCHAR(20), Status VARCHAR(10), PercStatus INT, Owner VARCHAR(25))
rgma> show tables
rgma> describe ITATutTable
…….| NetworkUDPPacketLoss | | NetworkFileTransferThroughput | | ITATutTable |
+-----------------+------------+-------------------------+----------------+------------------+| Column name | Type | Primary key | Can be NULL | +-----------------+-------------+------------------------+----------------+------------------+| COD_Test | INTEGER | Yes | No | | Application | VARCHAR(20) | No | Yes | | Status | VARCHAR(10) | No | Yes | | PercStatus | INTEGER | No | Yes | | Owner | VARCHAR(25) | No | Yes | | MeasurementDate | DATE | No | No | | MeasurementTime | TIME | No | No | +-----------------+------------+-------------------------+----------------+------------------+
E-infrastructure shared between Europe and Latin America
Producer Properties
• Using the command line tool you may work with one producer at a time.
• The current producer type can be displayed using:rgma>show producer
• Set the latest retention period for tuples published by the producer
rgma>set producer latestretentionperiod|lrp <time value> [<units>]
• Set the history retention period for the producer. If the producer does not support history queries this command has no effect.
rgma>set producer historyretentionperiod|hrp <time value> [<units>]
• Producer which handles the INSERT statement. The SQL INSERT statement may be used to add data to the system:
rgma> INSERT INTO <Tablename> VALUES (’a’, ’b’, ’c’, ’d’)
E-infrastructure shared between Europe and Latin America
Exercise 2: create a Producer
rgma> set producer latestProducer type : continuous latest
rgma> set producer latestretentionperiod 50 secondsSet producer LRP to 50 seconds
rgma> set producer historyretentionperiod 2 minutesSet producer HRP to 2 minutes
rgma>describe ITATutTable
rgma> insert INTO ITATutTable values(001,'TestProducer','Start',10,'Valeria');
Inserted 1 row into ITATutTable
E-infrastructure shared between Europe and Latin America
Consumer Properties (1)
The behaviour of Consumer varies according to the type of query being executed. In R-GMA there are three basic types of query:
LATEST Queries: only the most recent tuple for each primary key
HISTORY Queries: all historical tuples for each primary key
CONTINUOUS Queries: returns tuples continuously as they are inserted.
• The type of query can be changed using the SET QUERY command as follow:
rgma> SET QUERY LATEST | CONTINUOUS | HISTORY
• The current query type can be displayed using
rgma> SHOW QUERY
E-infrastructure shared between Europe and Latin America
Consumer Properties (2)
• The maximum age of tuples to return can also be controlled. To limit the age of latest or historical tuples use the MAXAGE property.
rgma> SET MAXAGE <time> seconds|minutes|hours|days
• The current maximum tuple age can be displayed using
rgma> SHOW MAXAGE
• To disable the maximum age, set it to none:
rgma> SET MAXAGE none
E-infrastructure shared between Europe and Latin America
• The final property affecting queries is timeout.
– For a latest or history query the timeout exists to prevent a problem (e.g. network failure) from stopping the query from completing.
– For a continuous query, timeout indicates how long the query will continue to return new tuples. Default timeout is 1 minute and it can be changed using
rgma>SET TIMEOUT <time> seconds|minutes|hours|days
• The current timeout can be displayed using
rgma>SHOW TIMEOUT
Consumer Properties (3)
E-infrastructure shared between Europe and Latin America
• Querying data uses the standard SQL SELECT statement:
rgma> SELECT * FROM <table_name>
• Set the output format for results. 'table' formats the results in a table, tsv outputs tab-separated results and csv outputs comma-separated results:
rgma> SET output table|tsv|csv
Consumer Properties (4)
output csv:
1,TestProducer,Start,10,Valeria,2006-12-07,22:07:36,
set output tsv:1 TestProducer Start 10 Valeria 2006-12-07 22:07:36
E-infrastructure shared between Europe and Latin America
Exercise 3: create a Consumer
rgma> set query latestSet query type to latest
rgma> set maxage 1 minutesSet max age to 1 minutes
rgma> set timeout 50 secondsSet timeout to 50 seconds
rgma> set output tsvSet output format to 'tsv‘
rgma>select Application,Status FROM ITATutTable
TestProducer Start
TestProducer Step1
TestProducer Step2
E-infrastructure shared between Europe and Latin America
Exercise 4: Producer & Consumer
Continuos Producer and Consumer: (NOTE. Open 2 rgma client tool, one for Consumer the other for Producer.)
• Consumer’s client:rgma> set query continuous rgma> set timeout 50 secondsrgma> set maxage 30rgma> set output csvrgma> select * from ITATutTable
rgma> • Producer’s client:rgma> set producer continuous rgma> insert INTO ITATutTable values(004,'TestProducer','Step3',30,'Valeria');rgma> insert INTO ITATutTable values(005,'TestProducer','Step4',40,'Valeria');rgma> insert INTO ITATutTable values(006,'TestProducer','Step5',50,'Valeria');rgma> insert INTO ITATutTable values(007,'TestProducer','Step6',60,'Valeria');rgma> write history Prod_comm.rgma
E-infrastructure shared between Europe and Latin America
Exercise 5: Producer & Consumer
Continuos Producer and Consumer:
(NOTE. Open 1 rgma client tool only for Consumer. Open one shell for Producer.)
• Consumer’s client:
rgma> set query continuous
rgma> set timeout 60 seconds
rgma> set output csv
rgma> select * from ITATutTable
• Producer’s shell:rgma -c “set producer continuous”
rgma -c “insert INTO ITATutTable values(004,‘A','Step1',100,'Valeria');”
rgma -c “insert INTO ITATutTable values(005,‘B','Step2',100,'Valeria');”
E-infrastructure shared between Europe and Latin America
Exercise 6: Producer & Consumer
Continuos Producer and Consumer:
(NOTE. Open 1 rgma client tool only for Consumer. Open one shell for Producer.)
• Consumer’s client:
rgma> set query continuous
rgma> set timeout 60 seconds
rgma> write results Results.rgma
rgma> select * from ITATutTable
• Producer’s shell:
rgma -f Prod_comm.rgma
E-infrastructure shared between Europe and Latin America
R-GMA Web Browser
E-infrastructure shared between Europe and Latin America
https://egris198.eela.ufrj.br:8443/R-GMA
12
3
E-infrastructure shared between Europe and Latin America
1
2
5
4
3
https://egris198.eela.ufrj.br:8443/R-GMA
E-infrastructure shared between Europe and Latin America
12
4
3
E-infrastructure shared between Europe and Latin America
Jobs using R-GMA commands
E-infrastructure shared between Europe and Latin America
script.sh
#!/bin/sh
echo "I'm doing application A...now I finished."
$RGMA_HOME/bin/rgma <<EOF
insert INTO ITATutTable values(001,'A','Step1',100,'Valeria');
EOF
echo "I'm doing application B...now I finished."
$RGMA_HOME/bin/rgma <<EOF
insert INTO ITATutTable values(002,'B','Step2',100,'Valeria');
EOF
echo "Bye bye"
E-infrastructure shared between Europe and Latin America
my.JDL file
Type = "Job";
JobType = "Normal";
Executable=“/bin/sh";
Arguments = “script.sh";
StdOutput="stdout.log";
StdError="stderr.log";
InputSandbox={“script.sh"};
OutputSandbox={"stdout.log","stderr.log"};
E-infrastructure shared between Europe and Latin America
Exercise 7
>edg-job-list-match myJDL.jdl
>edg-job-submit –o jobID myJDL.jdl
>edg-job-status –i jobID
>edg-job-get-output –i jobID
Continuos Consumer Properties:
>rgma
rgma> set query continuous
rgma> show query
rgma> set timeout 3 minutes
rgma> select * from ITATutTable
E-infrastructure shared between Europe and Latin America
R-GMA API to create Producer and Consumer
E-infrastructure shared between Europe and Latin America
User Producer and Consumer
API available for Java, C, C++ and Python
Users may by-pass API if they wish, but API is the easiest way to use R-GMA services
E-infrastructure shared between Europe and Latin America
Use Case for Monitoring Application
• Ingredients:• An application, called A, that does something and
produce data;• A producer object used by A application to publish its
relevant data;• A script to prepare the environment and launch the A
application;• A JDL;• A rgma client or R-GMA Web Browser to act like a
Consumer.
E-infrastructure shared between Europe and Latin America
My Producer (Java): PP (1)
. . . . . . . . . .
ProducerProperties props = null;
if (producerType.equals("CONTINUOUS"))
{ props = new ProducerProperties(Storage.MEMORY, 0); }
else if (producerType.equals("LATEST"))
{ props = new ProducerProperties(Storage.DATABASE, ProducerProperties.LATEST); }
else if (producerType.equals("HISTORY"))
{ props = new ProducerProperties(Storage.DATABASE, ProducerProperties.HISTORY); }
else
{ System.err.println("Invalid producer type (" + producerType + ").");
System.exit(1);
}
Producer Properties
Type: PRIMARY
Storage type: MEMORY
Termination Interval: 20 (minutes)
Query type: CONTINUOUS
Latest Retention Period: 60 (minutes)
History Retention Period: 60 (minutes)
E-infrastructure shared between Europe and Latin America
. . . . . . . . . .
PrimaryProducer pp = null;
ResourceEndpoint endpoint = null;
Try
{ ProducerFactory factory = new ProducerFactoryStub();
TimeInterval ti = new TimeInterval(20, Units.MINUTES);
pp = factory.createPrimaryProducer(ti, props);
String predicate = "WHERE Owner = '" + owner + "'";
TimeInterval historyRP = new TimeInterval(60, Units.MINUTES);
TimeInterval latestRP = new TimeInterval(60, Units. MINUTES);
pp.declareTable(tableName, predicate,historyRP,latestRP);
...
My Producer (Java): PP (2)
E-infrastructure shared between Europe and Latin America
My Producer (Java)(3)
. . . . . . . . . .
public void insert(String status, int percstat){
String insert = "INSERT INTO ITATutTable (COD_Test, Application, Status, PercStatus,Owner) VALUES (" + cod +",'" + appname +"','" + status + "'," + percstat + ",'" + owner + "')";
try {
pp.insert(insert);
} catch
. . . . . . . . . .public void close()
{
try {
pp.close();
}
E-infrastructure shared between Europe and Latin America
Application: A
public class A { public static void main(String[] args){
long startTime = System.currentTimeMillis() / 1000;
if (args.length != 3) { System.err.println("Usage: java A <Cod_Test> <Owner> <producerType>"); System.exit(1); }
PP mytest = new PP(new Integer(args[0]).intValue(),args[1],args[2]); for( int i = 0, j = 0; i <= 10; i++, j+=10) mytest.insert("Step"+i,j);
mytest.close();
long endTime= System.currentTimeMillis() / 1000; System.out.println("resourceId=" + mytest.getResourceID() ); System.out.println("startTime=" + startTime); System.out.println("endTime=" + (System.currentTimeMillis() / 1000)); }}
E-infrastructure shared between Europe and Latin America
Script: startAPP.sh (1)
if [ -z "$RGMA_HOME" ]
then
export RGMA_HOME=$EDG_LOCATION
fi
export LD_LIBRARY_PATH=$RGMA_HOME/lib:$EDG_LOCATION/externals/lib
#Java Environment
if [ "$X509_USER_PROXY" != "" ]
then
securityProp="-DX509_USER_PROXY=$X509_USER_PROXY"
else
securityProp="-DTRUSTFILE=$TRUSTFILE"
fi
CLASSPATH="$CLASSPATH:$RGMA_HOME/share/java/glite-rgma-api-java.jar:$RGMA_HOME/share/java/glite-rgma-stubs-servlet-java.jar:$GLITE_LOCATION/share/java/glite-security-trustmanager.jar:$GLITE_LOCATION/share/java/glite-security-util-java.jar:$GLITE_LOCATION/externals/share/java/bcprov-jdk14-122.jar:$GLITE_LOCATION/externals/share/java/log4j.jar:."
export CLASSPATH
E-infrastructure shared between Europe and Latin America
Script: startAPP.sh (2)
COMM_EXE=A.class
chmod +x $COMM_EXE
# Arguments
cod=$1
owner=$2
queryType="$3"
if [ "$JAVA_HOME" != "" ]
then
COMM_JAVA=$JAVA_HOME/bin/java
else
COMM_JAVA=java
fi
$COMM_JAVA -classpath $CLASSPATH:. -DRGMA_HOME=$RGMA_HOME $securityProp A $cod $owner $queryType
if [ $? -ne 0 ]
then
echo "Program $cod reported failure." 1>&2
exit 1
fi
E-infrastructure shared between Europe and Latin America
JDL with User Producer Application
Type = "Job";
JobType = "Normal";
Executable=“/bin/sh";
Arguments = "startAPP.sh 1001 Valeria CONTINUOUS";
StdOutput="stdout.log";
StdError="stderr.log";
InputSandbox={"startAPP.sh","pp.class“,”A.class”};
OutputSandbox={"stdout.log","stderr.log"};
E-infrastructure shared between Europe and Latin America
More information
• R-GMA overview page.
http://www.r-gma.org
• R-GMA documentation in EGEE
http://hepunx.rl.ac.uk/egee/jra1-uk/
• R-GMA in E-GRIS Sitehttps://egris198.eela.ufrj.br/R-GMA
• R-GMA API in E-GRIS Wiki Site
E-infrastructure shared between Europe and Latin America
Questions…