irmis3 status gabriele carcassi 15 oct 2010. irmis3 status this talk is about: what is currently...

IRMIS3 StatusGabriele Carcassi

15 Oct 2010

IRMIS3 Status

This talk is about:• what is currently downloadable from the website• what is ready for production• what is actively supported

Project infrastructure

Production server at BNL since January 2009

“alpha” - AuthZ and AuthN only designed Likely implemented by the end of the year

Build and test serverAutomated tests

IE or firefox on Windows/Linux/Mac supported

SourceForge project (mailing list, bug tracking, wiki).Monthly release cycle.

IRMIS3 SERVICE ARCHITECTURE

IRMIS3 architecture

IRMIS DB(MySQL)

Data service

Javaapplets

AJAXcomponents

Scripts and CLI(perl, Python, …)

Browser(Firefox, IE, …)

XML protocol

Web server(Glassfish)

Data service enforces business rules(clients can be dumb)

Data service allows db schema orbusiness rule changes while in production

Data service provides a good abstractionlayer on which to integrate

Software stack

XML protocol (REST style WS)

Few

dat

abas

e ut

ilitie

s: b

acku

p,

cons

iste

ncy

chec

k, e

tc…

Clie

ntS

erve

r

Inte

grat

ion

with

ext

erna

l to

ols

(i.e.

phy

scs)

3rd p

arty

Per

l/Pyt

on

scrip

ts

3rd p

arty

Jav

a ap

ps J

ytho

n sc

ripts

Web applications

JavaScript bridgeApplets and

Widgets

Java Client API

Data Service layer

Database layer

Getting data<componentType id="90" description="Oscilloscope: 4 Channel @ 500MS/s; 500MHz BW" name="HP54540A"> <manufacturer id="5" name="Agilent (HP)"/> <formFactor id="5" description="Freestanding"/> <functions> <function id="46" description="CCMS"/> <function id="8" description="Instrument"/> </functions> <requires> <interface id="143" description="GPIB_Slave" relType="control"/> <interface id="5" description="Freestanding" relType="housing"/> <interface id="78" description="120VAC" relType="power"/> </requires> <presents> <interface id="234" description="Port" relType="control"/> </presents> <ports> <port name="Chnl 1"> <portType id="13" name="BNC-F" group="RF Connectors"> <pinDesignator id="303" name="1"/> </portType> <pin usage="Chnl 1" pinDesignatorId="303"> <signalType id="1" direction="IN"/> </pin> </port> <port name="Chnl 2">...</port> <port name="Chnl 3">...</port> <port name="Chnl 4">...</port> <port name="RS232">...</port> ... </ports></componentType>

This is an example of what the data service might deliver

XML returnedREST style (data service)

Id references

De-normalized data

Writing data<transaction xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='http://xml.bnl.gov/schema/irmis' xsi:schemaLocation='http://xml.bnl.gov/schema/irmis irmis.xsd'> <create> <manufacturer name="Acopian"/> </create> <update> <componentType id="2" name="Site" description="Overall facility. Root of housing hierarchy."> <manufacturer id="1" name="None"/> <formFactor id="1" description="Virtual"/> <properties/> <functions/> <requires/> <presents/> <ports/> </componentType> </update> <delete> <formFactor id="4" description="IndustryPack"/> </delete> <connect> <cable color="blue" label="AA" portAId="123" portBId="321"/> <conductor pinAId="123" pinBId="321"/> <conductor pinAId="124" pinBId="322"/> </connect> </transaction>

XML describing a transaction sent through a POST

allows remote/disconnected work

no lock “FOR UPDATE”

idempotent actions (error or same result)

different independent “commands”

Implementation

MySQLJPA

JAXB

SAXBusiness

logic

Servlet

Read

Write

JAVA SE 6JAVA EE 5GLASSFISH 2.X/3.X

Stream out XML as soon as possibleto reduce delay

JAXB to convert transaction requestto Java objects

JPA vs JDBC: simpler queries to maintain6% performance loss

Performance

Dell Precision M4300 laptop, Intel Core Duo T7500, 2.20 GHz - APS dataset• Entire component data set streamed in about 5 seconds (31338

components, 63911 relationships)• Most other queries under 100 ms (manufacturers 88ms, interfaces

95ms)

Roughly half of the time is spent in DB access and half in XML generation• Streaming starts right after DB access

Only tuning done is on the database query (was “good enough”)• Other areas: hardware/os tuning, xml generation improvements,

profiling, caching, …

IRMIS3 CLIENT API

Client API

To access the service, XML and HTTP is all you need• Available in all languages

Two libraries provided that allow to work with objects• PHP API from MSU• Java API from BNL

Only going to talk about the Java API

Java API: design principles

The Java API implement the XML protocol and provides objects that represent the data

All reads are cached: no need to keep caches in your code.

All reads are synchronized• you can have reads on multiple threads on the same data and

will not trigger multiple reads on the server (extremely useful for GUIs)

Java API design principles

The read access is object oriented• Data is exposed through interfaces

- Component, ComponentType, Manufacturer

• Some classes function as “finders” for the classes containing the data- Components.getById(123), FormFactors.getByName(“Rack”)

• You can follow references from one object to the other- component.getComponentType().getManufacturer()

Java API design principles

The write access is protocol oriented• You create a transaction• You call methods to add elements in the transaction

- Manufacturers.createManufacturer(transaction, “ACME”)

• You save the transaction, which may fail• The read access is refreshed only after the data is saved

Importing components from Excel (using Jython)

from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *

workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)

nRows = sheet.getRows()nColumns = sheet.getColumns()

irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))

transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"

Importing from Excel (Jython)

from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *

workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)

nRows = sheet.getRows()nColumns = sheet.getColumns()

irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))

transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"

Load excel file

Importing from Excel (Jython)

from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *

workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)

nRows = sheet.getRows()nColumns = sheet.getColumns()

irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))

transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"

Connect to the service

Importing from Excel (Jython)

from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *

workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)

nRows = sheet.getRows()nColumns = sheet.getColumns()

irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))

transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"

Loop over the rows

Importing from Excel (Jython)

from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *

workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)

nRows = sheet.getRows()nColumns = sheet.getColumns()

irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))

transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"

Get data outCreate components

Fluent API

Navigating through hierarchies can be tedious On top of the “beans” API there is a fluent API

(internal domain specific language)

Search API

Builders

Change Set API

IRMIS3 USER INTERFACE

UI design

Web interface: uses applets and Web Start (both greatly improved with Java 6u10)

Uses Swing standard components plus open source (SwingX, Flamingo, JIDE, …)

UI elements can optionally be used in custom pages and integrated with javascript

UI available only for Components and Cabling

UI elements

Component hierarchy tree widget (filtering, sorting, search, …)

Component pathwidget

Component applications

Cabling applications

STATUS BY AREA

Components

Support for component and component types is production level• Data model has been tested extensively through previous use at

other sites• Performance tests were done using APS datasets

Most of the work now is in UI development• Most applications at their 3rd iteration

Cabling

Support for cabling is still beta• No considerable amount of data has been put into it• No performance tests done (see lack of data)

UI at their second iteration Need users!

Lattice

There is preliminary support for lattice• Only through the API, no UI

A separate tree structure of “elements”, each of an “element type”. Each relationship has alignment information.

Lattice cannot be modifiedin place: copy on write createsa new version

Right now, provided as-is1

1

1

1

1

2

2

PV

PV service and API is considered alpha. No UI yet. No crawler scripts provided yet.