irmis3 status gabriele carcassi 15 oct 2010. irmis3 status this talk is about: what is currently...
TRANSCRIPT
IRMIS3 StatusGabriele Carcassi
15 Oct 2010
IRMIS3 Status
This talk is about:• what is currently downloadable from the website• what is ready for production• what is actively supported
Project infrastructure
Production server at BNL since January 2009
“alpha” - AuthZ and AuthN only designed Likely implemented by the end of the year
Build and test serverAutomated tests
IE or firefox on Windows/Linux/Mac supported
SourceForge project (mailing list, bug tracking, wiki).Monthly release cycle.
IRMIS3 SERVICE ARCHITECTURE
IRMIS3 architecture
IRMIS DB(MySQL)
Data service
Javaapplets
AJAXcomponents
Scripts and CLI(perl, Python, …)
Browser(Firefox, IE, …)
XML protocol
Web server(Glassfish)
Data service enforces business rules(clients can be dumb)
Data service allows db schema orbusiness rule changes while in production
Data service provides a good abstractionlayer on which to integrate
Software stack
XML protocol (REST style WS)
Few
dat
abas
e ut
ilitie
s: b
acku
p,
cons
iste
ncy
chec
k, e
tc…
Clie
ntS
erve
r
Inte
grat
ion
with
ext
erna
l to
ols
(i.e.
phy
scs)
3rd p
arty
Per
l/Pyt
on
scrip
ts
3rd p
arty
Jav
a ap
ps J
ytho
n sc
ripts
Web applications
JavaScript bridgeApplets and
Widgets
Java Client API
Data Service layer
Database layer
Getting data<componentType id="90" description="Oscilloscope: 4 Channel @ 500MS/s; 500MHz BW" name="HP54540A"> <manufacturer id="5" name="Agilent (HP)"/> <formFactor id="5" description="Freestanding"/> <functions> <function id="46" description="CCMS"/> <function id="8" description="Instrument"/> </functions> <requires> <interface id="143" description="GPIB_Slave" relType="control"/> <interface id="5" description="Freestanding" relType="housing"/> <interface id="78" description="120VAC" relType="power"/> </requires> <presents> <interface id="234" description="Port" relType="control"/> </presents> <ports> <port name="Chnl 1"> <portType id="13" name="BNC-F" group="RF Connectors"> <pinDesignator id="303" name="1"/> </portType> <pin usage="Chnl 1" pinDesignatorId="303"> <signalType id="1" direction="IN"/> </pin> </port> <port name="Chnl 2">...</port> <port name="Chnl 3">...</port> <port name="Chnl 4">...</port> <port name="RS232">...</port> ... </ports></componentType>
This is an example of what the data service might deliver
XML returnedREST style (data service)
Id references
De-normalized data
Writing data<transaction xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns='http://xml.bnl.gov/schema/irmis' xsi:schemaLocation='http://xml.bnl.gov/schema/irmis irmis.xsd'> <create> <manufacturer name="Acopian"/> </create> <update> <componentType id="2" name="Site" description="Overall facility. Root of housing hierarchy."> <manufacturer id="1" name="None"/> <formFactor id="1" description="Virtual"/> <properties/> <functions/> <requires/> <presents/> <ports/> </componentType> </update> <delete> <formFactor id="4" description="IndustryPack"/> </delete> <connect> <cable color="blue" label="AA" portAId="123" portBId="321"/> <conductor pinAId="123" pinBId="321"/> <conductor pinAId="124" pinBId="322"/> </connect> </transaction>
XML describing a transaction sent through a POST
allows remote/disconnected work
no lock “FOR UPDATE”
idempotent actions (error or same result)
different independent “commands”
Implementation
MySQLJPA
JAXB
SAXBusiness
logic
Servlet
Read
Write
JAVA SE 6JAVA EE 5GLASSFISH 2.X/3.X
Stream out XML as soon as possibleto reduce delay
JAXB to convert transaction requestto Java objects
JPA vs JDBC: simpler queries to maintain6% performance loss
Performance
Dell Precision M4300 laptop, Intel Core Duo T7500, 2.20 GHz - APS dataset• Entire component data set streamed in about 5 seconds (31338
components, 63911 relationships)• Most other queries under 100 ms (manufacturers 88ms, interfaces
95ms)
Roughly half of the time is spent in DB access and half in XML generation• Streaming starts right after DB access
Only tuning done is on the database query (was “good enough”)• Other areas: hardware/os tuning, xml generation improvements,
profiling, caching, …
IRMIS3 CLIENT API
Client API
To access the service, XML and HTTP is all you need• Available in all languages
Two libraries provided that allow to work with objects• PHP API from MSU• Java API from BNL
Only going to talk about the Java API
Java API: design principles
The Java API implement the XML protocol and provides objects that represent the data
All reads are cached: no need to keep caches in your code.
All reads are synchronized• you can have reads on multiple threads on the same data and
will not trigger multiple reads on the server (extremely useful for GUIs)
Java API design principles
The read access is object oriented• Data is exposed through interfaces
- Component, ComponentType, Manufacturer
• Some classes function as “finders” for the classes containing the data- Components.getById(123), FormFactors.getByName(“Rack”)
• You can follow references from one object to the other- component.getComponentType().getManufacturer()
Java API design principles
The write access is protocol oriented• You create a transaction• You call methods to add elements in the transaction
- Manufacturers.createManufacturer(transaction, “ACME”)
• You save the transaction, which may fail• The read access is refreshed only after the data is saved
Importing components from Excel (using Jython)
from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *
workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)
nRows = sheet.getRows()nColumns = sheet.getColumns()
irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))
transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"
Importing from Excel (Jython)
from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *
workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)
nRows = sheet.getRows()nColumns = sheet.getColumns()
irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))
transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"
Load excel file
Importing from Excel (Jython)
from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *
workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)
nRows = sheet.getRows()nColumns = sheet.getColumns()
irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))
transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"
Connect to the service
Importing from Excel (Jython)
from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *
workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)
nRows = sheet.getRows()nColumns = sheet.getColumns()
irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))
transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"
Loop over the rows
Importing from Excel (Jython)
from gov.bnl.irmis.api import *from java.net import URIfrom java.io import Filefrom jxl import *
workbook = Workbook.getWorkbook(File(“components.xls"))sheet = workbook.getSheet(0)
nRows = sheet.getRows()nColumns = sheet.getColumns()
irmisDB = IrmisDB.getInstance();irmisDB.setURI(URI("http://cs-build.nsls2.bnl.gov:8080/IRMISDataService/data"))
transaction = irmisDB.createTransaction();for row in range(1, nRows): componentTypeName = sheet.getCell(0, row).getContents(); componentType = ComponentTypes.getByName(componentTypeName); serialNumber = sheet.getCell(1, row).getContents(); Components.createComponent(transaction, componentType, serialNumber)transaction.save();print "Operation successful"
Get data outCreate components
Fluent API
Navigating through hierarchies can be tedious On top of the “beans” API there is a fluent API
(internal domain specific language)
Search API
Builders
Change Set API
IRMIS3 USER INTERFACE
UI design
Web interface: uses applets and Web Start (both greatly improved with Java 6u10)
Uses Swing standard components plus open source (SwingX, Flamingo, JIDE, …)
UI elements can optionally be used in custom pages and integrated with javascript
UI available only for Components and Cabling
UI elements
Component hierarchy tree widget (filtering, sorting, search, …)
Component pathwidget
Component applications
Cabling applications
STATUS BY AREA
Components
Support for component and component types is production level• Data model has been tested extensively through previous use at
other sites• Performance tests were done using APS datasets
Most of the work now is in UI development• Most applications at their 3rd iteration
Cabling
Support for cabling is still beta• No considerable amount of data has been put into it• No performance tests done (see lack of data)
UI at their second iteration Need users!
Lattice
There is preliminary support for lattice• Only through the API, no UI
A separate tree structure of “elements”, each of an “element type”. Each relationship has alignment information.
Lattice cannot be modifiedin place: copy on write createsa new version
Right now, provided as-is1
1
1
1
1
2
2
PV
PV service and API is considered alpha. No UI yet. No crawler scripts provided yet.