the rise of the robots… alasdair allan school of physics, university of exeter alasdair allan...

The rise of the robots…The rise of the robots…Alasdair AllanSchool of Physics, University of Exeter

Alasdair AllanSchool of Physics, University of Exeter

VOTech DSRP, Sorrento, 6 - 9th March

2

eSTAR in a nutshell


3

The eSTAR network

© Nik Szymanek

User Agents

Embedded Agent The VO

GatewayService

Embedded Agents

OGLE IIIAlertAgent

GCNAlertAgent

Broker


4

• What does VOEvent have to do with the VO?– event messages from DBs– event messages from workflow– data mining for added semantic content– testbed for agent architectures in the VO

• Why should I be interested?– major drivers for the adoption of the VO

VOEvent and VOTech?


5

What we’ve done so far?

• Convenience layer and parser (Perl & C++)– building common cases (including GCN)– different parser methodologies– http://voevent.sourceforge.net/

• Prototype event network live and on sky• Limited (hard-wired logic) brokering• Archiving and persistent storage• RSS test feed(s)

– programmatic, or human readable?


6

Where are we going?

• Event brokering (aggregators)– time critical– non-time critical (data mining?)

• Event archiving and persistent storage– REST interface– persistence of data products

• Search service– REST interface

• Event Feeds– pull, or push?


7

Unanswered questions

• Protocol standard(s?)– How the documents are passed…

• Transport standard(s?)– How the documents are carried…


8

Protocol work

The recent work on interoperability between eSTAR and

RAPTOR/TALONS has show the importance of,

• “ack” messages

• “iamalive” messages

GatewayService

eSTARBroker


9

Mini-interoperability meeting

• Tentative agreement on a protocol standard– “ack” and “iamalive” message types

• Tentative agreement on transport standards– pull via RSS feeds– push via vanilla TCP/IP

TALONSCaltech

eSTAR


10

Should we mandate transport?

• Why? Nobody thought about RFC 1149 when IP datagram packets were standardised…

• See http://www.blug.linux.no/rfc1149/

http://www.blug.linux.no/rfc1149/index.html






11

An architecture diagram showing; publishers, subscribers, relays and repositories, along with the more complicated aggregators and brokers.

“Atomic” services

• Publisher• Subscriber• Relay• Repository

• Author• Author


12

UML diagram

CREDIT: Philip Warner, NOAO


13

Event feeds

There are two ways to feed events to consumers,

• Pull model (feeds)– polling– e.g. RSS feed

• Push model (forwarding)– via REST, SOAP or vanilla TCP


14

Pull model

• Advantage– under client control

• Disadvantage– as slow as the polling interval

A pull model, e.g. RSS, is never going to work for rapidresponse cases like GRB or transient follow-up.


15

Push model

• Advantage– theoretically faster, depending on architecture

• Disadvantage– administrative nightmare for publisher

A push model, e.g. GCN, is the only way to do rapidresponse work. However it isn’t really optimal, so we’llprobably be stuck using both approaches. Shouldn’toptimise our standard for distribution via RSS model.


16

Aggregators

Why should we aggregate event feeds?

• Consolidation– Do we need to support multiple publishers?

• Removal of duplicate events

• Added semantic content

• Trust issues


17

Storage

Why should we permanently store event messages?

• The data itself will be replicated elsewhere in the VO.

• Why do we care about the original message?

• If we want to search, we have to store.

• However, to me, the best use case for persistent storage is actually event feeds…


18

<enclosure>

• Its an optional sub-element of the RSS <item> tag that allows the feed publisher to include a link to a file.

• It has three required attributes. The url=“ ” attribute says where the enclosure is located, length=“ ” says how big it is in bytes, and type=“ ” says what its type is, a standard MIME type. e.g,

<enclosure url=“http://estar.org.uk/event.pl?id=886-1” length=“2440” type=“application/xml+voevent” />

• This is how podcasting works..


19

RSS feed

<?xml version="1.0" ?>

<rss version="2.0">

<channel>

<title>VOEvent GCN Notices</title>

<language>en</language>

<description>RSS feed for GCN notices</description>

<link>http://voeventnet.caltech.edu/GCN.html</link>

<item>

<title>MILAGRO_Source trigger 886-1</title>

<description>

<p>Possible GRB The event time is 2005-12-01T16:24:15 UT. Location RA 232.1248 Dec 62.9797 (J2000)</p>

</description>

<pubDate>Fri, 02 Dec 2005 16:09:52 PST</pubDate>

<link>http://voeventnet.caltech.edu/Notices/ivo_GCN_MILAGRO_58_886-1_2005-12-01T16:24:15.xml</link>

<author>Scott Barthelmy GCN [email protected]</author>

<comments>http://gcn.gsfc.nasa.gov/gcn3_archive.html</comments>

</item>

.

.

.

</channel>

</rss>


20

RSS feed

<?xml version="1.0" ?>

<rss version="2.0">

<channel>

<title>VOEvent GCN Notices</title>

<language>en</language>

<description>RSS feed for GCN notices</description>

<link>http://voeventnet.caltech.edu/GCN.html</link>

<item>

<title>MILAGRO_Source trigger 886-1</title>

<description>

<p>Possible GRB The event time is 2005-12-01T16:24:15 UT. Location RA 232.1248 Dec 62.9797 (J2000)</p>

</description>

<pubDate>Fri, 02 Dec 2005 16:09:52 PST</pubDate>

<link>http://voeventnet.caltech.edu/Notices/ivo_GCN_MILAGRO_58_886-1_2005-12-01T16:24:15.xml</link>

<author>Scott Barthelmy GCN [email protected]</author>

<comments>http://gcn.gsfc.nasa.gov/gcn3_archive.html</comments>

<enclosure url=“http://estar.org.uk/event.pl?id=886-1” length=“2440” type=“application/xml+voevent” />

</item>

.

.

.

</channel>

</rss>


21

Infrastructure changes for VOEvent

• Main change needed is to the registry, need to be able to register the “atomic” VOEvent services– author, publisher, relay, repository, subscriber

• Also need to handle services which do more than one atomic operation,– aggregator (relay + repository)– broker (relay + repository + publisher)


22

Infrastructure changes for agents

• Again the main (only?) change needed is to the registry, need to be able to register the agents.

• New types of services, – agents at telescopes– agents doing science– brokers for data mining


23

Things I’m going to promise you…

• Deliverables– agents deliverables– VOEvent standards & prototypes– integration into the VO


24

VOEvent deliverables

• Prototype “first cut” infrastructure• VOEvent parsers (Perl & Java)• Library to co-ordinate with Registry

– convenience layer to hide registries• Standards work, input into standards documents• VOEvent prototype network

– message handling & forwarding– storage & archiving– search services


25

Agent deliverables

• Event Broker (Agent)– aggregation of messages– data-mining for added semantic content– publishing its own event feed

• Query Broker (Agent)– marshalling asynchronous queries– encapsulation queries, hides low level interfaces

• Neural networks– more intelligent, intelligent agents– enhanced decision making for the VO


26

Integration with the VO?

• Emitting VOEvent messages from Workflow

• Wrapping ACR and Workflow inside agents

• Distributed storage of event messages in VOSpace


27

Conclusions

• Two different groups of people working on VOEvent

• Conflict between these groups now (mostly) resolved

• Standards moving forward, a lot of prototypes now being written at Exeter, Caltech and Los Alamos

• A good testbed for the intelligent agent work

the rise of the robots… alasdair allan school of physics, university of exeter alasdair allan...

Documents

votech dsrp

noao slide

nutshell slide

university of exeter

vanilla tcp slide

model feeds

rss feeds

event archiving