developing applications that harness the agricultural and natural resource grid (and linking your...

Developing applications that harness the Agricultural and Natural Resource Grid

(and linking your own data)

Matthew Laurenson and Seishi NinomiyaNational Agricultural Research Center

TsukubaJapan

Contents•Concepts

–Remote Procedure Calls (RPC)

–Brokers

•Using a broker–Metadata requests

–Data requests

–Issues

–Linking your own data

•SOAP•New brokers and future enhancements

Overall Goal: To help you make “portable” software applications (DSS).

Current Situation: Web-based but…

• Applications closely coupled to databases

• Limited range of applications at each site

• Duplicated development at each site

Soils

Ag and NR Grid – a “Hotel Room” for Agricultural and Natural Resource DSS

Room

Weather Elevation

Maps

Connectors for local data

DSSDSS DSSDSS

User interface in national language

New ZealandNew Zealand

Soils

Weather Elevation

Maps

地

気象 地面の高さ

地図

日本

中国中国

土壤

气象 海拔

交通图

Goal of Ag and NR Grid:To provide this “hotel room” in your

country, so you can use models developed by others (and they can use your models).

Distributed Computing Concepts

• Remote Procedure Calls

• Brokers to handle heterogeneous databases

Remote Procedure Calls (RPC)

• RPCs let a program execute a procedure on a remote computer as if it were local.

Remote Procedure Call

My Computer

remote=findRemote()

A=5

B=6

C=remote.add(A,B)

Print C

...

Remote Computer

procedure add(x,y)

return x+y

}

...

Remote Procedure Calls (RPC)

• RPCs let a program execute a procedure on a remote computer as if it were local.

• Newer RPC mechanisms let programs send and receive data objects.

• RPC mechanisms: CORBA, .NET, Java RMI, SOAP...

• RMI – only Java programs can use• SOAP – used by other languages as well

RPC vs Local Calls

• Latency (delay)– RPCs much slower than local calls– difficult to predict how long call will take

• May fail– network problems– problems with other computer (“server”)

• Security issues– firewalls...

But...

• Can distribute processing and data over multiple machines

• Great when multiple users must share the same dynamic data item or communicate with each other.

Mediated Architecture – “Brokers”

Broker

DomainDatabases

(Heterogeneous) Applications

Consistent data access(using RPC mechanism)

MetBroker

MetBroker

Korea

Japan

New Zealand

WeatherDatabases

Data Retrieval

Applications

South Africa

USA

Data RetrievalServlet

Weather Risk

Crop Growth

Disease Risk

Web Browsers

Contents•Concepts

–Remote Procedure Calls (RPC)

–Brokers

•Using a broker–Metadata requests

–Data requests

–Issues

–Linking your own data

•SOAP•New brokers and future enhancements

Data and Metadata Requests

Metadatarequest

Metadataresult

Broker Database(various)

Application

Standardrequest

DB-specificquery

DB-specificresult

Standardresult

Metadata Data

Interaction between applications and brokers uses objects – data packages

Example - MetBroker Services

• Weather Data– Single station– All stations in a region

• Metadata (data about data)– Database details (coverage of globe, login req.)– Regions within databases– Station details (latitude, longitude, start date,

end date, elements recorded, resolution...)

Elements of a Station Data Request

Weather station

Required weather elements

Time interval

Username/password

Resolution

Allow summarizing

Send Request to MetBroker and Receive Result

Data ResultTsuchiura

Operated 1976-present1999/1/3 06:00-1999/2/1 00:00

10.1, -0.2, 012.6, 3.8, 0

11.3, -1.5, 0.3...

To MetBroker

Data Requestamedas40341

rain, air temperature1999/1/1-1999/2/1

dailyOK to summarize

Query Results - Single Station

Station& query details

rain air temp. solar rad.

Station details anddata sequencesare encapsulatedin a singleresult object

Broker

Temperature

Rainfall

Radiation

Client

Spatial Request

WakayamaRainfall DB

AMeDASDaily max/min temperatures & rainfall for all stations in region

Spatial Request Background map dynamically retrieved from http://www.mapblast.com

Query results - multiple stations

Using Brokers – Implementation Issues

RMI Blocked by Firewalls

ClientApplication

Java Broker

Server MachineClientApplication

Java

Wrapper Servlet

HTTP

RMI

RMI

Client-side firewalls

Applet Using Multiple Brokers

Spatial Risk Applet

MetBroker

DEMBroker

ChizuBroker

Background maps

Elevation data

Meteorological data

Resource Server

Application text

Without Multiple Threads - Sequential

MetBroker

DEMBroker

ChizuBroker

Background maps

Elevation data

Meteorological data

Resource Server

Application text 12 seconds

Spatial Risk Applet

With Multiple Threads – Parallel

MetBroker

DEMBroker

ChizuBroker

Background maps

Elevation data

Meteorological data

Resource Server

Application text 5 seconds

Spatial Risk Applet

JavaBeans for Java GUI programs

• For rapid GUI development in Java can use existing MetBeans

• Transparently handle firewalls, latency...

• Making a simple data retrieval application takes about 30 minutes

• Used in many MetBroker demo applications

Easy Development of New Applications

Databases Linked Using “Drivers”

DatabasesApplications

Broker

Linking a Database to a Broker

• Need to write one Java class (a “driver”) with two or three main methods

• Typically use Java Database Connectivity (JDBC) drivers to query relational databases

• Can reuse existing classes for Oracle, Postgres, and MySQL JDBC.

• Drivers can read data from files via HTTP, or parse HTML tables.

• “Test harness” available for driver development

How much work?

• Add a new database to MetBroker– roughly working - 3 days– tested more thoroughly – 1 week– (assuming that database is readily accessible)

• Make a MetBroker applet– working prototype in one day– (assuming basic science is “done”)

Developer Resources

• Demos, documentation, downloads, pictures– http://www.agmodel.net

• Mailing lists – subscribe at web site

• Brokers are open source under the GNU Lesser General Public License.– Can be used in commercial applications– Any improvements must be released under same

license

• Linking your data to the grid lets you use grid tools developed by others.

• It doesn’t mean you must share your data with everyone (password protection).

• The key to interoperability is the interface

Reminders

Contents•Concepts

–Remote Procedure Calls (RPC)

–Brokers

•Using a broker–Metadata requests

–Data requests

–Issues

–Linking your own data

•SOAP•New brokers and future enhancements

SOAP

• An XML-based RPC mechanism

• Independent of programming language

• SOAP toolkits for C++, Visual Basic, Delphi, Java, ASP, PHP, Perl, Python…

• Firewall-friendly

• Basis of Microsoft .NET architecture

• A SOAP Web service is described in standard format using a .WSDL file(Web Services Description Language)

Visual Basic client application

Applicationcode

MS SOAPToolkit

VB function calls

VB objects

MetSOAP

SOAP over HTTP

MetBroker

Delphi client application

Applicationcode

Borland SOAPToolkit

Delphi function calls

Delphi objects

SOAP over HTTP

RMI

Example - MetSOAP

SOAP Summary

• Positives:– Opens brokers to a range of programming languages

– Already the standard for Web-based RPC

– Avoids most firewall issues

• Negatives– Can’t reuse Java classes - must “mirror” in each language

– Interoperability is still imperfect (but improving fast)

– Inefficient for large data sets

Typical Array in SOAP

• ...  • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">21.79</item> • <item xsi:type="xsd:float">21.79</item> • <item xsi:type="xsd:float">23.3</item> • <item xsi:type="xsd:float">21.11</item> • <item xsi:type="xsd:float">19.49</item> • <item xsi:type="xsd:float">18.48</item> • <item xsi:type="xsd:float">16.52</item> • <item xsi:type="xsd:float">14.73</item> • <item xsi:type="xsd:float">14.73</item> • ...

Model Deployment Options - Summary

• Java applet– flexible, responsive once loaded but requires plug-in

• Java servlet– simple browsers, rapid load, but clunky interface

• SOAP– direct access for Delphi, VB…

• Existing models (FORTAN,C,BASIC)– Use simple Java app. to retrieve data and write to a file

Contents

•Concepts–Brokers–Remote Procedure Calls (RPC)

•Using a broker–Metadata requests–Data requests–Issues–Linking your own data

•SOAP•New brokers and future enhancements

New Brokers

• DEM Broker - Digital Elevation Model data– Japan 50m resolution– World 1km resolution

• ResourceServer– Internet-based service for software localization

• ChizuBroker – Web map services– www.mapfan.com (Japan)– www.mapzone.co.nz (New Zealand)

• SoilBroker – soils data

Future enhancements

• Broker-side interpolation (using DEMBroker)– temporal– spatial

Server

Temperature

Rainfall

Radiation

Client

“Virtual station”Broker-side interpolation at single site

Future enhancements

• Broker-side interpolation (using DEMBroker)– temporal– spatial

• Other brokers– Place names– Crops and crop models (prototyped)– Agrichemicals

Final Points

• The Grid provides a stable, powerful foundation for Ag and NR software

• You can now develop grid applications in your favourite programming language.

• NARO is keen to help you to both develop new applications and link new databases.

• Database owners around the world will be keen to evaluate your applications.

Thank you