Linked ProcessAn Internet-scale distributed computing

framework

Center For Nonlinear Studies

Joshua Shinavier

August 5th, 2009

• Internet-scale distributed computing

• eXtensible Messaging and Presence Protocol (XMPP)

• Linked Process specification

• Current Linked Process implementation

• Demos

Overview

2

Abstract

3

The LANL-based Linked Process project takes a new approach to Internet-scale distributed computing. While existing large-scale

grid computing projects are typically very constrained in the kinds of computational tasks which can be performed, the kinds of

devices which can participate in computation, and in the overall architecture of the system, the Linked Process specification

provides the foundation for a much larger and more general-purpose distributed computing platform. Any device supporting the eXtensible Messaging and Presence Protocol (XMPP), be it a

supercomputer or a cellular phone, is a potential node in a global compute cloud, communicating with other nodes in a manner similar to human chat. The implementation currently under development provides a simple API and supports a number of

popular scripting languages, allowing software developers to write distributed applications with ease. This presentation will provide

an overview of the Linked Process specification and discuss a range of potential uses of the technology.

• Marko A. Rodriguez (LANL)

• http://markorodriguez.com/

• Joshua Shinavier (RPI / LANL)

• http://fortytwo.net/

• Peter Neubauer (Neo Technology)

• http://www.linkedin.com/neubauer/

• Max O. Bond (Santa Fe Complex)

• Mick Thompson (Santa Fe Complex)

• http://davidmichaelthompson.com

Contributors (to date)

4

• distributed computing

• combines computational power of multiple machines

• makes effective use of local resources

• Berkeley Open Infrastructure for Network Computing (BOINC)

• supports SETI@home, PrimeGrid, etc.

• cloud computing

• Amazon EC2

• Google App Engine

Internet-scale distributed computing

5

• eXtensible Messaging and Presence Protocol

• deals with “presence” and asynchronous message passing among clients and servers

• open standard

• based on machine-independent Jabber identifiers (JIDs)

6

XMPP is everywhere

7

• uses XMPP messaging for inter-machine communication

• any XMPP-enabled device may participate

• bring mobile devices into the cloud

• augment the compute power of a single device

• grid computing

• specification is called LoP, for “Linking Open Processors”

8

Linked Process

LoP is like chat...

9

... but among software agents

10

LoP architecture

11

• a job is a task to be performed by a virtual machine, e.g.

• computationally intensive operations

• manipulation of local resources

• LoP allows you to:

• submit a job -- <submit_job/>

• check on the status of a job -- <job_status/>

• abort a job -- <abort_job/>

Jobs: units of computation

12

Example: submitting a job

13

<iq from="lp1@gmail.com/LoPVillein/1234"

to="lp2@gmail.com/LoPVM/EFGH"

type="get" id="xxxx">

<submit_job xmlns="http://linkedprocess.org/2009/06/VirtualMachine#"

vm_password="abc123pass">

var temp=0;

for(i=0; i&lt;10; i++) {

temp = temp + 1;

}

temp;

</submit_job>

</iq>

<iq from="lp2@gmail.com/LoPVM/EFGH"

to="lp1@gmail.com/LoPVillein/1234"

type="result" id="xxxx">

<submit_job xmlns="http://linkedprocess.org/2009/06/VirtualMachine#">

10

<submit_job/>

</iq>

request

response

• VM is maintained by an XMPP client

• manages jobs and data-typed “bindings” (variables) -- <manage_bindings/>

• provides a scripting environment using a particular language (e.g. JavaScript, Ruby, etc.)

• may be terminated at any time -- <terminate_vm/>

Virtual machines: addressable “computers” in the cloud

14

Example: spawning a VM

15

<iq from="lp1@gmail.com/LoPVillein/1234"

to="lp2@gmail.com/LoPFarm/ABCD"

type="get" id="xxxx">

<spawn_vm xmlns="http://linkedprocess.org/2009/06/Farm#"

vm_species="javascript" />

</iq>

<iq from="lp2@gmail.com/LoPFarm/ABCD"

to="lp1@gmail.com/LoPVillein/1234"

type="result" id="xxxx">

<spawn_vm xmlns="http://linkedprocess.org/2009/06/Farm#"

vm_jid="lp2@gmail.com/LoPVM/EFGH"

vm_password="abc123pass"

vm_species="javascript" />

</iq>

request

response

• farm is maintained by an XMPP client

• provides access to virtual machines

• farm allows you to:

• spawn a virtual machine -- <spawn_vm/>

• query for information about the environment (e.g. language support, security restrictions, etc.) -- disco#info

• multiple farms may share the same “bare” JID

Farms: LoP service providers

16

Example: service discovery

17

<iq id="zzzz" from="lp2@gmail.com/LoPFarm/ABCD" to="lop1@gmail.com/LoPVIllein/1234" type="result"> <query xmlns="http://jabber.org/protocol/disco#info"> <identity category="client" name="LoPVM" type="bot" /> <feature var="http://jabber.org/protocol/disco#info" /> <feature var="http://linkedprocess.org/2009/06/Farm#" /> <x xmlns="jabber:x:data" type="result"> <field label="supported virtual machine species" var="vm_species" type="list-single"> <option label="groovy 1.5.6 (groovy )"> <value>groovy</value> </option> <option label="javascript 1.6 (Mozilla Rhino 1.6R7)"> <value>javascript</value> </option> ... </field> <field label="exercise a permission" var="permission" type="boolean"> <value>1</value> </field> <field label="exit the farm process" var="exit" type="boolean"> <value>0</value> </field> <field label="execute a program" var="exec" type="list-multi" /> <field label="create a file system link" var="link" type="list-multi" /> ... </x> </query></iq>

• jobs operate within a VM sandbox

• subject to named permissions, e.g.

• file I/O

• network I/O

• introspection

• password-protection of VMs

• can specify limits on VMs per farm, number of jobs in a queue, VM and job timeouts, etc.

• XMPP supports SSL (Secure Socket Layer)

Security

18

• based on Java 1.6

• takes advantage of built-in Java security, scripting framework

• job scheduler serves as a miniature operating system

• supports a number of scripting languages

• JavaScript, Ruby, Python, Groovy

• support for additional languages is easy to add

• https://scripting.dev.java.net/

• deployed farms in New Mexico, New York, Sweden

Implementation (to date)

19

AWK

BeanShell

ejs

FreeMarker

Groovy

Jaskell

Java

JavaScript

JSR 223 supports...

20

Jelly

JEP

Jexl

jst

JudoScript

JUEL

OGNL

Pnuts

Python

Ruby

Scheme

Sleep

Tcl

Velocity

XPath

XSLT

• distributed primality testing

• Linked Data

• for the future:

• LoP API for Google’s MapReduce

• computational support for mobile devices

• distributed matrix operations, image processing, etc.

Demos

21

• http://linkedprocess.org/

• Rodriguez, M.A., “A Reflection on the Structure and Process of the Web of Data,” Bulletin of the American Society for Information Science and Technology, American Society for Information Science and Technology, volume 35, number 6, ISSN: 1550-8366, LA-UR-09-03724, pages 38-43, August 2009.

• XMPP Core spec: http://xmpp.org/rfcs/rfc3920.html

• XMPP Instant Messaging and Presence spec: http://xmpp.org/rfcs/rfc3921.html

• XMPP Extensions: http://xmpp.org/extensions/

See also

22