Connecting to Remote Repositories: Implementing the Docproxy Interface in Sakai

OAE

Jim Martino, Software Engineer, Johns Hopkins University

Outline of talk

Disclaimers How docproxy works Our Implementation details Demo Future directions

212th Sakai Conference – Los Angeles, California – June 14-16

Disclaimers

• Sakai3 / Nakamura is currently a moving target. This work was done against Nakamura 0.9

• I have had to tweak a few things to get stuff to work according to my understanding of how they should work. My understanding may be incomplete.

312th Sakai Conference – Los Angeles, California – June 14-16

Simple diagram (not to scale)

search

files

...

...

...

Sling

docproxy

Sakai3 UX

Nakamura

What happens when the UX requests something from Nakamura?

• UX hits a URL, which Sling handles

• Sling looks at the resourceType and path of the node at the URL, and resolves which servlet to hand it off to

In the case for content, the URL is of resourceType sakai/pooled-content.

Giving Sling an undecorated pooled content URL returns the bits if there is no extension.

If the extension metadata.tidy.json is appended, the metadata for the file is returned in JSON

512th Sakai Conference – Los Angeles, California – June 14-16

Locally stored file example

{ "jcr:path": "/_p/m/da/qi/i2/jqnmZuUQc63Z7mAoIA9iisGc", "jcr:name": "jqnmZuUQc63Z7mAoIA9iisGc", "sakai:fileextension": "usage", "sakai:permissions": "public", "sakai:directory": "default", "sling:resourceType": "sakai/pooled-content", "jcr:uuid": "17c29862-6431-41de-8bcf-6efb7e087527", "jcr:mixinTypes": [ "rep:AccessControllable" ], "jcr:createdBy": "admin", "sakai:pooled-content-file-name": "usage", "sakai:copyright": "creativecommons", "jcr:created": "2011-05-19T16:28:24-04:00", "sakai:pool-content-created-for": "jhopkins", "jcr:primaryType": "sakai:pooled-content"}

Sakai link node example

{ "jcr:path": "/_p/t/u7/a2/u4/FAaUGgxcgeUU5he8mo2jS5f", "jcr:name": "FAaUGgxcgeUU5he8mo2jS5f", "sakai:pooled-content-url": "http://www.dataconservancy.org", "sakai:directory": "default", "sling:resourceType": "sakai/pooled-content", "jcr:uuid": "9fb38225-aeeb-4d64-bd6a-205c9a858755", "jcr:createdBy": "admin", "jcr:mixinTypes": [ "rep:AccessControllable" ], "sakai:pooled-content-revurl": "http://www.dataconservancy.org", "sakai:pooled-content-file-name": "http://www.dataconservancy.org", "sakai:copyright": "creativecommons", "jcr:created": "2011-06-07T15:45:02-04:00", "jcr:primaryType": "sakai:pooled-content", "sakai:pool-content-created-for": "jhopkins" }

docproxy

What do we need to handle external files? The docproxy bundle addresses this. There are:

• several new Sling resourceTypes:

• servlets to handle these resourceTypes

• interfaces to ensure the action of the servlets will be generic, and to ensure compatibility with everything else

812th Sakai Conference – Los Angeles, California – June 14-16

ExternalDocumentResultMetadata interface

getType - the ExternalRepositoryProcessor type that created the object.getURI – return a URI identifying the remote objectgetContentType – return content type of the remote objectgetContentLength – return content length of the remote objectgetProperties – return a map of properties associated to the object

There is another class called ExternalDocumentResult which extends theExternalDocumentResultMetadata interface by adding one additional method:

getDocumentInputStream - returns an input stream from which the body of the documentcan be read

ExternalRepositoryProcessor interface

The ExternalRepositoryProcessor communicates with the external repository.

An ExternalRepositoryProcessor implements these methods:

updateDocument - Creates or updates a new resource in the remote system getDocument - Gets an ExternalDocumentResult from the external repositorygetDocumentMetadata - Gets just the metadata of the External Document.search - Searches for results matching the search properties.removeDocument - Removes the specified document from the external repository.getType - What kind of external repository this processor should handle.

resourceTypes

We have several new Sling resourceTypes:

• One type for external resources themselves, of course

• One type for the repository itself for operations which do not address a specific external resource

• One type for searching against the external repository

1112th Sakai Conference – Los Angeles, California – June 14-16

Docproxy nodes needed for an implementation

external-repository node /dcs.json:{ "sling:resourceType": "sakai/external-repository", "jcr:mixinTypes": "mix:referenceable", "sakai:repository-processor": "dcs",}

external-repository-search node /var/search/docproxy/dcs.json:{ "sling:resourceType": "sakai/external-repository-search", "sakai:repository-ref": "/dcs", "sakai:query-language": "xpath", "sakai:search-prop-q": "{q|*}", "sakai:search-prop-offset": "{offset|0}", "sakai:search-prop-max": "{max|10}"}

sample external-repository-document node /jhopkins/A-377_2_r.jpg { "sling:resourceType": "sakai/external-repository-document", "sakai:repository-processor": "dcs", "sakai:external-id" : "http://dcservice.dataconservancy.org:8080/dcs/entity/1238"}

Servlets

We also have servlets to handle the new resource types. Servlet resolution will depend primarily on the resourceType and node path, with further refining from any extensions and selectors appended to the URL. These servlets then call the appropriate methods in the repository processors.

1312th Sakai Conference – Los Angeles, California – June 14-16

Some binding information for docproxy servlets, and which processor methods they call

CreateExternalDocumentProxyServlet: resourceTypes = { "sakai/external-repository" }, selectors = { "create" }

Calls: updateDocument

DeleteExternalDocumentProxyServlet: resourceTypes ={ "sakai/external-repository" }, selectors = { "delete" }

Calls: removeDocument

ExternalDocumentSearchServlet: resourceTypes = { "sakai/external-repository-search" }, extensions = { "json" }

Calls: search

ExternalDocumentMetadataServlet: resourceTypes = {"sling/nonexisting", "sakai/external-repository-document" }

selectors = "metadata”, extensions = "json"

Calls: getDocumentMetadata

ExternalDocumentProxyServlet: resourceTypes = { "sling/nonexisting", "sakai/external-repository-document" }

Calls: getDocument

Example transaction

search

files

...

...

...

Sling

servlet

Sakai3 UX

Nakamura

(to external repo)

docproxy bundlerepoprocessor

(back)

Sample ExtgernalSearchResultSet from dcs search service

{"results":[

{"Content-Type":"image/tiff","Content-Length":-1,"uri":" http://dcservice.dataconservancy.org:8080/dcs/datastream/http%3A%2F%2Fdcservice.dataconservancy.org%3A8080%2Fdcs%2Fentity%2F7685 ","repository-type":"dcs","properties":{"id":"http://dcservice.dataconservancy.org:8080/dcs/entity/7685","fixity":"9e7f24755e177648074b73455ca3625e","extant":"false","fileName":"HRA-449_2_r.tif"}},

{"Content-Type":"image/jpeg","Content-Length":-1,"uri":" http://dcservice.dataconservancy.org:8080/dcs/datastream/http%3A%2F%2Fdcservice.dataconservancy.org%3A8080%2Fdcs%2Fentity%2F7679 ","repository-type":"dcs","properties":{"id":"http://dcservice.dataconservancy.org:8080/dcs/entity/7679","fixity":"03b64f52806ce570354593c23509658f","extant":"false","fileName":"HRA-449_1_s.jpg"}},

{"Content-Type":"image/jpeg","Content-Length":-1,"uri":" http://dcservice.dataconservancy.org:8080/dcs/datastream/http%3A%2F%2Fdcservice.dataconservancy.org%3A8080%2Fdcs%2Fentity%2F7687 ","repository-type":"dcs","properties":{"id":"http://dcservice.dataconservancy.org:8080/dcs/entity/7687","fixity":"403111477f19298ca6bb0a01843401f2","extant":"false","fileName":"HRA-449_2_r.jpg"}},

{"Content-Type":"image/tiff","Content-Length":-1,"uri":" http://dcservice.dataconservancy.org:8080/dcs/datastream/http%3A%2F%2Fdcservice.dataconservancy.org%3A8080%2Fdcs%2Fentity%2F7683 ","repository-type":"dcs","properties":{"id":"http://dcservice.dataconservancy.org:8080/dcs/entity/7683","fixity":"c8abd7e0dedf13cd7d3942b15a07aff0","extant":"false","fileName":"HRA-449_1_s.tif"}}],

"items":25,"total":4}

Implementation Details

• Our implementation is based on the url processor in the docproxy bundle, since our repository exposes RESTful services.

• Data packaging from the remote repository affects repository processor implementation.

• UX integration may also be affected by data packaging structure.

1712th Sakai Conference – Los Angeles, California – June 14-16

Initial Design

The first implementation was for a read-only repository, with as tight an integration as possible with native content in the UX.

Requirements:

Remote content should be an option in the content search menu

Should be able to merge remote search results with other remote searches and with local searches

Imported content should behave as much like native content as possible

1812th Sakai Conference – Los Angeles, California – June 14-16

Design decisions

• Use sakai links to point into docproxy document nodes

• Insert additional methods, configurations, etc. into existing UX files

1912th Sakai Conference – Los Angeles, California – June 14-16

Data Conservancy link node

{

"jcr:path": "/_p/s/3u/mx/yv/lubutLvOZ7CySSDeObimuI", "jcr:name": "lubutLvOZ7CySSDeObimuI", "sakai:pooled-content-url": "http://sakaidc.mse.jhu.edu:8080/jhopkins/A-399_4_s.jpg", "sakai:directory": "default", "sling:resourceType": "sakai/pooled-content", "jcr:uuid": "128a83cd-90f9-41f6-9b14-621be376751c", "jcr:createdBy": "admin", "jcr:mixinTypes": [ "rep:AccessControllable" ], "sakai:pooled-content-revurl": "http://sakaidc.mse.jhu.edu:8080/jhopkins/A-399_4_s.jpg", "sakai:pooled-content-file-name": "A-399_4_s.jpg", "sakai:copyright": "creativecommons", "jcr:created": "2011-06-03T15:35:13-04:00", "jcr:primaryType": "sakai:pooled-content", "sakai:pool-content-created-for": "jhopkins" }

Sakai OAE login screen

Skip Class logs in ...

Skip's home page

The content tab, with Data Conservancy added to the list of facets

We enter the search term, with “Content I Manage” as the active facet

Sakai does not find any matching content

Change the facet to Data Conservancy, and we see remote hits (DUs)

We select the first one, and get a list of files in the Rock Sample A-399 DU

We click on A-399_2_r.jpg to see if it's something we want

We go back, then import it, and now see it in our content

Clicking on this, we pull up the link. Let's add some tags.

Ok ... now let's search for it under Content I Manage

Ta-da!

Further Directions

• Clean up current implementation

• Implement Create/Update method

May need to develop a widget to handle submissions

• Continue to review code to see if the new use cases are supported

3412th Sakai Conference – Los Angeles, California – June 14-16