d8.2 application-specific gateways, developed in the first ...d8.2.pdfjra2.1 statistical seismology...

SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481

Project number: RI 283481

Project acronym: SCI-BUS

Project full title: SCIentific gateway Based User Support

Research Infrastructures INFRA-2011-1.2.1e-Science environments

D8.2 Application-specific gateways, developed in the first release

REPORT

Due date of deliverable: 30/09/2012

Start date of project: 01/10/2011

Actual submission date: 30/09/2012

Duration: 36 months

Lead beneficiary: ETH Zürich WP8

Dissemination: PU Version: 2.0

D8.2 Application-specific gateways developed in the first release SCI-BUS 283481

WP8:JRA2 2

1 Table of Contents

1! Table of Contents ............................................................................................................... 2!

2! Status and Change History ................................................................................................ 4!

3! Glossary ............................................................................................................................. 6!

4! Introduction ......................................................................................................................... 7!

5! Gateways ........................................................................................................................... 8!

5.1! JRA2.1 Statistical Seismology Science Gateway (METU) .......................................... 9!

5.1.1! SSS-Gateway Architecture ................................................................................. 10!

5.1.2! SSS-Gateway Status and Plans ......................................................................... 11!

5.2! JRA2.2 Swiss Systems Biology Community Science Gateway (ETH Zurich) ........... 16!

5.2.1! iPortal Architecture .............................................................................................. 17!

5.2.2! iPortal Design ...................................................................................................... 18!

5.2.3! iPortal Status and Plans ...................................................................................... 19!

5.3! JRA2.3 German MoSGrid Chemist Community Science Gateway (EKUT) .............. 22!

5.3.1! Architecture ......................................................................................................... 22!

5.3.2! Status and Plans ................................................................................................. 23!

5.4! JRA2.4 Amsterdam Medical Centre Hospital Gateway (AMC) .................................. 27!

5.4.1! AMC e-bioinfra gateway Architecture .................................................................. 27!

5.4.2! AMC e-bioinfra gateway Status and Plans .......................................................... 30!

5.4.3! Deployment plans and instructions for SA1 ........................................................ 31!

5.5! JRA2.5 PireGrid Community Commercial Gateway (UNIZAR-BIFI) .......................... 32!

5.5.1! The PireGrid Community Commercial Gateway ................................................. 32!

5.5.2! Hadoop over OpenStack specific portlet ............................................................. 33!

5.5.3! Deployment plans and instructions for SA1 ........................................................ 37!

5.6! JRA2.6 Commercial Gateway for Business Process Optimization (SIMSOFT) ........ 38!

5.6.1! SimBusPro Portal Architecture ............................................................................ 40!

5.6.2! SimBusPro Status and Plans .............................................................................. 40!

5.6.3! SimBusProDeployment plans and instructions for SA1 ...................................... 41!


WP8:JRA2 3

5.7! JRA2.7 Blender Rendering Community Gateway (LU) .............................................. 42!

5.7.1! Blender Rendering Community Gateway Status and Plans ................................ 42!

5.7.2! Blender Rendering Community Gateway Architecture ........................................ 43!

5.8! JRA2.8 Citizen Web Community Gateway (EG) ........................................................ 46!

5.8.1! eDOX Workflow and Architecture ....................................................................... 46!

5.8.2! Who can use the portal ....................................................................................... 47!

5.8.3! Registering and logging in ................................................................................... 48!

5.8.4! Using the portal ................................................................................................... 48!

5.8.5! Security ............................................................................................................... 51!

5.8.6! eDOX Status and Plans ...................................................................................... 51!

5.9! JRA2.9 Astrophysics Community Science Gateway (INAF) ...................................... 52!

5.9.1! VisIVO Science Gateway .................................................................................... 52!

5.9.2! VisIVO Mobile ..................................................................................................... 54!

5.9.3! VisIVO Architecture ............................................................................................. 55!

5.9.4! VisIVO Status and Plans ..................................................................................... 55!

5.10! JRA2.10 Build and Test Public Gateway for Application Developers (4D) .............. 57!

5.10.1! 4D Build and Test Portal Architecture ............................................................... 57!

5.10.2! 4D Build and Test Portal Status ........................................................................ 58!

5.11! JRA2.11 Helio-Physics Gateway (TCD) .................................................................. 62!

5.11.1! Helio-Physics Portal Architecture ...................................................................... 66!

5.11.2! Helio-Physics Portal Status and Plans .............................................................. 67!


WP8:JRA2 4

2 Status and Change History

Status: Name: Date: Signature:

Draft: Peter Kunszt 13/09/2012 n.n. electronically

Reviewed: Jozsef Kuti 23/09/2012 n.n. electronically

Approved: Peter Kacsuk 30/09/2012 n.n. electronically

Version Date Pages Author Modification

1.0 05/09/2012 All Peter Kunszt Creation

1.1 12/09/2012 All JRA2 Content for all Portals except Blender

added13

1.2 13/09/2012 33-37 Julius

Tuomisto Content for Blender Community added

1.3 13/09/2012 30-31 Ferhat Kaya Content of Simsoft was modified

1.4 14/09/2012 41-42 Eva Sciacca Content of INAF was modified

1.5 14/09/2012 8-13 Cevat Sener Content of the Statistical Seismology

Science Gateway was modified

1.6 17/09/2012 33-36 Ferhat Kaya Content of Simsoft was modified by

adding workflow screenshot

1.7 20/09/2012 31-33 Peter Kunszt Adding content for Unizar, adding figure

numbers for semisoft

1.8 25/09/2012 All Peter Kunszt

Corrections implemented where possible based on reviewer feedback

(G.Terstyanszky and A. Szabo). Comments included for people to correct

where necessary.

1.9 25/09/2012 28 Silvia

Olabarriaga AMC reviewer comments encorporated

1.10 26/09/2012 51-52 Eva Takacs 4D comments encorporated

1.11 26/09/2012 5.8 EGroup, P.Kunszt New Section 5.8


WP8:JRA2 5

1.12 26/09/2012 36-39 Ferhat Kaya Simsoft comments encorporated

1.13 28/09/2012 5.5 G.Ruiz UnizarBIFI redone

1.14 28/09/2012 5.11 G.Pierantoni TCD section improved

1.15 28/09/2012 5.8 P.Tihanyi EGROUP section improved

2.0 28/09/2012 all Eva Feuer Format changes


WP8:JRA2 6

3 Glossary

BPMN Business Process Model and Notation

CO Confidential

DCI Distributed Computing Infrastructures

DG Desktop Grid

DoW Description of Work

GAM General Administration Manager

JRA Joint Research Activity

NA Networking Activity

TMB Technical Management Board

PDF Portable Document Format

PMB Project Management Board

PU Public

QR Quality Reviewer

SA Service Activity

SG Service Grid

SVN Subversion, version control system

WP Workpackage


WP8:JRA2 7

4 Introduction

The main objective of WP8 (JRA2) is to develop the application-specific gateways based on the generic gateway technology provided by JRA1. The gateways will be deployed and operated by WP4 (SA1). This deliverable reports on the development of the first release of the application-specific gateways.

Deliverable D8.1 is the actual software and documentation of the developed gateways.

All individual gateways depend on the generic technology provided by JRA1:

• WS-PGRADE

• gUSE

• Liferay

The implementation language is JAVA but the individual portals depend on various other external components and have different dependencies. The overall architecture for JRA2 is shown in Figure 1. Each Gateway has specialized on certain components.

Figure 1: Overall generic architecture.

!"#$%&'#()'*+",-#'.**

/!01&2/*/34)53)*6,7)8,9:*

;+0*

;<<*/7#')*=#'*!"#$%**

+",-#'.*7#*,33)::*,<<:*45*!"#$%:**

!"#$%*+'#>4%)'**

;',6'4%*?/<,45@A*;'.)54,5*6'4%A*&,"B3*6'4%A*&)"C4,5*6'4%*&0D0*E):(7#<*6'4%A*&$"C,'4,5*6'4%A*!"6'4%*?!F4")@A*!GH+!IJH*KG*#=*J6JJ*

!'#,B,5*6'4%A*J6L0E*?J3#5#.9*6'4%A*07,"9@*60ME;*7',4545C*C'4%A6'4%*0')",5%A*I$56'4%*?I$5C,'4,5*N,B#5,"*6'4%@*

0O)'6'4%*?+#'7$C,"*,5%*/<,45@A*H,7F6'4%*?/<,45@*H#/6'4%*?H#")3$")*/4.$",B#5*6'4%*#=*E16'4%@A*P5#8")%C)6'4%*H,",9:4,*

+4')6'4%*?/<,45*,5%*D',53)@A*/))16'4%*?/#$7F1J,:7*J$'#<),5*6'4%@*/84::6'4%A*Q$'(4:F*6'4%A*2P*N6/A*2P*RF47)*L#:)*6'4%*

KG!J*?!)57',"*J$'#<),5*6'4%@A*R):7.45:7)'*E):(7#<*6'4%**

!"$:7)':S**+&/A**M/DA*!#5%#'A**/6J*

TTT*

*

*************************************************************************************!"#$%&'(&)*(+,-%&*$(

E):(7#<*6'4%:S*

&G0N!A*U7').1R)OA*G$'6'4%*

/$<)'13#.<$7)':*

R)O/)'>43)*V*;+0*

J$3,"9<7$:*G<)5N)O$",*

*CM47)*

H4%%")8,')***

*2N0!GLJ*

H4%%")8,')***

*;L!*

H4%%")8,')***

;<<*W* ;<<*X* ;<<*Y*

.-/( 01&23)(456( /78(

TTT**

TTT**

+'4>,7)*!"#$%:*

;+0* ;+0* ;+0* ;+0*

!#..)'34,"*!#.<#5)57:*

/!01&2/*C)5)'43*C,7)8,9*

6)'.

,5*H

#:6'4%*

!#..$5479*

/7,B:B3,"*/)4:.

#"#C9*

!#..$5479*

&")5%)'*L

)5%)'45

C*!#

..$5479*

;.:7)'%,.*H

)%43,"*

!)57)'*!#.

.$5479*

/84::*+'#7)#.

43:*

!#..$5479*

;:7'#<

F9:43:*

!#..$5479*

I)"4#*<F9:43:*

!#..$5479*

&$:45

)::*+

'#3)::*

!#..$5479*

/#Z8

,')*O$4"%*,5%*

7):7**!#.

.$5479*

!4B[)5*R

)O*

!#..$5479*

+4')6'4%*!#.

.)'34,"*

!#..$5479*


WP8:JRA2 8

5 Gateways

Suggested structure:

1. Architecture, with a block diagram containing all components, JRA1 dependencies and external dependencies

2. Status: Complete planned feature list

a. Implemented in 1st release

b. Planned for future releases

c. Testing status


WP8:JRA2 9

5.1 JRA2.1 Statistical Seismology Science Gateway (METU) After the successful completion of the EU FP7 SEE-GRID-SCI project, the three of its six Seismology VO applications were ported onto the Seismology Portal, based on the P-GRADE Portal, for ease of access. These applications are Earthquake Location Finding, Seismic Risk Assessment (SRA) and Numerical Modelling of Mantle Convection. Although a set of fundamental methods of statistical seismology were implemented within the SRA application, there are many more advanced and recent statistical techniques used by seismologists to model the seismicity.

The Statistical Seismology Science Gateway, SSS-Gateway, developed within the framework of the SCI-BUS project aims to provide a comprehensive set of statistical methods for e-scientists working in seismology. SSS-Gateway will provide an environment in such a way that skilled users can develop their own seismology models by accessing the gateway services, while novice users can easily use the existing applications.

This science gateway aims to implement the following foremost statistical seismology functions (SSFs), reflecting the latest advances in the literature, for the international seismology community:

SSF1: Integration of multi-source data for increased reliability and quality

SSF2: Determination of probability distributions and their input parameters

SSF3: Robust techniques for the parameter estimation in models and relations

SSF4: Complex predictive modeling of earthquake phenomena in time/space domains

SSF5: Ground motion prediction equations

SSF6: Complex logic-tree and sensitivity analysis in probabilistic seismic hazard assessment

SSF7: Statistical calculations for seismic risk

SSFs listed above constitute logical paths from basic steps of statistical seismology towards the composite seismic forecast models. They are implemented by SSF Workflows, and the gateway offers a choice of three service levels for its users with different skills to access them:

Simple: A user can run and get the result of any of SSFs for her/his data and parameters easily through the GUI provided.

Advanced: A user can code her/his models in a programming language (such as FORTRAN, C/C++, Octave etc) calling any combination of the web services provided to access the SSF workflows.

Expert: A user can design and use her/his own improved workflows by embedding the SSF workflows provided.


WP8:JRA2 10

5.1.1 SSS-Gateway Architecture SSS-Gateway is developed on top of the generic-purpose gateway services of WS-PGRADE and gUSE, as shown in Figure 2. Here, the SSF Workflows, implementing the statistical seismology functions, are handled by gUSE and the end-user interaction and SSS-Gateway portlets are maintained by Liferay.

Figure 2: Architecture of SSS-Gateway.

The key points of this architecture are given below:

Each SSF needs development of a workflow and a simple-use portlet to access and use the SSF Workflow. Through the ASM API, these portlets access the generic-purpose gUSE services to access, submit and monitor the SSF Workflows as well as input handling, DCI management and output retrieval operations. These portlets both allow the users to easily manage the parameters (Figure 3) for the related SSF Workflow and give them the chance to examine the results directly without downloading as seen in Figure 4.

In addition to use of the ASM API, the advanced-use portlet utilizes the Remote API to facilitate execution of user code calling the SSF Workflows in the form of web services. The portlet uses the ASM API for preparing and executing a workflow which wraps the user code and auxiliary files, while the Remote API calls are used for handling the execution of an individual SSF Workflow requested by the user code. On Figure 5 the advanced-use portlet is shown with a sample user code calling some SSF Workflows.


WP8:JRA2 11

With the expert-use support purely provided by the WS-PGRADE portlets, users can have their own high-level, improved workflows by embedding the SSF Workflows. They can also manage their own datasets through these portlets.

For executing the SSF Workflows, users need to submit a valid certificate to access a European DCI. Exception to this requirement is that, for limited trial purposes only, a user can directly run SFFs on the local cluster “Pomegranate” even when the user holds no such certificate.

5.1.2 SSS-Gateway Status and Plans The major planning steps are summarized in the table below.

Task Month Status WS-PGRADE/gUSE Installation M1 Done M1 Startup and Development Plan M2 Done M2 Test Plan – Initial M2 Done M2 Requirements Analysis M3 Done M3 Architectural Design M3 Done M4 SSF1 (Workflow + Simple-use Portlet) M4 Done M6 SSF2 (Workflow + Simple-use Portlet) M5 Done M7 Advanced-use Portlet M6 Done M7 SSF3 (Workflow + Simple-use Portlet) M7 Done M8 Review & Revise M8 Done M9 SA2 Test Integration Started M9 Done M9 SSF4 (Workflow + Simple-use Portlet) M10 Done M11 SSF5 (Workflow + Simple-use Portlet) M11 In Progress Review & Revise M12 In Progress - - M13-M15 SSF6 (Workflow + Simple-use Portlet) M16-M17 SSF7 (Workflow + Simple-use Portlet) M18-M19 Review & Revise M20-M21 - - M22-M30 Review & Revise M31-M34

This table lists only the JRA2 activities regarding the development of SSS-Gateway. As seen in the table, with a few exceptions, the steps are completed as planned. First, the WS-PGRADE/gUSE environment was set up. Next, the development plan as well as the initial version of the test plan was prepared. This test plan was then detailed and updated whenever needed.

After completing the requirements analysis and architectural design, SSFs (a workflow and its simple-use portlet for each) and the advanced-use portlet are constructed one after the other. A construction phase here includes activities from detailed design to unit testing. The review & revise steps are performed on all the modules developed previously. There, the


WP8:JRA2 12

activities involve clean up, bug-fixing, updates as a result of feature requests, inclusion of new methods and options, improvements, unit tests etc.

Figure 3: Parameter management example in SSF1 portlet.

Figure 4: Examining results in SSF2 portlet.

As indicated in the table, the developments of the first four SSFs and advanced-use portlet are completed within this first phase as detailed below:

• SSF1


WP8:JRA2 13

In order to construct a consistent and reliable earthquake database, the catalogues for related regions has to be compiled from different sources by considering both historical and instrumental seismicity. SSF1 lets users to integrate earthquake catalogs selected or provided by applying crosschecks through new statistical techniques to ensure that repetitions are not included. The users may specify bounds for locations, dates and magnitudes of the earthquakes in the final integrated catalogue (Figure 3). It is also possible to set some integration parameters, such as time and space windows, used in the catalogue integration process. SSF1 also performs completeness analysis on the integrated catalogue data and provides visual output, which is also customizable through input parameters, for the results of this analysis.

• SSF2

Due to uncertainties in magnitudes, source to site distances, holding times between earthquakes etc determination of probability distribution functions (PDF) of these variables are very important. By means of this SSF, users can apply statistical methods to estimate PDFs, with their input parameters, of catalogues and source zones provided through the GUI as seen in Figure 4. The users may choose among the most commonly used estimation methods, such as least squares and maximum likelihood, for estimating the input parameters of magnitude-frequency relations. SSF2 presents the estimated parameter values as both textual and visual outputs to the users.

• SSF3

This SSF uses robust techniques for estimation of magnitude-frequency relationship parameters. The inputs for this SSF are an earthquake catalogue and seismic sources for which the parameters are to be estimated. Using the provided inputs, SSF3 first calculates skewness and kurtosis values for the earthquake occurrences related to each provided seismic source. Then, the distributions to which the residuals fit and related distribution parameters are tried to be determined. Finally, using these distributions and parameters, the magnitude-frequency relationship parameters for each source are estimated using robust methods. The estimated parameter values together with the used distribution for each given seismic source are included in the output of SSF3. Q-Q plots and the calculated skewness and kurtosis values are also presented as outputs to the user for further examination.

• SSF4

This SSF uses different complex predictive models of earthquake phenomena in time-space domains for seismic hazard estimation. The inputs for SSF4 are an earthquake catalogue and required information such as the coordinates and the style of faulting about seismic sources. The parameter settings in this SSF involve selections of the


WP8:JRA2 14

attenuation relation and the predictive model to be used in the hazard estimation. As output, SSF4 produces seismic hazard estimation based on time, in the form of probability values. For the Poisson model, the probabilities are calculated for earthquake occurrences. On the other hand, Renewal models produce the probabilities of ground motion exceeding specific levels.

• Advanced-use

The advanced-use portlet requires the user source code and the necessary input and parameter files for SSFs called from the user code, bundled together in an archive file. The portlet then lists the contents of the uploaded archive and also provides the chance to check the source code with SSF calls highlighted (Figure 5). After the execution is completed, the portlet allows the user to download an archive file containing all the outputs of individual SSFs which are called from the user code.

Currently, the fifth SSF is under development with a delay because of operational problems and its complexity, and the last two SSFs are planned for the next release. Since the expert-use is supported by the JRA1 technologies, no additional development is required for it.

Status:

• Each portlet constructed is sent to SA2 once the SA2 test integration is started. For these tests, a tester account is provided. Till the end of M11, the web tests of the first two SSFs are completed by SA2.

• SSS-Gateway is available with all the portlets developed at http://portal.grid.org.tr.

• This gateway will be deployed on a new server to be acquired and then operated by SA1, for which plans and instructions will be provided.

• The users of the earlier Seismology Portal have started to try and test this new gateway.


WP8:JRA2 15

Figure 5: Advanced-use portlet.


WP8:JRA2 16

5.2 JRA2.2 Swiss Systems Biology Community Science Gateway (ETH Zurich)

The Swiss Proteomics Portal existed as a Gridsphere-based P-GRADE solution at the time when SCI-BUS has started. We have ported the existing portal to the WS-PGRADE Liferay-based technology. The motivation to move to a more scalable and better maintainable system comes from the very large amounts of data produced by the Swiss Systems Biology initiative, SystemsX.ch.

We call the new portal iPortal and it is running at https://iportal.ethz.ch.

The already existing workflows have been rewritten in the WS-PGRADE based system. Several new workflows have been implemented, as well as dedicated portlets for better user experience:

• Workflow submission wizard. This is a very proteomics specific set of pages guiding the users through the configuration of existing concrete workflows.

• Data management portlet (openBIS interaction).

• Monitoring portlet for simple job monitoring. This portlet shows the monitoring information in a simplified manner to our users so that they are not overload with too much information as in the generic monitor. It also allows administrative users to see and rescue other user’s jobs, which is great for support activities.

• Infrastructure initialization portlet


WP8:JRA2 17

5.2.1 iPortal Architecture

Figure6: Architecture of iPortal.

The JRA1 components are being made use of through the ASM API and the WS-PGRADE. Expert users can make use of the portal through the generic interface as usual. The LSF connector has been developed by JRA1 and ETH in a joint collaboration. We also integrate with our openBIS data management server over a specialized interface.

The Liferay-based web-interface contains all of the components of WS-PGRADE. However, we have found that only our expert users are able to use the generic interface to create new workflows and to navigate through the monitoring pages. We have therefore developed dedicated special-purpose portlets that our users can interact with and deployed them next to WS-PGRADE. Our portlets interact with gUSE through the ASM API. We also have openBIS display and visualization in the portlets of the Gateway so that users can select and navigate through their data.

!"#$$%&'()%!*#+,*+%%-./0'),1%23#4%

!&!-2%%%%

!"#$%&'((((((

567%8.*9(%-('$:+3%;<3':'$=%

%8!>%

-.,,+*:.3%%%

%?@-%

&#44(+"93+%%%

)*"++(,%-.$-/"0+(1-//23".'(

45)6(

718(9%":4$(

%A!BC2@?D5%

%%

%#C.3:9(%C.3:(+:$%

%%

-;$398)()$%<$%(&3:(7&.&=&+$(

-;$398)(>,8(

567%!:.391+%@+$.'3*+$%

>)?(>,8(


WP8:JRA2 18

5.2.2 iPortal Design

Figure7: iPortal Wizard – first step

We have implemented several portlets in the iPortal. For easy submission of predefined workflows, the Workflow Wizard guides the users through a series of well-defined steps to configure and parametrize the proteomics search and quantification workflows. Figure7 shows the first step of the Wizard’s 5-step process:

1. Workflow type selection: Search or quantification. We will add selective reaction monitoring (SRM) and other workflow types. Rosetta workflow was planned initially earlier but due to popular demand we implemented quantification first.

2. Workflow selection. Depending on the type there are several concrete workflows to choose from, depending on the analysis desired.

3. Data selection. The openBIS table is shown for the given user’s data. Filtering options are available for quick data selection.

4. Parameters. Parametrization of the workflow. The user can save custom parameter sets or load previously stored ones (custom or public)

5. Submission: summary of all data, big submit button.

The wizard is now being re-engineered using the Liferay portlet API to provide an even more flexible interface in the future where the individual steps and interactions can be customized by assembling widgets.

Once the workflow has been submitted, the user can monitor and manage his or her jobs through our simplified workflow monitoring portlet (Figure8). We have created a dedicated view that also enables administrators of the system to interact with other user’s jobs to rescue or debug them.


WP8:JRA2 19

Figure8: iPortal Monitoring portlet view and detailed monitoring page that shows up when clickin on an individual job to be monitored.

5.2.3 iPortal Status and Plans The most important planning steps are summarized in the table below.

Task Month Status WS-PGRADE dev server M2 !"#$%&'%Workflows form P-GRADE M2 !"#$%&'%Web Interface Design M2 !"#$%&(%Deliverable 2.1 Review M3 !"#$%&(%WS-PGRADE production server M3 !"#$%&(%Initial User Testing M3 !"#$%&)%TPP Workflow M4 !"#$%&(%Web Interface Implementation: Entry Page M4 !"#$%&(%Web Interface Implementation: TPP M4 !"#$%&(%


WP8:JRA2 20

Testing plan M4 !"#$%&)%LFQ Workflow M5 !"#$%&(%Rosetta Workflow M6 !$*+,$-%AAI integration M6 !"#$%&.%Web Interface: Update Layout M6 !"#$%&/0%Sequest Workflow M5 !"#$%&.%SRM Workflow M10 1#%23"43$55%Web Interface: Cleanup code M10 1#%23"43$55%Workflows & Web Interface: Extend functionality

M10

Testing Plan / Implementation M14 Swath Workflow M15 Full Modeling Workflow M15 Grid access and login M16

The most important tasks could be done on time. The portal has been ported, set up and is being used in production by a very large proteomics laboratory of the ETH Zurich. The development will continue on workflows that matter most to the lab. There are already more workflows than we anticipated to be done for SCI-BUS, so we are exceeding our expectations already. For the remainder of the project we intend to implement the missing workflows, the SRM workflow (in more than one version) is already in progress. The Rosetta workflow is delayed but the demand was also really low so we have reprioritized it for a later date.

Status:

• The proteomics portal has been moved from Gridsphere / P-GRADE to LifeRay / WS-PGRADE and gUSE. The base code of the old portal has entirely rewritten to enable the easy and fast publication of new workflows and make the maintenance simpler.

• The security architecture has also been adapted.

• OpenBIS connectors and API have been put in place

• An infrastructure initialization portlet has been designed and implemented which set up the accounts of the users automatically. As an average lab member has very limited computer skills this portlet makes easier the usage of the portal for them and decreases the number of the errors and the needed support work.

• A development (imsb-ra-pgrade.ethz.ch) and a production gateway (iportal.ethz.ch) have been set up. NOTE: these servers cannot be reached from outside the ETH because both servers reside inside the ETH firewall. We plan to set up a public service as well once we have a data hosting service for data uploaded by external people.

• The very same portal will be operated by SA1. Currently whoever is interested in the usage of the iPortal will need an ETH guest account. We will improve this in the next release.


WP8:JRA2 21

• Testing has started in cooperation with SA2, whose members have received ETH guest accounts to be able to access the ETH gateway.

• The code of the workflow wizard and monitoring portlet is integrated to the ETICS build system provided by SA2 and can be built on this server.

• The iPortal gateway has over a dozen registered users who use the workflows regularly, submitting thousands of jobs through hundreds of workflows per month.


WP8:JRA2 22

5.3 JRA2.3 German MoSGrid Chemist Community Science Gateway (EKUT)

MoSGrid (mosgrid.de) focuses on the configuration and provision of Grid services for molecular simulations. MoSGrid shall enable the usage of D-Grid-Infrastructure for high-performance computing in the field of molecular simulation including annotation of the results with metadata and their provision for data mining and knowledge generation. MoSGrid aims to support the users in all fields of molecular simulation calculations. Via portlets, the user can access data repositories storing information on molecular properties as well as predefined workflows. The latter can be used to easily create jobs for repeatedly occurring tasks and execute them on the Grid (Preprocessing and Job Submission). The portal also supports the analysis of the resulting calculation results with portlet-based tools. Semantic annotation of simulation results facilitates data mining, cross-referencing of different result data sets, and avoids the repetition of simulations already done by other users. The data repository also permits external referencing of simulation results.

MoSGrid is supported by the German Federal Ministry of Education and Research until 12/2012.

5.3.1 Architecture The MoSGrid Portal relies on a layered architecture as already depicted inFigure 1. As MoSGrid is integrated into the German D-Grid infrastructure, UNICORE is required as grid middleware. The decision to use XtreemFS as storage solution was also made very early on. These components were integrated seamlessly with gUSE hiding the complex interactions from the user behind portlets, based on WS-PGrade and Liferay. Therefore the architecture of the MoSGrid portal complements the generic layout of gUSE (Figure 9).

Figure 9: Overall layered architecture of the MoSGrid Portal.

With the release of gUSE 3.4 and the introduction of the DCI-Bridge modifications and changes to the UNICORE Submitter were necessary. This submitter handles the communication between gUSE Services and the grid middleware layer UNICORE. In close collaboration between the Zuse Institute Berlin, MTA Sztaki Budapest, the University of Tübingen and others, a new submitter was developed, tested and deployed. The submitter is now part of the official gUSE releases (version 3.5 and higher).


WP8:JRA2 23

5.3.2 Status and Plans Beside general improvements two main goals were defined for the participation of MoSGrid in SCI-BUS: Molecular Structure Visualization and Semantic Metadata Search.

5.3.2.1 Molecular Structure Visualization After evaluation of possible solutions ChemDoddle (http://web.chemdoodle.com/) was chosen to be the main representation framework for molecular structures. Currently, it is implemented in independent portlets for testing and representation purposes. The portlet allows the upload of commonly used structure formats such as pdb, mol, or sdf. The atom type and coordinate information is converted into JSON which is natively supported by ChemDoodle and offers significant advantages concerning speed and robustness. Users can then manipulate the representation, for example, zoom in, change the background colour, canvas size or choose different structural representations. Finally, it is possible to download the high-quality image for further usage (Figure 10).

Figure 10: Snapshot of the molecular structure visualization portlet in MoSGrid based on ChemDoodle.


WP8:JRA2 24

As ChemDoodle relies on WebGL and the concepts are very similar, we decided to visualize numerical data in a similar way, using dygraphs (http://dygraphs.com/). The output of almost all molecular simulations contains one or more datasets, which are best displayed as graphs.

Figure11: Snapshot of the data visualization portlet in MoSGrid on basis of dygraphs.

Direct integration of these summary graphs collating information over very long simulation runs permits the user to verify some basic properties of the simulation at a glance.

The graph shown in Figure11 shows the course of the energy during an energy minimization. Just from inspecting this figure, it is obvious that the minimization converged and the simulation terminated successfully. No further data transfers or conversions are required from user’s side.

The visualization capabilities recently have been integrated into the monitoring tab of the simulation portlet. This enables direct access from the data browsing panel from a user typically starts the analysis of simulation results.

Future work in this area will focus on the extension of functional aspects and improvements to the user interface. For the docking domain it is anticipated to have a ChemDoodle-based visualization enabling the flipping through the highest-scoring ligands docked into a given target. We also want to add an editor function for small molecules. Here it is anticipated to directly enable a classification and metadata annotation, as described later (see next chapter).

For the testing provided by SA2 (4DSoft) relevant details were discussed. The focus is put on the appearance in different browsers, especially regarding the visualization with WebGL (ChemDoodle/dygraphs).


WP8:JRA2 25

5.3.2.2 Semantic Metadata Search In order to annotate simulation results with metadata, an XML description language was developed on basis of the Chemical Markup Language (CML). The Molecular Simulation Markup Language (MSML) is fully compatible to CML but contains additional features for improved job and workflow description. Each workflow is described by an individual MSML template containing all information related to it. This covers all simulation steps, possible settings and file transfers. But there can also be alternative templates which enable the user to customize the imported workflow. For example there might be a template for novice and for experienced users, where the latter contains additional settings and parameters. From this information the so-called PortletAPI dynamically creates all required upload fields and selection panels a user needs to specify upon job submission. All this information is appended to the template and carried along the workflow life cycle. Each individual job may be linked to parser extracting scientific information out of raw simulation results, which is also appended to the MSML template. Finally this file is converted into JSON and indexed using UNICORE capabilities. The MSML file structure thus looks like the following:

<CML> <Job List> <Job 1> <Initialization> <Uploads> <Parameter List> <Molecule structure(s)> <Adapterconfig> <Parserconfig> </Initialization> <Environment> <Parameter List> </Environment> <Finalization> <Property List> <Molecule structure(s)> </Finalizationn> </Job 1> ... </Job List> </CML>

The main element is a job list which has an entry for each job in the workflow. A job element contains an initialization, environment and finalization element. The initialization element specifies all desired user input which involves all uploads and parameters. There is also a configuration element for adapter and parser. The adapter element specifies how input shall be transformed. For example some jobs require an input file in a special format, others a string of command line arguments. The parser element is reserved for parser configurations and will be used to specify how to extract metadata from the jobs result files. The environment element contains environmental parameters like the desired number of nodes, number of cores, amount of physical memory and simulation time (wall-time). The finalization element is reserved for storing simulation results and will get filled by the parser jobs. Most of


WP8:JRA2 26

these steps already have been realised within the MoSGrid consortium. Currently, the last two steps are pending. The already existing parser needs to be extended to cope with data from various simulation programs. Furthermore the indexing needs to be implemented.

Future work will focus on finalization of all basic functionality as described above. As for all steps at least a proof of principle had been successfully made and most parts are already fully implemented, we are confident to achieve this part till end of 2012. After that a search function can be implemented and evaluated. Additionally it is desirable to link and visualize scientific information stored in MSML directly.

5.3.2.3 Community Meetings The MoSGrid Consortium is preparing multiple community meetings in order to attract new users and to provide a basic training in usage of the portal. In late September, early October 2012 the events will take place in Dortmund, Dresden, Köln, München and Tübingen.


WP8:JRA2 27

5.4 JRA2.4 Amsterdam Medical Centre Hospital Gateway (AMC) The AMC hospital gateway1 aims at facilitating data-intensive research on distributed infrastructures such as the Dutch e-science Grid and HPC Cloud. It serves mainly biomedical researchers, but also provides a collaboration platform for interaction between these researchers and the e-science experts at the hospital.

The AMC gateway serves a user community with varied profiles in neuroscience, DNA sequencing, biostatistics, e-science and distributed computing. It provides functions to manage data and results, to start and monitor computations/workflows, and to retrieve/visualize provenance of workflow executions. Liferay social networking features (communities and user roles) are used to provide customized views for: workflow developers (with WS-PGRADE and MOTEUR), brain imaging, DNA sequencing and e-bioscience users – see Figure 12. Documentation is available on-line2.

Figure 12 – Screenshot of AMC gateway, showing list of available communities

5.4.1 AMC e-bioinfra gateway Architecture Before the start of the SCI-BUS project the AMC already had a gateway3 for the e-bioinfra platform4. This system is based on the MOTEUR workflow management engine5 and a few additional components, such as the DIANE pilot job framework6, which performs meta-scheduling on gLite resources, and the Provenance system7, which collects and stores information about workflow execution using the Open Provenance Model (OPM).

1http://www.ebioscience.amc.nl/wspgrade34 2http://www.bioinformaticslaboratory.nl/twiki/bin/view/EBioScience/EBioInfraGatewaySCIBUS 3http://www.ebioscience.amc.nl/ebioinfragateway/ 4http://www.bioinformaticslaboratory.nl/twiki/bin/view/EBioScience/EBioInfra 5https://nyx.unice.fr/wikifarm/fr.unice.i3s.modalis/doku.php?id=softwares:moteur:start 6http://it-proj-diane.web.cern.ch/it-proj-diane/ 7Madougou et al Provenance for distributed biomedical workflow execution. Proceedings of HealthGrid, p.91-100,2012


WP8:JRA2 28

The goal of the AMC in SCI-BUS is to port this gateway completely to the WS-PGRADE platform, converting the workflows to WS-PGrade and the web applications into LifeRayportlets. So far, however, only a small fraction of this objective has been realized, as illustrated in Figure13. Currently the MOTEUR-based and the WS-PGrade based platforms co-exist under the same Liferay container, and the user can change from one view to the other on the “Go To” menu. The security and file management portlets are shared by both interfaces.

Figure13 – Architecture of the AMC e-bioinfra gateway. In white the new portlets developed in the first year of the project are shown.

A set of simple portlets have been developed to interface with the legacy e-bioinfra services:

The Workflow Selector (Figure14) provides filtering options (user, workflow name, status and date) or with Provenance-based options (component name and error type). The Workflow Dashboardportlet (Figure15) shows a table with all selected workflows, including status, links to logs and other monitoring information about the workflow execution.


WP8:JRA2 29

Figure14– New MOTEUR workflow selection portlet

Figure15 – New MOTEUR workflow dashboard portlet

The Provenance Viewer portlet (Figure 16) displays statistics for the workflow run (number of inputs, generated outputs, executed processes) and the execution trace for the complete workflow, including retried jobs. A graph viewer allows panning and zooming to explore the provenance information.


WP8:JRA2 30

Figure 16 – New provenance viewer portlet.

5.4.2 AMC e-bioinfra gateway Status and Plans Due to a serious problem with manpower, the gateway development has been interrupted in June 2012. Also before then the progress was behind expectation. Therefore, in the first year just a small fraction of the planned work has been achieved.

The gateway has been operational and used for various tests since May 2012 with the following characteristics:

• configuration of the gateway for the Dutch grid resources using VLEMED VO;

• definition of communities for the various user profiles

• simple portlets to access the e-bioinfra legacy services (workflow dashboard and provenance retrieval)

• usage by beta users only, namely internal members of the e-bioscience development team and advanced users who closely collaborate with the development team in DNA sequencing, Neuroscience and Metabolomics applications.

The next steps to be taken as soon as manpower has been replaced are:

Release 1

• Upgrade the installation (currently too old)

• Complete MOTEUR workflow submission portlet

• Release gateway for workflow developers (both MOTEUR and WS-PGrade)


WP8:JRA2 31

• Add new resources to the gateway (Dutch HPC cloud, local clusters)

Release 2

• Port legacy neuroscience applications to WS-PGrade platform

• Add basic data management and visualization facilities

• Release gateway for neurosciences

5.4.3 Deployment plans and instructions for SA1 The two planned releases will be initially deployed in the same server and operated by the development team. The operation instructions will be designed in a future phase.


WP8:JRA2 32

5.5 JRA2.5 PireGrid Community Commercial Gateway (UNIZAR-BIFI)

5.5.1 The PireGrid Community Commercial Gateway The PireGrid Community Commercial Gateway is a generic purpose gateway that intends to facilitate the use of the computing resources allocated at UNIZAR-BIFI for companies, researchers and users in general.

The need to create a gateway appeared within the context of the PireGrid project. It was aimed to introduce companies, especially SMEs, to distributed computing, and some applications were adapted to GRID. During the PireGrid project some demos and presentations were made, fact that awoke the interest of some companies. By that time, Cloud technologies were becoming more and more used. Some of these companies were also using Hadoop a well-known map-reduce implementation to analyze data and solve keypair-value algorithms.

Due to these requirements, we decided to implement the presented solution as an easy way of access to a Hadoop SaaS over our cloud infrastructure, adapting the existing WS-PGRADE developments.

Figure 17 – Architecture of PireGrid Gateway.


WP8:JRA2 33

Our gateway, thanks to the DCI Bridge, makes available two types of distributed computing resources:

• AraGrid: It is a scientific and technological infrastructure financed with FEDER funds. It has 4 nodes: a central node located in the overall management BIFI (Zaragoza, Campus Rio Ebro) and three computing nodes with support services, located in the Faculty of Sciences (San Francisco Zaragoza-Campus), EscuelaPolitécnica Superior de Huesca(EPSH) and EscuelaUniversitariaPolitécnica de Teruel (EUPT). Each node has approximately 480 computing cores and a total of about 2000 across the infrastructure. This is a glite based grid infrastructure which is currently part of Spanish NGI, Ibergrid and EGI initiatives.

• OpenStack cloud testbed: In the context of SCI-BUS project, a OpenStacktestbed has been deployed at BIFI because both the research and the company worlds have been demanding a cloud infrastructure that was available for developing, testing, and even for generating final results. This testbed allows the user to wake up virtual machines with different kinds of architectures, different operating systems, and so on. Currently it has around 30 nodes available.

The problem of these infrastructures is that sometimes they are difficult to use for non-experienced users. You have to know about Unix and programming concepts, you have to know how to use the terminal, managing resources, and so on. That’s the reason why projects such as gUSE-WSPGRADE were started. These projects consist of the development of gateways that work over common Internet browsers and have a friendly interface that allows the user to use these infrastructures transparently.

5.5.2 Hadoop over OpenStack specific portlet

5.5.2.1 Current status Our gateway is being developed over gUSE/WS-PGRADE. Apart from the functionalities that it offers and the infrastructures that it supports, we wanted to go one step further, supporting a new infrastructure, cloud with OpenStack. Furthermore, as our environment demanded us, we considered the implementation of the map-reduce paradigm (using the Hadoop implementation) over this virtual infrastructure, and this is what we have achieved since this project started.

gUSE/WS-PGRADE is based on LifeRay, a well-known general purpose portal, that can be easily extended with independent modules called portlets. We have developed a specific portlet that allows the user to create Hadoop jobs and run them in our infrastructure transparently. This portlet is currently being configured and installed in the production portal, and it is currently installed in two different development machines.

The first beta version is fully functional, as planned on month 12, but it still has a few bugs that are being analysed and corrected. We are planning to run the first test batch in the next two weeks, and a full demo will be shown during the Booth at EGI TF2012 in Prague the 17th September 2012.

This portlet has two main parts:

• The interface, which is divided in two tabs. The first one is a form that allows the user to submit the jobs he has prepared with all the parameters that he wants. He can specify also the number of Hadoop slaves that he wants to deploy in the virtual infrastructure. The second one is a status summary where the user can check his jobs, download the input and the results, check the logs to view the progress, etc. It uses AJAX to avoid the user to refresh the page manually, so it updates the status


WP8:JRA2 34

automatically.

• The core. It has different parts, such as an extension to the database of gUSE, a library that uses the woorea API to manage the OpenStack infrastructure from the portal, and all the functions needed to run all the process in background, showing the user all the information he needs to control it, such as real time logs.

The key of the WS-PGRADE portlet developed was to find out the best way to connect it to our Openstack cloud infrastructure using Java (the portlet is developed with Java).

After studying and analyzing different technologies like Amazon API, deltacloud, libcloud, hybridcloud... we finally select the Java binding for Openstack API (http://wiki.openstack.org/LanguageBindings) due to that this is the official way to use the Openstack API and it is one of the most up to date bindings. The way this Java binding works is creating a JSON string and then, make a call to the OpenStack endpoint using this JSON string. After studying and analyzing different technologies like Amazon API, deltacloud, libcloud, hybridcloud... we finally selected the Java binding for Openstack API (http://wiki.openstack.org/LanguageBindings) because it is the official way to do it and because it is one of the most up-to-date bindings.

The Hadoop infrastructure itself has been deployed with the preparation of an specific virual image. This image contains all the Apache Hadoop libraries needed to launch any kind of job independently of the input and output data defined.

In this first version, the image is based in OpenSuse 11.X, but any Operating System version that supports the Hadoop implementation could be used.

In order to run the users jobs, some Hadoop configuration files are created in runtime because they need to be customized according to some parameters such as the private IPs of the slaves, the arguments of the job, etc. These files are created and destroyed every time a job is run.


WP8:JRA2 35

Figure 18: Job Status

Figure 19: Portlet for Job Import


WP8:JRA2 36

5.5.2.2 Future work During the next months we are planning to make some improvements in the portlet in order to make it more robust and stable, and we are going to improve the performance as much as possible. We are going to support more ways to use big input data files because it is very expensive in terms of resources to upload and copy them each time to the server and to the virtual machines of the infrastructure.

The progress of these new developments related to data management in the cloud and specifically in the Hadoop implementation are mapped to the deployment of new features of the OpenStack cloud infrastructure which will grow as the needs of the simulations and the usage of the platform evolves.

Task Month Status Requeriments collection for portal adaption M3 !"#$%&(%gUSE and WS-PGRADE installation M4 !"#$%&)%Web Interface Design M4 !"#$%&)%Portal prototype M6 !"#$%&6%Interface implementation M7 !"#$%&7%API for Hadoop implementation M8 !"#$%&/0%Infrastructure integration M10 !"#$%&/'%Interface tests M7 1#%83"43$55%Hadoop API tests M9 1#%83"43$55%Global tests M12 1#%83"43$55%WS-PGRADE production server M12 !"#$%&/'%

5.5.2.3 Testing status The testing phase for the portal has just begun. The first version of the full working portal has been released during the 12th month of the project as it was planned, this is the reason why external tests have not been run yet. Apart from that, we have run our own tests in order to assure that each part of the development and its integration were working independently.

5.5.2.4 Usability One of the main goals of this project was to make a portal that was as easy as possible to use and quite simple. We have designed it always thinking in those terms. The parameters for creating a new Hadoop job are only the needed ones, they are divided in different sections, and they are precisely specified. All the infrastructure management is transparent for the user, and we have also taken into account some suggestions of our community of users in terms of usability such as the automated update of the status pages, the usage of AJAX, and so on.

Although we think that it is really easy to use the portal, we have published a user guide in PDF in which all the process of creating a job and running it is detailed. There is also available a tutorial video in which users can see the full process of a real execution.


WP8:JRA2 37

5.5.3 Deployment plans and instructions for SA1 The portal is currently available and fully working. The portlet developed has been already deployed in our production environment.

The Hadoop portlet has been designed taking into account that it should be easy install. This way, SA1 and JRA2 at BIFI have been working together giving one to another feedback continuously. The portlet needs to be customized and a few parameters (only the really needed ones) need to be changed for each infrastructure, but they are isolated in one single file. They are related to accessing to the OpenStack infrastructure and the virtual machines deployed.


WP8:JRA2 38

5.6 JRA2.6 Commercial Gateway for Business Process Optimization (SIMSOFT)

SimBusPro Gateway is developed to help optimizing business processes through simulation by utilizing the SCI-BUS framework. The main benefit is to simulate lots of business processes in a Grid/Cloud environment by the utilization of distributed computing and to analyze the differences between them efficiently by visualized reports. Gateway supports business process models that are modelled with BPMN2 editor and have simulation parameters. Gateway can’t be used for modelling business processes.

Current Gateway runs at http://95.9.177.225:8080/liferay-portal-6.0.5/ for test and demonstration purposes.

A simple workflow is created for gateway with one input port, one output port and one job for simulation application. All jobs of gateway utilize this workflow.

Figure 20: Simulation workflow, gateway utilizes it for all jobs.

End users only use the gateway web interfaces whereas portal administrators use the WS-PGRADE and gateway web interfaces. The Users upload their business process models as inputs to the gateway via the web interface and send them for simulation.


WP8:JRA2 39

Figure 21: Model Detail List Page, Business Processes are ready to simulation

Users monitor the status of jobs. When the status of all jobs turn into finished, they compare the results’ reports and continue this cycle as long as they want.

Figure 22: Group Report Page, Comparison report is viewed


WP8:JRA2 40

5.6.1 SimBusPro Portal Architecture

Figure 23:Architecture of SimBusPro Portal

SimBusPro Gateway is implemented as a portlet in Liferay, which utilizes gUSE with ASM API and WS-PGRADE interfaces. It has its own database.

gUSE is integrated with Simsoft Local Desktop Grid through 3g-Bridge component. Simsoft Local Desktop Grid was set up in the scope of SCI-BUS project and it is a SZTAKI Local Desktop Grid package.

Business Process Simulation Application is migrated into Desktop Grid Environment to run the simulation jobs. It is purely written in Java and interacts with desktop grid through the Genwrapper component.

5.6.2 SimBusPro Status and Plans Task Month Status Preparation of SRS M2 !"#$%&'%Installing of gUSE& WS-PGRADE M2 !"#$%&'%Installation of 3G-Bridge M2 !"#$%&'%Installing of Local Desktop Grid M2 !"#$%&'%Integration with gUSE and Desktop Grid M3 !"#$%&)%First Real Business Process Model (Tempered Glass Production )

M4 !"#$%&)%

Second Real Business Process Model (Cement Production )

M4 !"#$%&)%

Preparation of Test Plan M4 !"#$%&)%The functionality of Uploading and Simulating User Inputs

M5 !"#$%&6%


WP8:JRA2 41

The functionality of Viewing Result Report for one input

M6 !"#$%&6%

Simulation Outputs will be Migrated into Main Database of Portlet

M7 !"#$%&7%

The functionality of Viewing Result Report for Multiple Input

M8 !"#$%&7%

Parsing and Saving of User inputs within Gateway

M9 !"#$%&9%

Detail Test Cases M9 !"#$%&9%SA2 Integration M9 1#%23"43$55%The functionality of Producing Alternatives for Imported Business Process Model

M10 1#%23"43$55%

Web Interface Enhancement M10 !$*+,$-%Migrate the newer gUSE version M11 !$*+,$-%Setup of Production environment M11 !"#$%&/0%Third Real Business Process Model (Service Route)

M22 1#%23"43$55%

Extend functionality M24 %Web Interface Enhancement M24 %Improving in terms of Usability M24 %

In the first year of project development and production environments was set up. Two Core functionalities were implemented which are simulating user inputs and generating one report for multiple inputs for comparison. Implementing of third core functionality which is producing alternatives for imported business process is in progress.

SA2 integration has started and considerable amount of work was done.

Tasks which are web interface enhancement and migrate to newer gUSE version is delayed because we have decided to perform these tasks after the task of producing alternatives for imported business process model is finished.

For utilizing and testing purposes Simsoft works with an industrial partner (Aselsan Inc.) for optimization of Service Route Business Process. In the following year gateway’s functionality will be extended for providing the needs of this cooperation.

Main focus of development for following year will be user usability and improved functionality.

5.6.3 SimBusProDeployment plans and instructions for SA1 Deployment Instructions can be found in the Administration Guide. Deployment plans will be provided to SA1.


WP8:JRA2 42

5.7 JRA2.7 Blender Rendering Community Gateway (LU) Renderfarm.fi is the most accessible distributed rendering platform on the Internet. It is – and always will be - a completely free service that enables its users to render their animations or stills by using the computing power of volunteers from around the world. Renderfarm.fi encourages everybody to take part in both the online community and the collaborative rendering. By using Renderfarm.fi, 3D artists and animators benefit from being able to use higher image quality and higher resolutions when rendering. By dividing the work among hundreds of volunteer computers, an animation that takes months to render on single machine can be completed in a matter of days.

The gateway runs at http://www.renderfarm.fi and supports Blender, the world’s most popular open source 3D suite. The service has over 10 000 registered user accounts to date, though not all of those could be considered active users. After experiencing a queue of over 200 sessions during the summer of 2012, a new policy was put up. This policy is defined as: “We will not render your session if your description does not clearly state that the animation is used in some kind of production, may it be a film project, commercial work or personal portfolio. Any kind of tests will be rejected if there is anything more urgent in the queue. If the description does not state that the render is for some kind of project it is assumed to be a test render.”

End users use the service via a specialized uploader script developed specifically for Blender. The upload is entered in to the queue and after the administrators check the session to be valid and secure, the session enters the pipeline, is distributed to the rendering clients and rendered. Each finished frame is available nearly immediately to the end user upon completion. When the whole session is completed, a low-resolution animation of the session frames is created and placed in to the session gallery for all users (including the volunteers) to see.

5.7.1 Blender Rendering Community Gateway Status and Plans Task Month Status gUSE installation M2 !"#$%&'%Cycles renderer support M6 !"#$%&6%BURP features M7 !"#$%&:%BURP gUSE Manager M9 !"#$%&9%Gateway frontend improvements M6/M1

2-M14 1#%23"43$55%

BURP stabilization M11 1#%23"43$55%Uploader improvements M12 1#%23"43$55%gUSE client M14 ;"<%5<+3<$-%

First year development efforts for the BURP/BOINC (volunteer computing) based Blender rendering gateway in SCI-BUS have focused on improving the user experience and providing a service level comparable to non-volunteer computing based rendering solutions. WS-PGRADE/gUSE integration is also being carried out specifically for the purpose of rendering simulations.


WP8:JRA2 43

The introduction of the new Blender Cycles rendering engine (a more realistic and modern alternative to the Blender Internal rendering engine already supported) has increased demand for a solution such as Renderfarm.fi. This trend continues as for the first time in the service’s history the rendering queue is nearly a month long as the amount of available work units supersedes the computational power of the participating computers.

The web backend is currently being redesigned using the Django framework as the CMS used up to this point, Drupal, has proven to be sub-optimal for the users (both artists and volunteers) needs and is hard to maintain. Further development efforts have also been put into the rendering client, though virtualization (for added security) efforts have proven unsatisfactory so far.

5.7.2 Blender Rendering Community Gateway Architecture

Figure 24:Current architecture of the Blender Rendering Community Gateway

The current architecture of the service does not offer WS-PGRADE/gUSE integration, but relies entirely on the legacy BOINC/BURP solution and volunteer computing. The web frontend is handled by Drupal.


WP8:JRA2 44

Changes affecting the legacy architecture and UX within the first year of SCI-BUS:

• Blender Cycles rendering engine integration

o Subsampling on the grid

! Divide sample work over parts. Example: if 100 samples is wanted in final result then we can divide into i.e. 10 render instances with different seed. Results will be merged into the final part on the server.

! Caveat: cannot accept sessions with keyframed seed and sample count.

• Uploader improvements

o Better handling of user credentials

o Login once

o Refactor of code into module

o pySpace?

• BOINC

o Changes to validator and assimilator to use BURP XMLRPC API

• BURP

o Extend XMLRPC API to get rid of Java-PHP bridge (in progress)

o Jam handler - automatically detect and resolve slow and broken work units

o Stage queue - allow admin to accept more session than can be put on the grid

! Some admin configurable limits have been introduced with which the amount of outstanding work units can be controlled. Currently around 40k-50k work units are on the grid at any time (provided there are enough sessions in the backlog too)

o Cleanup handler - automate disk space recovery

o Validator based on Perceptual Difference (http://pdiff.sourceforge.net) for sessions with replication larger than 1. Todo still: expose all pdiff parameters per session to the admin UI

• New Django-based frontend prototype with single-page, javascript driven site

• pySpace development for Blender, to allow custom editors and tools as spaces, WIP.


WP8:JRA2 45

Figure 25:Targeted architecture for the Blender Rendering Community Gateway

The appended architecture offers a WS-PGRADE/gUSE component for the computation of simulations. The web frontend has been switched from Drupal to a custom Django (Python) and javascript based solution.


WP8:JRA2 46

5.8 JRA2.8 Citizen Web Community Gateway (EG) The Citizen Web Community Gateway is a new R&D project of E-Group in the domain of Document Management Systems. E-Group has already developed a complex Document Management System called eDOX. The Citizen Web Community Gateway will provide access to the eDOX functionality, described below

The aim of the Citizen Web Community Gateway is to act as a gateway between the paper-based and the electronic world. Nowadays, converting paper-based documents into authentic electronic documents that also support content processing ever more important both in the government and business sectors. Paper to electronic document processing is also mandated by legislation therefore conformance to regulations will be guaranteed during R&D.

5.8.1 eDOX Workflow and Architecture The exact steps of document processing are the following:

Preparation of Scanned Document module

• The content of the scanned documents have to be split to separate pages. This is the base requirement for processing documents on parallel threads.

Professional OCR/ICR module

• The content of the scanned documents have to be recognized to be able to process it automatically.

• This step needs OCR (Optical Character Recognition), ICR (Intelligent Character Recognition) or perhaps FR (Form Recognition) solutions for analyzing typewritten or handwritten characters (or maybe voice recognition solution in case of audio files). Using language-specific dictionaries can enhance the rate of recognized words and expressions, but this step is computationally intensive. This is where we intend to make use of the Grid and Cloud resources that are made available through SCI-BUS technology to accelerate the process.

Indexing and Metadata Processing module

• The metadata of the scanned documents must be given in order to enhance the functionality of searching and retrieving data.

• This step needs human interaction to validate the automatically generated metadata. If automatic metadata would not be available, the human resource administrator would need to enter the whole metadata structure by hand, which is a very time consuming process. However, an intelligent application (e.g. by using form recognition or keyword analysis) can automatically extract all metadata, based on parameters such as type of document (e.g. invoice) or issue date (e.g. 2011-10-01).Such automated metadata analysis is also computationally intensive, where the usage of Grid or Cloud resources can again significantly accelerate the process.

Cryptographic Service Provider module

• The scanned document and its metadata structure have to be packaged and signed. • This step needs a cryptographic module such as OpenSSL or SDX (Signed

Document eXpert, an E-GROUP product). SDX is fulfills international legal and


WP8:JRA2 47

technical requirements and creates the necessary XML electronic signatures. The signatures have to be verified (after a revocation period) in order to embed CRLs (Certificate Revocation Lists) or OCSP responses. If the electronic signature is valid, the related paper-based document can be discarded (shredded).

WEB interface

+

splitdocument

user

signing

inverted indexMIReG

metadata

CLOUD

DB

pages HTML/DOCXwith layout/style

PDF/TIFF document

metadata structure and

rule set

signed documentwith metadata

WS-PGRADE/gUSE/DCI-Bridge

CloudBroker

OCR+

spellchecker

indexing+

gatheringmetadata

Remote API

Figure 26: eDOX architecture.

5.8.2 Who can use the portal The document management portal can be used

• directly by registered and successfully authenticated user via web-based GUI or REST API,

• from an electronic records and document management system solutions which communicates with this portal in the background.


WP8:JRA2 48

5.8.3 Registering and logging in The registration shall be performed at E-Group Administration. For more information please visit E-Group official website or contact us via e-mail or phone.

5.8.4 Using the portal This Section contains a description of the user interface.

The web-based GUI can be accessed after successful user authentication:

Figure 27: User Authentication

Next, the user is asked to select a file (output of scanner: e.g. pdf, tiff) to be uploaded and selected by pressing „Upload” button:


WP8:JRA2 49

Figure 28: Selection of files

After that the process starts automatically:

Figure 29: File uploaded and process started

On the next screen “Process”, all of the files, documents are listed, which are being processed. The metadata of the documents consist of the necessary fields specified by the adequate Hungarian regulations:

• Filled out by the administrator (on an admin interface): a. Physical size of the paper document; b. Name of the copying institute and person; c. Name of the copying system and regulation;

• Blank: a. Document type b. Creation date


WP8:JRA2 50

Figure 30: Status: In Progress

So, the user can see, that some of the metadata haven’t been yet filled: this is exactly the point of the processing, the module automatically recognizes them.

Figure 31: Documents processed

On this screen the user can see all of the necessary indexing fields (metadata). In addition the system creates the resulting digitally signed file. Now the user can get rid of his/her paper based document. Of course he/she have to archive from now on the electronic version. It can be done either by eDOX Archiver or by him/herself after downloading the signed package.


WP8:JRA2 51

5.8.5 Security The following security-related functions are applied:

• user authentication: username/password is sent via an SSL/TLS-protected communication channel where SSL/TLS client authentication shall be performed (SSL/TLS client certificate is issued during user registration process),

• protected external communication: SSL/TLS-protected communication channel is used to submit data to third-party solutions (gUSE/WS-PGRADE/DCI-Bridge of SZTAKI).

5.8.6 eDOX Status and Plans The most important planning steps are summarized in the table below.

Task Month Status Planning workflow based on legislation &/% !"#$%&/%Testing OCR softwares &(% !"#$%&(%Implementing“Preparation of Scanned Document module”

&)% !"#$%&)%

Implementing“Professional OCR/ICR module”

&6% !"#$%&6%

Implementing“Indexing and Metadata Processing module”

&9% !"#$%&9%

Implementing“Cryptographic Service Provider module”

&/0% !$*+,$-%

Remote API implementation &//% !"#$%&/'%First release (based on D8.1 and D8.2) &/'% !"#$%&/'%

5.8.6.1 Implemented in 1st release The table at above shows the functionalities that were implemented in the first release. One step of the whole workflow was delayed. The reason of that was E-Group wanted to integrate its own cryptographic module, called SDX. The SDX application runs in Microsoft Windows environment, but the virtual images available at CloudBroker do not use such operating system. This means that “Cryptographic Service Provider module” will have to be redesigned to use e.g. OpenSSL instead.

5.8.6.2 Planned for future releases We will enhance the algorithms that analyze file contents. Recently the algorithms just use regular expressions (“regexp”) and dictionary files, but in future versions also the style/layout information and other logics shall be used to retrieve successfully the needed metadata.

5.8.6.3 Testing status The tests covered just interface integration and basic functionality. In the future deeper test will have to be performed to verify the whole system.


WP8:JRA2 52

5.9 JRA2.9 Astrophysics Community Science Gateway (INAF) INAF has integrated the existing VisIVOWeb and VisIVO Smartphone applications with the WS-PGRADE generic gateway to offer new, exciting and easily accessible opportunities not only to specialized users, e.g. astrophysical researchers, but also to the wider public, e.g. high-school education or innovative citizen science for new scientific developments. The integration with WS-PGRADE/gUSE infrastructure enables the porting and running of VisIVO tools8 on DCIs for large-scale data analysis and exploration in astrophysics and many other scientific disciplines.

5.9.1 VisIVO Science Gateway The new portal is named VisIVO Science Gateway and it is running athttp://visivoportal.oact.inaf.it:8080. INAF has also a development version of the portal for development and testing purposes available at http://visivo-server.oact.inaf.it:8080

Figure 32: A VisIVO Science Gateway typical working session

Several workflows were designed and implemented in the WS-PGRADE based gateway. Some workflows were implemented for local and remote importing of the users’ datasets, one workflow for filtering operations and three workflows for scientific movie creation. The following two figures shows two workflows: one for the remote importing of datasets and the other for a panoramic movie creation. The remote importing workflow allows to produce significant information for meta data exploration such as basic statistics on data values, histogram calculation and plotting or a sample extraction of uploaded datasets. This meta data are available through the gateway and some of them can be modified by the user (e.g. renaming the VBT or the related fields). 8http://sourceforge.net/projects/visivoserver/


WP8:JRA2 53

The panoramic movie workflow generates four movies with different camera position paths on the generator port: from 0° to 360° azimuth rotation, from 0° to 90° elevation rotation, from 90° to -90° elevation rotation and from -90° to 0° elevation rotation. The four movies generation executed in parallel is then merged thanks to the collector port.

Figure 33: two VisIVO workflows for remote dataset importing (left figure) and

panoramic movie creation (right figure). VisIVOportlets were implemented to easily submit those workflows:

• VisIVO Importer: converts user datasets into VisIVO Binary Tables (VBTs) internal format and extract all necessary meta data providing the dataset for further exploration into the portal.

• VisIVO Filters: converts imported datasets from an original data table into a new one performing several operations (e.g. can create a Volume from a table dataset)

• VisIVOViewer :displays customized views of 3D renderings. • Properties : displays the meta data of the imported datasets and produced images. • Data Management : allows the management of imported datasets and produced

images and movies and allows the navigation within the different services offered by the portal connecting to the VisIVO DB.

• Panoramic and Dynamic Movie : creates scientific movies starting from a produced image.

• Custom Movie : creates scientific movies from all produced images from an imported dataset.


WP8:JRA2 54

5.9.2 VisIVO Mobile The VisIVO Mobile application allows smartphone devices to exploit VisIVO Gateway functionalities to access large-scale astrophysical datasets residing on a server repository for analysis and visual discovery. Through interactive widgets, customized visualizations (images or movies) can be generated and stored on the remote server. The application notifies users when requested visualizations are available for retrieving on their smartphones and allows easy sharing of data, images and movies via e-mail or common social networks.

Figure 34: some screen shots of VisIVO Mobile application: authentication, data

management, image visualization and sharing


WP8:JRA2 55

5.9.3 VisIVO Architecture

Figure 35: Architecture schema of VisIVO Gateway connected with VisIVO Mobile

In VisIVO Gateway, advanced users are enabled to create, change, invoke, and monitor workflows accessing to all of the components of WS-PGRADE/gUSE, while standard users are provided with easy-to-use specific web based user interfaces hiding all the technical aspects of the visualization software and DCIs configurations and settings.gUSE Application Specific Module API has been included to the VisIVOportlets to reuse the implemented workflows stored in the local repository of gUSE. VisIVO Mobile configure and submit workflows residing on the VisIVO Gateway by means of the gUSE Remote API. These API enables the interface to the core gUSE services without the WS-PGRADE user interface component. Thus, they allow running and managing workflows by command line solutions consisting of curl based access wrapped in shell scripts.

5.9.4 VisIVO Status and Plans The most important planning steps are summarized in the table below.


WP8:JRA2 56

Task Month Status WS-PGRADE M2 !"#$%&'%Purchase new Hardware M2 !"#$%&'%Web Interface Design M2 !"#$%&'%gUSE Remote Api M3 !"#$%&)%VisIVO Mobile Design M4 !"#$%&)%VisIVO Gateway Internal Testing M4 !"#$%&)%Testing plan SA2 M4 !"#$%&)%Workflows Design M6 !"#$%&6%VisIVO Importer Workflow M9 !"#$%&9%VisIVO Filters Workflow M9 !"#$%&9%VisIVO Movie Workflows M9 !"#$%&/0%Web Interface: Update Layout M10 !"#$%&/0%Web Interface: Cleanup code M10 1#%23"43$55%Testing with SA2 M10 1#%23"43$55%

The most important tasks has been done on time. The portal has been ported, set up and is being used in production by INAF. The development will continue on workflows that matter most to the astrophysics community but also for other communities belonging to other visualization projects.

Status:

• The VisIVOWeb portal has been moved to WS-PGRADE and gUSE infrastructure.

• A development and a production instance have been set up.

• The very same portal will be operated by SA1. Currently whoever is interested in the usage of the VisIVO Gateway can create an account.

• Testing of VisIVO gateway has started in cooperation with SA2, whose members are able to access the gateway.

• The VisIVO Mobile is already connected to the portal enabling user data exploration and visualization.

Plans:

• Extend functionalities of the gateway and mobile application in terms of workflows and interfaces to support other user communities.

• Improve the usability of the gateway and mobile application. • Organize an international competition to use the gateway for creating the most

challenging scientific movie.


WP8:JRA2 57

5.10 JRA2.10 Build and Test Public Gateway for Application Developers (4D)

The 4D ETICS Build and Test Portal existed as a GWT-based web application submitting jobs on Condor/NMI based job submission middlewares. We have integrated the existing portal with the DCI Bridge job submission service provided by gUSE to extend the existing job submission capabilities with a more flexible, user centric cloud infrastructure. The motivation to move to cloud is that we are going to run the Gateway as anSaaS serving a large number of communities. Cloud integration is the most important implementation task to achieve this goal.

The upgraded portal is available at: etics3-test.4dsoft.hu

5.10.1 4D Build and Test Portal Architecture

Figure 36: Test Portal Architecture

In order to implement this architecture service level DCI Bridge integration is required with the already available build and test services, such as configuration, job submission and repository. As DCI Bridge is not released as a separate component, to have all this in place an almost full gUSE portal (without the front-end) is needed. Additionally, in most of the time, for the implementation of the gateway a development version of the gUSE containing the recent implementations of CloudBroker plugin has required.

The jobs are described injsdl - job description language applied for the most of DCI. Current version of the Gateway can submit jobs through the DCI Bridge to the local resource and to different cloud platforms provided by CloudBroker plugin. Current version of the Gateway do not integrates the billing and pricing portlet of the CloudBroker.

The web interface of the Build and Test Portal make use the newly implemented service integration. Now the users can select cloud resources as option when configuring build or test jobs. Then they can follow the status of the jobs in the 'Submission' portlet (see Figure 38 and Figure 40). To perform this operation a new service has been implemented. Finallythe


WP8:JRA2 58

results can be downloaded from the 'Repository' portlet (see Figure 40), as the results of the job's running are registered in the Registry.

The DCI Bridge and CloudBroker related service level implementation has been done in close collaboration with JRA1 people.

5.10.2 4D Build and Test Portal Status The most important task of our Gateway has been to realise cloud connection through DCI Bridge and its CloudBroker plugin. This important task was realised in time, the first version is already available. Taking into account the financial terms, extensive testing could be started when the open source cloud infrastructure (OpenStack) will be added. JRA1 works on that task. This task required lot of effort as DCI Bridge at the time of writing this deliverable has not been released as a separate component. Also the JRA1 cloud integration has been an on-going task during the implementation phase.

For the remainder of the project we are going to test extensively the cloud integration. An additional task is to improve the currently existing UI and to add more community functions.

Tasks

The table below summarises the most important planned tasks. Then there is a section on the current status and a short section on the next years plan.

Tasks Month Status WS-PGADE, gUSE installation M2 Done M2 Initial design of the new submission service M3 Done M3 Test Plan M4 Done M4 Redesign of the submission service M5 Done M5 UI changes M6 Done M7 gUSE, WS-PGRADE portal built from the CloudBroker integration development branch

1st year release Done M8

jsdl based job description adoption 1st year release Done M10 Job monitoring & register thread implementation 1st year release Done M11 Registration in repository 1st year release Done M12 Deployment tests 1st year release Done M10 Web tests 1st year release In progress UI enhancements 2nd year release In progress Mobile Interface 2nd year release in progress

Status

• The CloudBroker platform has been investigated (scibus.cloudbroker.com and platform.cloudbroker.com).

• The special gUSE version containing the most recent cloud integration is available (gUSE build from branch). No need to wait for the gUSE release to continue our Gateway's implementation.

• The service level integration of DCI Bridge and our Gateway's job submission service is done. The necessary UI changes for cloud set up have been done.


WP8:JRA2 59

• jsdl based job descriptions have been adopted

• A new job status monitoring and register service has been implemented

• Deployment packages and (build, tests) reports registration to the repository is done.

• A public instance containing the new implementations (etics3-test.4dsoft.hu) has been set up. Currently whoever is interested in the usage of the Gateway can initiate the registration process. We are going to improve that process to have a more straightforward registration in the next coming period.

• Testing has been started. Initial test plan was created. In this period, the focus has been on deployment tests. Extensive web acceptance tests could be implemented when the open source cloud infrastructures adoption will be available.

Figure 37: Selecting the new dci-bridge submitter when submitting a remote 4d etics build.


WP8:JRA2 60

Figure 38: The job sent to the dci-bridge was running!

Figure 39: The job sent to the dci-bridge had been successfully finished!


WP8:JRA2 61

Figure 40: The results of the new dci-bridge job had been registered to the 4D ETICS repository.

Plans

• Upgrading the current cloud integration functionality.

• User interface improvements to be able to operate the Gateway as aSaaS.

• Mobile interface for Dashboard and a very simple interface for job submission. No job configuration is planned.

• Web acceptance tests on open source cloud platforms.


WP8:JRA2 62

5.11 JRA2.11 Helio-Physics Gateway (TCD) The Heliophysics Advanced Propagation Model Portal is built on existing heliophysics services and offers an advanced propagation model.

Propagation models are a fundamental and very active field of research in heliophysics; they model the propagation of events and physical features such as Coronal Mass Ejections (CMEs), Solar Energetic Particles (SEPs) and Solar Wind (SW) across the solar system. The aim of such models is to discover the correlation between events in the Solar system.

The advanced propagation model offered by the portal offers the possibility of running the propagation model for large number of events for statistical studies and the possibility of finding optimal parameters for the models by looking at events at the sun and on different other locations across the solar system.

• The first step is the selection of a range of interest and the catalogue(s) that must be used to identify the relevant events. This step is represented in Figure 41 and Figure 42.

Figure 41, first step of the Advanced Propagation Model – Selection of time range

Figure 42, Interface for the selection of time range and catalogues


WP8:JRA2 63

• The second step (Figure 43 and Figure 44) is the identification of significant events within the selected time range.

Figure 43, second step of the Advanced Propagation Model – Selection of time range

Figure 44, List of events at source.

• During the third step (Figure 45 and Figure 46), the propagation model is executed for each event with different parameters.


WP8:JRA2 64

Figure 45, Third step of the Advanced Propagation Model – Execution of the propagation model for each event.

Figure 46, Selection of parameters sweep for the propagation Models

• During the fourth step (Figure 47), events are found within the ETA ranges


WP8:JRA2 65

Figure 47, fourth step of the Advanced Propagation Model – Selection of optimal parameters

Figure 48, event lists at the targets.


WP8:JRA2 66

5.11.1 Helio-Physics Portal Architecture

Figure 49, Architecture of the Prototype

The architecture of the first prototype is sketched in Figure 49.

The advanced propagation model exposes two interfaces: a portal hosted in a Liferay environment and a Web-Service Interface. Both interfaces are connected to the controller that uses the resource and service manager.

The controller hosts the logic of the service, it determines the number of executions of the propagation model from the event lists obtained from the event catalogues, it controls the execution of the propagation model and it defines the ranking of the events used to validate the results in the last steps.

The service manager wraps the client to the legacy HELIO services such as the HEC (HELIO Event Catalogue) and it extracts suitable information from the VOTable returned.

The resource manager acts as an abstraction layer to the processing service inherited from HELIO and extend it with the services offered by SCI-BUS. There are now two computational resources: a gLite grid and a fast computational resource that executes light jobs without the extra time required by the gLite middleware. These two middleware are connected through the HELIO Processing Service (HPS).


WP8:JRA2 67

By the end of 2012, TCD will decommission its Grid Middleware and will partly devote a cluster powered by torque to the Advanced Propagation model portal. The connection will be performed through the gUse libraries offered by SCI-BUS.

The propagation model itself is now a set of routines written in IDL that use a library called SSW.

For the first proof of concept the connections in red have not been completed. The full prototype is expected by M15 (with a delay of 3 months).

5.11.2 Helio-Physics Portal Status and Plans The most important planning steps are summarized in the table below.

Task Month Status WS-Pgrade and gUse connection with IDL environment

M9 !"#$%&9%

Portal proof of concept M9 !"#$%&/'%First prototype M12 1#%83"43$55%

=&/.>%Features of Features for the second prototype

M15 %

Implementation of second prototype M20 %

At the moment the portal is about three months late as the phase of inception and definition of features took longer than expected.

The proof of concept of the portal has been showed to the community in occasion of the fourth Coordinated Data Analysis Workshop(CDAW-4), hosted by the School of Physics in Trinity College Dublin. Three main actors of the community showed interest in the portal: The Observatory of Paris, The University of Helsinki and the School of Physics in Trinity. These three partners will be involved in the definitions of the features of the second prototype.

The next steps will be:

1) First complete prototype with connection through the torque cluster through gUse. Expected to be complete by month 15 (delayed from month 12).

2) Finalization of the features of the second prototype with the help of the members of the community (University of Helsinki, Observatory of Paris and School of Physics in Trinity College Dublin)

3) Implementation of the second prototype.