© 2013 hp development company l.p. the contrail demonstrator and other use cases christian...

© 2013 HP development company L.P.

The Contrail Demonstrator and other use cases

Christian Temporale, Hewlett [email protected]

1

contrail is co-funded by the EC 7th Framework Programme under Grant Agreement nr.

257438

contrail-project.eu


Why use cases?

• To evaluate, test and demonstrate the technology developed in the core research Work Packages.– Doing experiments and extensive tests on the CONTRAIL system– Report usability issues, suggest improvements, identify bottlenecks– Demonstrate that the project have both a scientific and business

impact

• Identify new requirements• Refine existing requirements

– Users’ feedback collection and analysis

2


Contrail use cases

• Distributed Provision of Geo-referenced Data– Implementation of a Virtual 3D Tourist Guide with an embedded 3D Digital

Globe, that let end users to navigate through data provided by federated Content Providers

• Multimedia Processing Service Marketplace– Implementation of a web portal - that serves as interface for end users,

whereas the backend interact with federated Content and Technology providers – offering multimedia services to end users

• Scientific Data Analysis– Implementation of a scientific data analysis service, mainly oriented to

neutron scattering community

• Electronic Drug Discovery – Bioconductor (†)– The Electronic Drug Discovery enables companies in the life science sector to

use modern bioinformatics tools/applications on a federated cloud system

3


UC1 - Distributed Provision of Geo-referenced Data

• Virtual 3D Tourist Guide (Google Earth like)– Users navigate the 3D digital globe via web browser– Detailed maps and points of interest

• Data set (from Data providers)– Geo-spatial data provided by SDIs (Spatial Data Infrastructures)– Geo-referenced data provided by Content Providers

• SDIs and Content Providers– Become part of the Contrail Federated Cloud– Get IT infrastructure from the Contrail Federated Cloud

4


UC1 – Data and content layers

5


UC1 - Distributed provision of geo-referenced data

Contrail Federated CloudContrail Federated Cloud

Private cloudPrivate cloud

InternetInternet

Public cloudPublic cloud

Private CloudPrivate Cloud

Data ProviderData Provider

VIN

Application ProviderApplication Provider

Configuration file

Data ProviderData ProviderData

ProviderData Provider




6


UC1 - Architecture

7


UC1 – Deployment in Contrail

8


UC1 – Main benefits from Contrail

• Fully decentralized approach to 3D Virtual World rendering• Enables Data providers to autonomously manage their

proprietary data in the federation• Choice of location storage inside the cloud

• Elasticity: scale services for traffic spikes– Peak load management

• Reduction of initial investments in IT infrastructures for small data providers

• Quick and efficient processing of complex raw geospatial data

• Support of constraints on data storage• Data protection

9


•Transform the selected Use Case into a market oriented demonstrator, open to final users

•Prove the concept of Contrail in a nearly production scenario, with all security and usability features

•At the end of the project the demonstrator will become full featured and can be easily transformed in commercial products by the industrial partners

Goals of the demonstrator

10


UC2 - Multimedia processing service marketplace: end user standpoint

end user

sign-up to System

subscribe to services

use contents

download

streaming

search contents

advanced search

enrich contents

11 contrail-project.eu

contents proposition

advanced features


UC2 - Actors

MultimediaMarketplace

Content Provider

TechnologyProvider

searches / uses / enriches contents

administers users, services, contents

requests digital transformation

provides adapted contentsFinal user

Administrator

searches / requests contents

providescontents

contrail-project.eu12


UC2 - Architecture

web server

Advanced content search

system

Content providers

Technologyproviders

Client

Data mining system

s

Presentation layer (V)

Business layer (C)

DAO layer (M)

Media digital processing broker

User data processing

+ AC File Payments

No SQL DB

13


UC2 - Advanced content search component


images DB (faces, keys)

Face detection, face recognition

Model training

Advanced content search system

New content (video)

Image and video

processing

Machine learrning


UC2 - Advanced content search – Example 1


Marketplace Business layer

Advanced Content Search

images DB

DoctorStrangelove.avi

Marketplace front end

“Peter Sellers”

Content Provider

Doctor Strangelove link

machine learning


Marketplace DB

314 1

2

3

4

5

6

Sellers.jpg

Name, code

Name OCV ID

Sellers 314

Content Tags

Doctor Strangelove

Bombs,Sellers

update tags

retrieve name

7

8

9

Get contents by tags (Sellers)

Face ID

314

Doctor Strangelove

1

2

3

4

5

6

video recognitionrequest

front-end activities

back-end pre-processing


UC2 - Advanced content search – Example 2


Marketplace Business layer

Advanced Content Search

images DB

DoctorStrangelove.avi

Marketplace front end

Content Provider

Doctor Strangelove link


Marketplace DB

314

Name OCV ID

Sellers 314

Content Tags

Doctor Strangelove

Bombs,Sellers

retrieve name

Get contents by tags (Sellers)

Face ID

314

314

Doctor Strangelove

image recognitionrequest

Update DB

Sellers

11

12

1

2

9

8

7

6

54

3

10


Data mining: Apache Mahout


• Mahout is a rich set of distributed scalable machine libraries that extracts insights from “Big data”

• Mahout’s core algorithms for clustering, classification and batch-based collaborative filtering are implemented on top of Hadoop using Map Reduce

• Mahout features used in the demonstrator:• Recommendation mining - based on users’

behavior• Data classification – based on existing

categorized contents


UC2 - System deployment (3rd rel.) - Current plan

s

Presentation layer (V)

Business layer (C)

DAO layer (M)


User data processing

+ AC File Payments

MySQL SQL DB

Technology Provider

contrail DaaS

Content Provider

WS

Advanced content search

HTTP

Data mining system

Contrail Fed. / conpaas

Contrail Fed. / conpaas

Contrail distr. storage

GAFS

Contrail VEP

Contrail VEP

18

SQL server manager

conpaas SQL service

conpaas web application hosting service

conpaas map-reduce service

conpaas KV store


UC2 - Deployment examples

• DB deployment on ConPaas• Web application deployment on ConPaas

19


UC2 - Main benefits from Contrail• Security

– Confidentiality and protection of whatever users’ data (contents, personal data, usage data)

– Legal constraints on data storage (e.g. geographical location)

• Reliability– Content storage reliability (→ SLA)– VM reliability (→ SLA)

• Performance– Bandwidth availability (→ SLA)– System response time (→ SLA)

• Contrail Federation – Technology providers immediately enabled to sell their know-how directly to

external operators (e.g. the Marketplace) without needing to invest on development of products accessible by end users

– Content providers need trustable storage infrastructures to preserve their data (their business comes from the commercial value of their contents)



UC2 - Other Contrail benefits

• ConPaaS services – Marketplace deployment (elasticity + scalability)

• Structured/distributed storage + GAFS – Contents localization– Legal restrictions on stored contents

• SQL servers (High Level Services) – No need to set up new DBs.

• Map/Reduce services – Advanced usage data processing– Extraction of statistical information from log files– Search indexes building

• VIN / VCP – Advanced configuration of network topology



UC3 - High Performance Real-Time Scientific Data Analysis

• Analyses data from neutron scattering experiments on biological samples at the ISIS facility at STFC

• Runs a large number of model-fitting simulations to determine geometrical shapes of chemical samples

• Presents a figure of merit for each simulation marking its closeness to the experimental data

Small AngleNeutron Scattering

22


UC3 - Neutron source data

• Complex parameter space for modeling analysis of biological structures

• 1,000 to 100,000 simulations to determine the molecular structure of biological samples

• Currently, only the most experienced researchers know how to navigate the parameter space

23


UC3 - Scientific data

• Experiments are expensive– Booked months in advance– Need to locate “interesting” parameters– Feed experiences back into model

• Manual search can take weeks– Large amount of time taken in trial-and-error– “Close enough” may result in only 10% of the experimental data

being used – wasteful

24


UC3 - Architecture

• SDA portal:– Simulations of target

structures based on complex parameter exploration

– Statistical analysis of generated data, allowing ranking on multiple aspects

– Feedback to user as to whether more experimental data is required in order to fully categorize the sample

25


SDA PortalConPaaS web application

hosting service

UC3: System deployment

ConPaaSBag-of-Tasks service

Function: Run a simulation task for each combination

of parameters

ConPaaS Map/Reduce service

Function: Analyze the simulation results and provide a ranking compared to

experimental data

GAFS

User defines parameters and experimental data

Location of simulation results in GAFS is a parameter to Map/Reduce

GAFS is used to store experimental data, simulation results, ranked data, and metadata (parameter settings, etc)

26


UC3 - Main benefits from Contrail

• “Horizontal scaling”– Allocate additional VM instances as needed (not like Grid, previously used)– Estimate expected completion time

• Predicable turnaround time → productivity

• Performance – Grid technologies did not guaranteed system responsiveness

• Single-Sign On• ConPaaS

– Bag-of-Tasks– Web application– MapReduce

• GAFS for sharing data and simulation output• Infrastructure reliability• Security• SLA Enforcement

27


WP12: Use Cases and Applications –CONTRAIL features in Use Cases

Features exploited in release 2.0 of Use Cases

Functionality Use Case 1 Use Case 2 Use Case 3 Use Case 4

Cloud Federation

SLA

Network Virtualization (VIN)

Storage Virtualization (GAFS)

VEP Computation Virtualization (IaaS)

ConPaaS SQL (High Level Services)

ConPaaS Web Servers (Runtime Environment)

ConPaaS Task Farming (Runtime Environment)

Monitoring

Identity Management (AuthN, AuthZ, Identity redirection)

Security and QoP

Map/Reduce

Accounting

28


WP12: Use Cases and Applications – CONTRAIL features in Use Cases

Features exploited in release 3.0 of Use Cases

Functionality Use Case 1 Use Case 2 Use Case 3

Cloud Federation

SLA

Network Virtualization (VIN)

Storage Virtualization (GAFS)

VEP Computation Virtualization (IaaS)

ConPaaS SQL (High Level Services)

ConPaaS Web Servers (Runtime Environment)

ConPaaS Task Farming (Runtime Environment)

Monitoring

Identity Management (AuthN, AuthZ, Identity redirection)

Security and QoP

Map/Reduce

Accounting

29


Thank you!

30


Funded under: FP7 (Seventh Framework Programme)

Area: Internet of Services, Software & Virtualization (ICT-

2009.1.2)

Project reference: FP7-IST-257438

Total cost: 11,29 million euro

EU contribution: 8,3 million euro

Execution: From 2010-10-01 till 2013-09-30

Duration: 36 months

Contract type: Collaborative project (generic)

contrail is co-funded by the EC 7th Framework Programme

31

http://contrail-project.eu

contrail-project.eu


Backup slides

© 2013 hp development company l.p. the contrail demonstrator and other use cases christian...

Documents

uc1 data

data storage data protection

data set

proprietary data

hp development company

small data providers

contrail demonstrator

contrail use cases