etadatasynchronization ack cyfronet agh kraków, …cgw'18, october 22-24, krakÓw. c....

CGW'18, October 22-24, KRAKÓW

METADATA SYNCHRONIZATIONPROTOCOL FOR ADECENTRALIZED NETWORK OFDATA PROVIDERS

Łukasz Opioła | Łukasz Dutka | Renata G. Słota | Jacek Kitowski

ACK CYFRONET AGH, Kraków, Poland

AGH University of Science and Technology, Kraków, PolandFaculty of Computer Science, Electronics and TelecommunicationsDepartment of Computer Science


Global data access for modern science

Challenges of global data access

Metadata synchronization protocol - requirements

Proposed concept of metadata synchronization protocol

Conclusions

1

2

3

4

5

AGENDA


GLOBAL DATA ACCESS FOR MODERN SCIENCE

GROUP 1

GROUP 2

GROUP 3


CHALLENGES OF GLOBAL DATA ACCESS

GROUP 1

GROUP 2

GROUP 3



GROUP 1

GROUP 2

GROUP 3

Transparent data access

Bob

Bob



GROUP 1

GROUP 2

GROUP 3


Cross-border collaboration

BobAlice

Bob



GROUP 1

GROUP 2

GROUP 3



Security & efficiency BobAlice



GROUP 1

GROUP 2

GROUP 3



Security & efficiency

Openness of the network

BobAlice

GROUP 4

James



GROUP 1

GROUP 2

GROUP 3





Autonomy (lack of trust)

GROUP 4

! !!

!

!

!



GROUP 1

GROUP 2

GROUP 3





Autonomy (lack of trust)

Reflecting existing hierarchiesGROUP 4


METADATA SYNCHRONIZATION PROTOCOL


GROUP 1

GROUP 2

GROUP 3

GROUP 4

METADATA SYNCHRONIZATION


Decentralized (retaining autonomy)

Globally consistent

Scalable – hundreds of providers

Secure – decentralized AAI*

Location discovery mechanisms

METADATA SYNCHRONIZATION - REQUIREMENTS

* AAI – authentication and authorization infrastructure


PROPOSED SOLUTION

Metadata synchronization protocolbased on

hybrid, multi-tier architecture


Gathers knowledge using metadata sync

Manages data on underlying storages

PROPOSED SOLUTION – PROVIDER SCOPEProvider service – basic building block

AAI – Authentication and Authorization Infrastructure


PROPOSED SOLUTION – ZONE SCOPEZone service – oversees a group of providers

Gathers and serves metadata

Trusted authority & mediator

AuthN & AuthZ center

Macaroon based tokens

Reflects existing hierarchies


PROPOSED SOLUTION – GLOBAL SCOPECooperation between Zones – hybrid architecture

Local authority center

Decentralized, P2P backbone

ZONE A ZONE B

PROVIDER 1 PROVIDER 2 PROVIDER 3

Local authority center


PROPOSED SOLUTION – METADATA SYNCMetadata handled by the Zone of origin – local AuthN & AuthZ center

CREATE / UPDATE / DELETE

READ

AAI AAI



READ

READ

ZONE A ZONE B

No concurrent modification conflicts Metadata sync overheads evenly distributed

PROVIDER 1 PROVIDER 2 PROVIDER 3


PROPOSED SOLUTION – METADATA SYNCMulti-tier client-server architecture

PROVIDER 1

PROVIDER 2

PROVIDER 3

ZONE A ZONE B

web interface web interfacenative client


PROPOSED SOLUTION – METADATA SYNCPublish / subscribe and multi-tier caching

FETCH & SUBSCRIBE

USER META

CACHE

CACHE CACHE

CACHE

CACHEZONE A ZONE B

PROVIDER 1

PROVIDER 2

PROVIDER 3


FETCH & SUBSCRIBE

FETCH & SUBSCRIBE

FETCH & SUBSCRIBE

FETCH & SUBSCRIBEUSER META

USER META

USER META

USER META

Vast performance improvement


PROPOSED SOLUTION – METADATA SYNCPublish / subscribe and multi-tier caching

PUBLISHPUBLISH

PUBLISH

PUBLISH

PUBLISH

CACHE

CACHE CACHE

CACHE

CACHEZONE A ZONE B

PROVIDER 1

PROVIDER 2

PROVIDER 3


MODIFIED

CACHE

CACHE CACHE

CACHE

CACHE

Eventual consistency


PROPOSED SOLUTION – AAIMacaroon-based decentralized authorization

TOKEN

location https://zone-a.com

public bd3f798c749b

<caveats> …

sig a7e108fa16baeffab0

ZONE A ZONE B

DISCOVERYMECHANISM

PROVIDER 1

PROVIDER 2

PROVIDER 3


FETCH & SUBSCRIBE

FETCH & SUBSCRIBE

FETCH & SUBSCRIBE


A) Open

B) Restricted

C) Isolated

PROPOSED SOLUTION – ZONE MODE


METADATA SYNC PROTOCOL IN GLOBAL DATA ACCESS

ZONE 1

ZONE 2

ZONE 3

GROUP 1

GROUP 2

GROUP 3

GROUP 4ZONE 4


Zone service acting as central authority and SP serverSynchronization Protocol (SP) for single zone scopeMacaroon based AAI for single zone scopeData provider service employing the SPCross Zone cooperation supportSync protocol supporting global, cross zone scale

CONCLUSIONS Global data access can be achieved by creating a decentralized network of data providers

We propose a metadata synchronization protocol for such network, based on hybdrid, multi-tier architecture with P2P backbone

Proposed concept is being implemented in Onedata, a distributed virtual file system


THANK YOU

https://onedata.org

The authors gratefully acknowledge the financial support of this work from AGH-UST

etadatasynchronization ack cyfronet agh kraków, …cgw'18, october 22-24, krakÓw. c....

Documents