yalms: a group drive api for cloud-based classrooms
DESCRIPTION
Although there are many cloud drive APIs and applications (clients mostly) today, they are not optimized for group activities in cloud-based classrooms. Specifically, such APIs are weak in face of realtime interactivity and large data bulks. YALMS is a newly developed open source educational package which in this presentation is augmented with a group drive functionality. This paper poses the optimization problem involved in a group drive in clouds and presents implementations of both server and client sides of the respective API.TRANSCRIPT
.
YALMS Basics
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 2/26...
2/26
.
YALMS Basics
.YALMS stands for .....
.YetAnotherLectureManagementSystem
• next generation digital classroom 02
• several robots: video annotation, content packing, multimedia storage, etc.
• can run in clouds and cloud-based classrooms◦ specifically, in VM-per-student environments
• knowledge extraction and social components
02 "Homepage of the YALMS software presented in this paper" https://github.com/maratishe/yalms (2013)
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 3/26...
3/26
.
YALMS : The Big PictureYALMS: Intro
YALMS.annotation
2013.07・ET研
2013.09・ET研
YALMS.streaming2014.01・ET研
YALMS.hiperf2013.10・ITE-MMS研
YALMS.cloud2013.11・IN研
YALMS.format
YALMS.done!
DASH Adaptive HTML5
PDF video
VM sync groupdrive
SSDmulticore
slices
stream recordreplay storage
2014.01・LOIS研
content
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 4/26...
4/26
.
YALMS.groupdrive : The Specs
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 5/26...
5/26
.
YALMS.groupdrive : Overall Idea
Master Copy
Slave Copy
Slave Copy
Slave Copy
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 6/26...
6/26
.
YALMS.groupdrive : The Specs
1. realtime async. asymm. P2P sync -- but notdistributed storage
2. fully distributed, can work without a master
3. smart and adaptive, fit for any environment◦ remote master, fully distributed, throughput-limited, clouds, etc.
4. efficient with large files -- a sub-file data grain
5. API with cross-platform implementation◦ Linux, Windows, VMs
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 7/26...
7/26
.
YALMS.groupdrive : Existing (bad)Solutions
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 8/26...
8/26
.
Bad Solution : BitTorrent
• a very bad solution• needs central registry -- Tracker
• good only with many peers, too much overhead with few◦ rarest-first policy is not fit for the problem
• good for download, but what about upload?
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 9/26...
9/26
.
Bad Solution : DSync
• a paper in 05 -- the closest rival
• aggregation frommultiple sources while optimizing load• not really distributed -- single point prospective -- o2m or o2o
• no practical considerations -- no measurement, ordered lists, etc.
05 H.Pucha+3 "Adaptive File Transfers for Diverse Environments" USENIX Annual Technical Conference (2008)
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 10/26...
10/26
.
Bad Solution : P2P Streaming
Share
…
Content Provider (origin)
P2P Network
• good for videostreaming 04
06
• again, bad forbi-directionalsyncs
04 myself+0 "Multi-Source Stream Aggregation in the Cloud" Wiley Book on Content Delivery (ACDN) (2013)
06 myself+0 "Extremely Scalable and Low Demand Live P2P Streaming based on Variable Bitrate" CANDAR (2013)
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 11/26...
11/26
.
Bad Solution : Group Virtualization
Service Provider
Virtual Wireless User
Wireless User
Wireless User
Wireless User
Application Service Service traffic • specific to
remoteservices 07
• does not workwell with fullylocal syncs
• not specific tosyncs
07 myself "Virtual Wireless User: A Practical Design for Parallel MultiConnect Using WiFi Direct..." Mobiquitous (2013)
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 12/26...
12/26
.
Proposal and Design
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 13/26...
13/26
.
Proposal : Fundamentals
.Objective..
.API + Implementation of the proposed group drive
• standard RESTful API• implemented in PHP
• cross-platform
• adaptive to e2e throughput -- response time
• avoid congesting the master
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 14/26...
14/26
.
Design : PUSH/PULL Basics• simple: 1 PULL and X PUSH rounds, X peers in each round -- X is scale
.The PULL..
.
• regularly ask peers for changes, new content comes in replies• do not push the newly recieved content
.The PUSH..
.• push toX peers in X rounds for quick distribution
.Conflicts...... ignored for now, treat like in GoogleDrive, etc.
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 15/26...
15/26
.
Design : The MFFxR Unit• basic list unit for PUSHes and PULLs -- ordered by decreasingthroughput
• Random Peer is basically for measurement/discovery• scale is a global parameter
Self
List:M List:MF
List:MFF List:MFF+R
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 16/26...
16/26
.
Design : Peer Discovery Roadmap
• dynamically incorporate new peers
REP( peers)
REQ() Done Update
peers
Master
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 17/26...
17/26
.
Design : PULL Roadmap• a roadmap rather than state machine
Align peers
MFFF… + R
REP( null)
REQ( state)
REP( block, epoch)
Scale (s)
Error, failure, timeout
Replace (if new block)
Done
Notify
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 18/26...
18/26
.
Design : PUSH Roadmap
Align
peers MFFF… + R
REP( OK)
REQ(block)
REP( Collision)
Scale (s)
Done
I already have it! (double up)
If M or R
Pick next from MFFF… for 2nd round
Notify
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 19/26...
19/26
.
Experiments
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 20/26...
20/26
.
Experiments
• one 10Gb file
• initial state: only masterhas the file
• try various environments
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5log( time)
0
0.2
0.4
0.6
0.8
1
Peer
com
plet
ion
ratio
YALMS.groupdrive Local Only (10 peers and scale 2)YALMS.groupdrive Remote Master (10 peers and scale 2)Traditional (1 client)Traditional (10 clients)
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 21/26...
21/26
.
Implementation and API
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 22/26...
22/26
.
Implementation : Basic Unit of Info.Exchange
• standard unit of data exchange -- JSON -- part of API 03• some parts are not used -- blob only used for PUSHes/PULLs
Datatype Parametername Comments
string relpath Relative to top folderstring filenamelong blockpos Position of the head byte of the block in file
int blocksize Fixed for each sessiondouble mtime Epoch time of last update for that block at Master
string abspath Path for this block at Master. Not used by Slave Copies.blob content Blob: binary string. Used when needed.
03 "Homepage of the YALMS.groupdrive project" https://github.com/maratishe/yalms.groupdrive (2013)
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 23/26...
23/26
.
Implementation : Actual API
• per-peer, MFFxR is outside of the API (part of implementation)
• state is a flat JSON of items (keys) and MD5 digests (values) 03
• APIs are standard RESTfull HTTP request/reply routines
• API1: pull(state)−−reply(dataunit)• API2: push(dataunit)−−reply(status)
03 "Homepage of the YALMS.groupdrive project" https://github.com/maratishe/yalms.groupdrive (2013)
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 24/26...
24/26
.
Implementation : Software Design• prefers Linux but will work on Windows using portable TinyWeb 03
OS,
Sys
tem
OS,
Sys
tem
WebServer
wgetReqs
RepsWeb
Server
wgetReqs
Reps
03 "Homepage of the YALMS.groupdrive project" https://github.com/maratishe/yalms.groupdrive (2013)
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 25/26...
25/26
.
That’s all, thank you ...
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 26/26...
26/26
.
[01] (2013)The enPiT Projecthttp://www.enpit.jp
[02] (2013)Homepage of the YALMS software presented in this paperhttps://github.com/maratishe/yalms
[03] (2013)Homepage of the YALMS.groupdrive projecthttps://github.com/maratishe/yalms.groupdrive
[04] myself+0 (2013)Multi-Source Stream Aggregation in the CloudWiley Book on Content Delivery (ACDN)
[05] H.Pucha+3 (2008)Adaptive File Transfers for Diverse EnvironmentsUSENIX Annual Technical Conference
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 26/26...
26/26
.
[06] myself+0 (2013)Extremely Scalable and Low Demand Live P2P Streaming based on Variable BitrateCANDAR
[07] myself (2013)Virtual Wireless User: A Practical Design for Parallel MultiConnect Using WiFiDirect...Mobiquitous
[08] (2013)Dropbox Core APIhttps://www.dropbox.com/developers/core
M.Zhanikeev, H.Koide -- [email protected] -- YALMS: A Group Drive API for Cloud-Based Classrooms -- http://tinyurl.com/kyutech131121 --- 26/26...
26/26