eran rom, ibm - bringing compute to openstack swift, openstack israel 2015
TRANSCRIPT
© 2015 IBM Corporation
Bringing Compute to Openstack Swift
Eran Rom, System and Storage Researcher, IBM Research - Haifa
© 2015 IBM Corporation2
The Storlet engine extends Openstack Swift by enabling the push down of filtering, transforming and analysis
tasks to Swift instead of bringing the data to the computation.
© 2015 IBM Corporation3
Proxy Tier
Storage Tier
Ubuntu 14.04
Storlet Stuff
Ubuntu 14.04
Storlet Stuff
Ubuntu 14.04
Storlet Stuff
Ubuntu 14.04
Storlet Stuff
Containers
Powered Swift
Cluster
© 2015 IBM Corporation4
Containers Powered Swift Cluster
Proxy Tier
Storage Tier
© 2015 IBM Corporation5
Image Registry
Proxy Tier
Storage Tier
Containers Powered Swift Cluster
Ubuntu 14.04
Storlet Stuff
Ubuntu 14.04
Storlet Stuff
Ubuntu 14.04
Storlet StuffUbuntu 14.04
Storlet Stuff
© 2015 IBM Corporation6
Containers Powered Swift Cluster
Image Registry
Proxy Tier
Storage Tier
Ubuntu 14.04
Storlet Stuff
Ubuntu 14.04
Storlet Stuff
Ubuntu 14.04
Storlet StuffUbuntu 14.04
Storlet Stuff
© 2015 IBM Corporation7
Proxy Tier
Storage Tier
Developer
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Writing and Deploying a Storlet
@Override public void invoke(ArrayList<StorletInputStream> inStreams, ArrayList<StorletOutputStream> outStreams, Map<String, String> parameters, StorletLogger logger) throws StorletException { … }
© 2015 IBM Corporation8
Proxy Tier
Storage Tier
Developer
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Writing and Deploying a Storlet
PUT http://my.swift.co.il/v1/AUTH_111/Storlet/watermarkstorlet-10.jar
© 2015 IBM Corporation9
Proxy Tier
Storage Tier
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Invoking a Storlet on GET
GET http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webmX-Run-Storlet : watermarkstorlet-10.jar
© 2015 IBM Corporation10
Proxy Tier
Storage Tier
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Invoking a Storlet on GET
Forward Request to an Object Server holding a Replica
GET http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webmX-Run-Storlet : watermarkstorlet-10.jar
© 2015 IBM Corporation11
Invoking a Storlet on GET
Ubuntu 14.04
Storlet Stuff
ffmpegIntercept Request
Invoke Deamon, and Establish Connection with the Account’s Docker Container
Pass Input and Output FDs
Send back the Object’s Metadata and the Object’s Data Stream
Continue the GET Flow, where the Storelt’s response is sent back to the Proxy
Object Server
© 2015 IBM Corporation12
Proxy Tier
Storage Tier
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Ubuntu 14.04
Storlet Stuff
ffmpeg
Invoking a Storlet on PUT
PUT http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webmX-Run-Storlet : MetadataExtractorStorlet-10.jar
© 2015 IBM Corporation13
Invoking a Storlet on PUT
Ubuntu 14.04
Storlet Stuff
ffmpegIntercept Request
Invoke Deamon, and Establish Connection with the Account’s Docker Container
Pass Input and Output FDs
Send back the Object’s Metadata and the Object’s Data Stream
Continue the Proxy PUT Flow with the returned Metadata and data stream
Proxy Server
© 2015 IBM Corporation14
Demo
http://159.8.49.36:8080/raimediabridge/login.html
© 2015 IBM Corporation15
The Storlet Engine is in the Open
https://github.com/Open-I-Beam/swift-storlets Comprehensive Documentation
• Installation guide• How to write, deploy and invoke storlets guide• How to tailor your own image for running storlets
S2AIO• Install swift and storlets in a single VM for experimentation
Public users will be able to experiment with the Storlet engine thanks to FiWare – An EU research project delivering a novel service infrastructure for the future internet (http://www.fiware.org/)
© 2015 IBM Corporation16
Media Workflow at NAB (with RAI)Storlets run as material is uploaded and downloaded
Metadata Enrichment
Storlet
Swift Object Store
UploadEnrichedMetadata
Data
myvideo.mxf
Metadata
Data
myvideo.mxf
Data
Loudness Calculation
Storlet<Loudness,-23>
myvideo.mxf
Fix Loudness
Storlet
© 2015 IBM Corporation17
Acknowledgements
The Storlets project is partially funded by the following EU research projects
Digital Preservation Inspired by Human Memory
IoT for Smart Cities