easy access to open stack object storage
DESCRIPTION
Slides for a talk that didn't happen because of time constraints (OpenStack London Meetup, Dec 2013).TRANSCRIPT
![Page 1: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/1.jpg)
Easy Access to OpenStack Object StorageJuan J. Martínez
@reidrachttp://www.usebox.net/jjm/
Merry OpenStack, LondonDec 2013
![Page 2: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/2.jpg)
What is Swift?
Object Storage
● Not a real file system● RESTful HTTP API● Blob and metadata (name,
content type, date-time, size, MD5)
● Support for pseudo-hierarchical directories
Real File System
● open, read, write, seek, close● chdir, mkdir, rmdir● remove, rename● chmod, chown, stat, lock● etc
Open Source software for creating redundant, scalable object storage using clusters of standardized servers to store petabytes of accessible data.
![Page 3: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/3.jpg)
CUSTOMERS
![Page 4: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/4.jpg)
What we have
![Page 5: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/5.jpg)
What we sell
![Page 6: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/6.jpg)
What customers want
![Page 7: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/7.jpg)
Solution: Proxies
![Page 8: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/8.jpg)
Back in 2011...● Building a product based on OpenStack Object Storage● Started working with Bexar● Went live with Diablo
● Not many end-user tools using OpenStack Object Storage API(actually, it was Rackspace Cloud Files™)
● Customers won’t start building things straight away● Internal tools WIP (snapshot VM, re-image from snapshot)● It was hard to explain the product*:
What is Amazon S3? I want Dropbox!
* disclaimer: I’m a developer
![Page 9: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/9.jpg)
Proxies: FTPftp-cloudfs
● Personal project of Chmouel Boudjnah (now @ https://github.com/cloudfs/ftp-cloudfs)● Open Source● Very cool, “almost working” (Jan 2011)● fs.py abstraction:
Swift API -> FTP
Memset started contributing to make it “production ready”:● Support for OpenStack Object Storage (was Cloud Files only)● Missing bits in fs.py, more tests, introduced some new bugs● Fixed performance problems: memcache all the things!● Finally migrated from python-cloudfiles to python-swiftclient (kept the name, it’s cool)● Keystone auth, resume downloads, listing +10k files, large file support (>5GB)
![Page 10: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/10.jpg)
Proxies: SFTPsftp-cloudfs
● Open Source sponsored by Memset Hosting Ltdhttps://github.com/Memset/sftpcloudfs
● ftp-cloudfs’ fs.py + paramiko (ssh2 protocol for Python)● Secure: client -> SSH -> sftp-cloudfs -> SSL -> swift● Not as performant as FTP (1.6 MB/s vs pyftpdlib 4.2 MB/s), CPU intensive● Only password based authentication● Partial SCP support contributed by Koert van der Veer● + everything supported by ftp-cloudfs
![Page 11: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/11.jpg)
Bigger than the API● +2 TB / mo traffic (in > out)● Small VMs running ftp-cloudfs, sftp-cloudfs and shared cache (memcache)● Load-balanced with LVS● Easy to scale (add more VMs!)● Use cases: backups, CMS plugins (CDN support), remote storage, ???● FTP > SFTP > API
Useful Open Source projects:● Numbers from PyPI
ftp-cloudfs: 1895 downloads last monthsftp-cloudfs: 1107 downloads last month
● Patches from several contributors (including hosting companies)● “Stable” and almost feature complete, a new release every now and then
![Page 12: Easy access to open stack object storage](https://reader033.vdocuments.mx/reader033/viewer/2022052900/5562eca1d8b42a38778b52c4/html5/thumbnails/12.jpg)
Questions?Juan J. Martínez
@reidrachttp://www.usebox.net/jjm/
Merry OpenStack, LondonDec 2013
Easy Access to OpenStack Object Storage