Download - Git with Swift Atlanta 2014
![Page 1: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/1.jpg)
Chmouel Boudjnah!OpenStack Summit 2014 Atlanta
OpenStack Swift as a Backend for Git
Chmouel Boudjnah - @chmouel - eNovance
![Page 2: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/2.jpg)
Chmouel Boudjnah - @chmouel - eNovance
The mandatory about me
![Page 3: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/3.jpg)
Chmouel Boudjnah - @chmouel - eNovance
About me
I am not the author, Fabien Boucher aka: @bzho
Chmouel Boudjnah - @chmouel - eNovance
![Page 4: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/4.jpg)
Chmouel Boudjnah - @chmouel - eNovance
About me
We both work for eNovance
![Page 5: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/5.jpg)
Chmouel Boudjnah - @chmouel - eNovance
About me
OpenStacker for quite some time now
![Page 6: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/6.jpg)
Chmouel Boudjnah - @chmouel - eNovance
So what is Swift?
![Page 7: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/7.jpg)
Chmouel Boudjnah - @chmouel - eNovance
• Object Storage • Not a file system • Eventually Consistent • HTTP REST API • Very Scalable • Software defined Storage
What is Swift
![Page 8: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/8.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Why would you want to use GIT with Swift?
![Page 9: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/9.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Why Swift with GIT
Why not?
![Page 10: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/10.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Why Swift with GIT
When you use a file system you have to :
Perform backups of your repositories regularly
Perform complex and risky operations to extend the storage capabilities for you repositories.
You have mostly have to manage it yourself
It’s an another piece in your infrastructure
You have to manage the downtime
Not fun
![Page 11: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/11.jpg)
Chmouel Boudjnah - @chmouel - eNovance
So what are the benefits?
![Page 12: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/12.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Why Swift with GIT
The advantages of using Swift
Safety: The inherent replication of a data store.
Capability: easily extend storage capacity.
High Availability: no single point of failure.
Maintainability: allow operations teams to maintain the cluster without any downtime.
In an OpenStack environment it’s usually already there.
![Page 13: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/13.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Let’s see how git internal works
![Page 14: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/14.jpg)
Chmouel Boudjnah - @chmouel - eNovance
![Page 15: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/15.jpg)
Chmouel Boudjnah - @chmouel - eNovance
![Page 16: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/16.jpg)
Chmouel Boudjnah - @chmouel - eNovance
So how did we plug that to Swift?
![Page 17: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/17.jpg)
Chmouel Boudjnah - @chmouel - eNovance
We did not want to modify the GIT client binaries.
![Page 18: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/18.jpg)
Chmouel Boudjnah - @chmouel - eNovance
We used a pure python GIT library called Dulwich
![Page 19: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/19.jpg)
Chmouel Boudjnah - @chmouel - eNovance
• Create, read, manage loose objects (blob, tree, commit, tag).
• Create, read, manage pack files, pack indexes. • Implement the Git smart protocol through git-
upload-pack and git-receive-pack. • Implement the Git, HTTP, SSH listeners to start
Dulwich as a Git server. • Provide a well designed interface to the storage
backend. (Repo, MemoryRepo).
Dulwich
![Page 20: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/20.jpg)
Chmouel Boudjnah - @chmouel - eNovance
How do we use Dulwhich
We implemented a custom repo called SwiftRepo as a backend to the git listener of Dulwich which communicate to OpenStack Swift
![Page 21: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/21.jpg)
Chmouel Boudjnah - @chmouel - eNovance
So about that SwiftRepo backend how does it work?
![Page 22: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/22.jpg)
Chmouel Boudjnah - @chmouel - eNovance
SwiftRepos backend to Dulwich
It doesn’t use the standard way to store Git references (one file by reference).
It doesn’t store loose objects but instead uses the pack format.
In addition to the .index that relate to each pack file we added .info object
It makes the Request to Swift concurrently.
![Page 23: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/23.jpg)
Chmouel Boudjnah - @chmouel - eNovance
How’s the benchmark then?
![Page 24: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/24.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Benchmark Setup
Benchmark performed on two architectures : Github : ● Git client from eNovance network <-> GitHub ! Dulwich/Swift : ●· Git client from eNovance network <-> Dulwich (Rax compute) <-> Rax Cloudfiles !Three different repos : ! ●· Swiftsync (873 objects with 293 commits) ●· Edeploy (2246 objects with 618 commits) ●· Swift (23480 objects with 3212 commit)
![Page 25: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/25.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Full repository push
![Page 26: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/26.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Clone
![Page 27: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/27.jpg)
Chmouel Boudjnah - @chmouel - eNovance
That’s it, hope you enjoyed.
![Page 28: Git with Swift Atlanta 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051816/5462587bb1af9f936c8b4f53/html5/thumbnails/28.jpg)
Chmouel Boudjnah - @chmouel - eNovance
Eventual Consistentie
Why Eventual Consistentie window is not an issue