codebits handivi
DESCRIPTION
Presentation on what happens behind the scenes at handivi.com by Celso Pinto for SAPO's Codebits 2008TRANSCRIPT
![Page 1: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/1.jpg)
Technology in a startup @ 7syntax
Celso PintoCoder, headbanger, biker and founder
![Page 2: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/2.jpg)
And Python lover
![Page 3: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/3.jpg)
Then: 2 + 1 idea
![Page 4: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/4.jpg)
Now: a 5 men army
![Page 5: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/5.jpg)
Fighting the evil empire!
![Page 6: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/6.jpg)
with
![Page 7: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/7.jpg)
A tech startup lives by it's tech
![Page 8: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/8.jpg)
TECH FAIL
![Page 9: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/9.jpg)
From the very back end...
http://flickr.com/photos/your_teacher/133262843/
![Page 10: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/10.jpg)
Question time!
![Page 11: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/11.jpg)
So, about data storage
![Page 12: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/12.jpg)
✔ Fast key based access✔ Cheetah-fast list fetches✔ Prefer to search with Lucene
Relational is dead... to me!
![Page 13: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/13.jpg)
✔ Fast key based access✔ Cheetah-fast list fetches✔ Prefer to search with Lucene
Cheetah fast reads
![Page 14: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/14.jpg)
✔ Standard mailboxes, accessible via POP/SMTP
Stuff I actually considered
![Page 15: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/15.jpg)
✔ Standard mailboxes, accessible via POP/SMTP✔ Fixed block files
Stuff I actually considered
![Page 16: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/16.jpg)
✔ Standard mailboxes, accessible via POP/SMTP✔ Fixed block files✔ MogileFS (plus Lucene for complete indexing)
Stuff I actually considered
![Page 17: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/17.jpg)
But then...
![Page 18: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/18.jpg)
Any day now
![Page 19: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/19.jpg)
✔ Easy sharding✔ Incredibly fast reads, faster than cheetahs!✔ Develop views in a(ny) familiar language✔ One instance can handle 500.000 databases✔ Each database can easily store millions of records✔ Online backups/snapshots and compaction
What we get from CouchDB
![Page 20: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/20.jpg)
✔ It's a “tad” slow on inserts✔ 50 inserts per second✔ We can push it to 500 by dropping fsync's and using
tmpfs
CouchDB has warts too :-(
But it doesn't matter: we <3 CouchDB!
![Page 21: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/21.jpg)
Any questions so far?
![Page 22: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/22.jpg)
Programming language(s)
![Page 23: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/23.jpg)
Programming languages
![Page 24: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/24.jpg)
Programming languages
![Page 25: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/25.jpg)
No!
![Page 26: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/26.jpg)
No!
![Page 27: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/27.jpg)
Programming languages
![Page 28: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/28.jpg)
The Snake
HisssssssssssssssssHiss hiss Hiss hiss Hissss hiss Hisss hisss Hissss hiss Hiss hiss Hiss hisss Hisss hisssssss Hiss hisss Hiss hissssss Hiss hiss Hisss hissss Hisss hiss Hiss hiss Hisss hiss Hisss hiss Hiss hiss Hiss hisss Hisss hisss Hiss hiss Hissss hissss Hiss hiss Hissss hisssssss Hiss hiss Hiss hiss Hiss hisss Hisssss hissss Hiss hiss Hissss hisssss Hiss hiss Hisss hiss Hisssss hiss Hisss hisss Hissss hiss Hisss hiss Hisss hiss Hissss hissssssss Hiss hisssssss Hissss hisss Hiss hisss Hissss hiss Hiss hisss Hiss hiss Hisss hisss Hissss hiss Hisss hissss Hiss hiss Hissss hisssssss Hiss hissss Hiss hissss Hiss hisss Hisss hissss Hisss hiss Hiss hisssss Hisss hisss Hissss hiss Hisss hisssssss Hiss hisss Hisss hissss Hiss hisss Hiss hissssss Hiss hisss Hissss hiss Hisss hiss Hiss hisss Hiss hiss Hisss hissss Hiss hiss Hissss hiss Hiss hisssss Hisss hisssssss Hiss hissss Hiss hisss Hissss hiss Hiss hiss Hiss hiss Hissss hisss Hissss hiss Hisss hisssssss Hiss hisssss Hisss hisss Hiss hiss Hiss hisss Hiss hiss Hisss hisss Hissss hiss Hisss hiss Hiss hiss Hiss hiss Hissss hisssss Hisssss hiss Hisssssss hissss Hiss hissss Hiss hisss Hisss hiss Hissss hissss Hiss hiss Hiss hisssss
* you need Acme::Python to understand this
![Page 29: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/29.jpg)
Only one restriction
4 space indentsBut you should be doing that already!
![Page 30: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/30.jpg)
Other advantages
Here we have Python Some other language
![Page 31: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/31.jpg)
After one year
Still Python Some other language
![Page 32: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/32.jpg)
On the mobile
![Page 33: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/33.jpg)
On the mobile
![Page 34: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/34.jpg)
On the web front
![Page 35: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/35.jpg)
On the web front
![Page 36: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/36.jpg)
Any questions?
![Page 37: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/37.jpg)
Frameworks vs. Libraries
![Page 38: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/38.jpg)
Siza Vieira
![Page 39: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/39.jpg)
The blue pill
![Page 40: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/40.jpg)
✔ Modular: we can replace have replaced many parts✔ Stable/solid: impressive # of tests, great docs!✔ Easily hackable: it's Python after all
Why Django?
![Page 41: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/41.jpg)
No, seriously...
![Page 42: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/42.jpg)
Alright, you win
![Page 43: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/43.jpg)
✔ Auth: we built our own from the start✔ Sessions: in-house developed stateless sessions✔ ORM: now on it's way to meet the dodo!✔ By extension: we never got the chance to use the
admin application
What we replaced
![Page 44: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/44.jpg)
✔ Templates: inheritance is great!✔ Request handling/routing rules engine✔ Forms: lots of validation there✔ … other bits I can't really remember
What we're keeping
![Page 45: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/45.jpg)
How we're using Django
![Page 46: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/46.jpg)
Any questions about Django?
![Page 47: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/47.jpg)
Messaging
![Page 48: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/48.jpg)
✔ Throttling, balancing, loose coupling✔ We can tap into any queues and experiment✔ Remember CouchDB? MQ is the MDA for each
“mailbox”
Why is it important to us?
![Page 49: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/49.jpg)
Lots of good stuff out there
![Page 50: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/50.jpg)
✔ ActiveMQ: stable, small and fast enough✔ Many protocols: we chose STOMP, nice to debug✔ Had to create AsyncIO Python lib
Our choice
![Page 51: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/51.jpg)
✔ Does MDB mean anything to you?✔ Python has no MDB container, had to create one✔ Python + processing (new in Python 3000):
✔ Configuration similar to Django's✔ 1 process => 1 destination consumer
Final bits on messaging
![Page 52: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/52.jpg)
Any questions about messaging?
![Page 53: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/53.jpg)
Wrapping up
![Page 54: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/54.jpg)
Current architecture
mobile ?mobile webmobile
perlbal
API (django apps)
static Ct. adapter
consumers
writes
accounts User 1 User 2 User nCouchDB
reads S3
![Page 55: Codebits Handivi](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c856bc4a7959d7758b4685/html5/thumbnails/55.jpg)
Any final questions?