crate shared nothing web backends - web backend meetup may 2014
DESCRIPTION
Talk about building real shared nothing web backends using crate data on Web Backend Meetup May 2014TRANSCRIPT
![Page 1: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/1.jpg)
Shared Nothing Web Backends
Matthias Wahl - developer @ crate.io
![Page 2: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/2.jpg)
Agenda
Shared Nothing
Crate
Shared Nothing Architectures using Crate
Cluster Workout
![Page 3: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/3.jpg)
Prepare for Workout
1.) bash -c "$(curl -L try.crate.io)"
https://cdn.crate.io/downloads/releases/crate-0.38.2.tar.gz
2.) sh$ bin/crate
0.) share our “craty” WLAN pw: select*fromcrate;
![Page 4: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/4.jpg)
Prepare for Workout
3.) start up the twitter tutorial
![Page 5: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/5.jpg)
Shared Nothing
![Page 6: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/6.jpg)
Shared Nothing
Wikipedia: !
a distributed computing architecture in which each
node is independent and self-sufficient, and there is no
single point of contention across the system. More
specifically, none of the nodes share memory or disk
storage.
![Page 7: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/7.jpg)
Shared Nothing
Origin:
!
Micheal Stonebraker
“The Case for Shared Nothing” (1986)
!
http://pdf.aminer.org/000/255/770/
the_case_for_shared_nothing.pdf
![Page 8: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/8.jpg)
Shared Nothing
Principle of most functional languages
Shared state/resources (memory) is what makes threading hard
No SPOF
Easy concurrency
![Page 9: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/9.jpg)
DB
Cache
App App App App
DB - Replica
Cache Cache
Bad Idea
![Page 10: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/10.jpg)
DB
Cache
App App App App
DB - Replica
Cache Cache
Bad Idea
![Page 11: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/11.jpg)
Crate
Austria and Berlin based Startup
founded 2013
open source since april https://github.com/crate/crate
![Page 12: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/12.jpg)
Crate
shared nothing massively scalable datastore
standing on the shoulders of giants
![Page 13: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/13.jpg)
Crate
![Page 14: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/14.jpg)
Crate
automatic sharding and replication
(semi-) structured models
blob support
SQL query language
![Page 15: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/15.jpg)
Crate
powerful fulltext search capabilities
complex aggregations in near real-time
- no joins
![Page 16: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/16.jpg)
Crate
crate-java
crate-jdbc
crate-python (+SQLAlchemy)
crate-ruby
…
CRATE Clients
![Page 17: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/17.jpg)
Shared Nothing with Crate
data and query execution is distributed among all nodes
no master/slave - all nodes are equal
automatic sharding & replication
![Page 18: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/18.jpg)
Shared Nothing with Crate
example architecture
crate
app
crate
app
crate
app
…
Load Balancer
![Page 19: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/19.jpg)
Shared Nothing with Crate
horizontally scalable as hell
still flexible enough for complex applications
high availability for your whole stack
![Page 20: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/20.jpg)
Shared Nothing with Crate
create table blogpost ( id string primary key, created timestamp, text string, creator string )
![Page 21: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/21.jpg)
Shared Nothing with Crate
from sqlalchemy import Column, String, DateTime from microblog.model import Base import uuid !!def genuuid(): return str(uuid.uuid4()) !!class BlogPost(Base): ! __tablename__ = 'blogpost' ! id = Column(String, default=genuuid, primary_key=True) text = Column('text', String, nullable=False) creator = Column('creator', String, nullable=False) created = Column('created', DateTime, nullable=False)
![Page 22: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/22.jpg)
Shared Nothing with Crate
@rpcmethod_route(request_method="GET") def list(self): """ Return all blogposts """ query = DBSession.query(BlogPost).order_by(BlogPost.created.desc()) blogposts = query.all() result = [] for post in blogposts: result.append({'id': post.id, 'created': post.created.isoformat(), 'text': post.text, 'creator': post.creator}) return {"data": {"blogposts": result}}
![Page 23: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/23.jpg)
Shared Nothing with Crate
get into detail: !
http://lovelysystems.github.io/lovely.microblog/
![Page 24: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/24.jpg)
Cluster Workout!!!
![Page 25: Crate Shared Nothing Web Backends - Web Backend Meetup May 2014](https://reader034.vdocuments.mx/reader034/viewer/2022051613/54c6b8534a79597d178b466a/html5/thumbnails/25.jpg)
CRATE
Thank you
web: https://crate.io/
github: https://github.com/crate
twitter: @cratedata
IRC: #crate
stackoverflow tag: cratedata