link walking with riak
DESCRIPTION
Describe how to create, manage and traverse links in Riak KV.TRANSCRIPT
Link Walking with Riak (KV)
Susan Potter / Finsignia
June 25, 2011
Riak Ecosystem
RiakSearch
RiakKV
RiakCore
Riak Ecosystem
RiakSearch
RiakKV
RiakCore
Riak Ecosystem
RiakSearch
RiakKV
RiakCore
Riak Ecosystem
RiakSearch
RiakKV
RiakCore
Riak Ecosystem
RiakSearch
RiakKV
RiakCore
% whoami
# curl http://susanpotter.net/whoami/windycitydb2011.json
• 15 years RDBMS• 3-4 years KV databases (CouchDB and now Riak)
• 1 year Redis• Using graph database in next part of platform
% whoami
# curl http://susanpotter.net/whoami/windycitydb2011.json
• 15 years RDBMS• 3-4 years KV databases (CouchDB and now Riak)
• 1 year Redis• Using graph database in next part of platform
% whoami
# curl http://susanpotter.net/whoami/windycitydb2011.json
• 15 years RDBMS• 3-4 years KV databases (CouchDB and now Riak)
• 1 year Redis• Using graph database in next part of platform
% whoami
# curl http://susanpotter.net/whoami/windycitydb2011.json
• 15 years RDBMS• 3-4 years KV databases (CouchDB and now Riak)
• 1 year Redis• Using graph database in next part of platform
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Scope of Talk
• Problem Domain• Linking Objects• Traversing Links• Link Maintainence
• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency
Problem Domain
Problem Domain
Linking[1/2]
curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’http://host:8098/riak/album/Born-This-Way�data ...
Linking[1/2]
curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’http://host:8098/riak/album/Born-This-Way�data ...
Linking[1/2]
curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’http://host:8098/riak/album/Born-This-Way�data ...
Linking[1/2]
curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’http://host:8098/riak/album/Born-This-Way�data ...
Linking[1/2]
curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’http://host:8098/riak/album/Born-This-Way�data ...
Linking[2/2]• Two Dimensional
both direction and type/tag
• 0..* links per objectof any type/tag
• Metadata traversingnot load, parse, then lookup cycle
• Model sufficiencyotherwise graph database might be more useful
• Link to other buckets(in same Riak cluster)
Linking[2/2]• Two Dimensional
both direction and type/tag
• 0..* links per objectof any type/tag
• Metadata traversingnot load, parse, then lookup cycle
• Model sufficiencyotherwise graph database might be more useful
• Link to other buckets(in same Riak cluster)
Linking[2/2]• Two Dimensional
both direction and type/tag
• 0..* links per objectof any type/tag
• Metadata traversingnot load, parse, then lookup cycle
• Model sufficiencyotherwise graph database might be more useful
• Link to other buckets(in same Riak cluster)
Linking[2/2]• Two Dimensional
both direction and type/tag
• 0..* links per objectof any type/tag
• Metadata traversingnot load, parse, then lookup cycle
• Model sufficiencyotherwise graph database might be more useful
• Link to other buckets(in same Riak cluster)
Linking[2/2]• Two Dimensional
both direction and type/tag
• 0..* links per objectof any type/tag
• Metadata traversingnot load, parse, then lookup cycle
• Model sufficiencyotherwise graph database might be more useful
• Link to other buckets(in same Riak cluster)
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1
=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1
=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1
=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1
=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1
=> object linked to album with tag/type “producer”
Traversing Links[1/4]
• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type
• GET /riak/album/Born-This-Way/artist,performer,1=> artists linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/artist,producer,1=> artists linked to album with tag/type “producer”
• GET /riak/album/Born-This-Way/_,performer,1=> object linked to album with tag/type “performer”
• GET /riak/album/Born-This-Way/_,producer,1=> object linked to album with tag/type “producer”
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1
=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1
=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1
=> all albums that “producers” on album A have released
Traversing Links[2/4]
• GET /riak/album/A/artist,_,1/artist,collaborator,1=> all artists associated with album A and all artists that have collaborated with these artists
• GET /riak/album/A/artist,performer,0/artist,collaborator,1=> all “collaborators” of “performers” on album A excluding “performers”
• GET /riak/album/A/artist,producer,0/album,release,1=> all albums that “producers” on album A have released
Traversing Links[3/4]
artist,performer,1
_,_,0
/album,release,1
• Bucket namee.g. artist, album
• Tag (or type) namee.g. performer, producer
• Return (or “keep”)useful in nested traversals
• Wildcarding’_’ for bucket and tag means “any”
• Nestingtraverse on multiple levels
• Multiple Sources?how to walk links from multiple objects?
Traversing Links[3/4]
artist,performer,1
_,_,0
/album,release,1
• Bucket namee.g. artist, album
• Tag (or type) namee.g. performer, producer
• Return (or “keep”)useful in nested traversals
• Wildcarding’_’ for bucket and tag means “any”
• Nestingtraverse on multiple levels
• Multiple Sources?how to walk links from multiple objects?
Traversing Links[3/4]
artist,performer,1
_,_,0
/album,release,1
• Bucket namee.g. artist, album
• Tag (or type) namee.g. performer, producer
• Return (or “keep”)useful in nested traversals
• Wildcarding’_’ for bucket and tag means “any”
• Nestingtraverse on multiple levels
• Multiple Sources?how to walk links from multiple objects?
Traversing Links[3/4]
artist,performer,1
_,_,0
/album,release,1
• Bucket namee.g. artist, album
• Tag (or type) namee.g. performer, producer
• Return (or “keep”)useful in nested traversals
• Wildcarding’_’ for bucket and tag means “any”
• Nestingtraverse on multiple levels
• Multiple Sources?how to walk links from multiple objects?
Traversing Links[3/4]
artist,performer,1
_,_,0
/album,release,1
• Bucket namee.g. artist, album
• Tag (or type) namee.g. performer, producer
• Return (or “keep”)useful in nested traversals
• Wildcarding’_’ for bucket and tag means “any”
• Nestingtraverse on multiple levels
• Multiple Sources?how to walk links from multiple objects?
Traversing Links[3/4]
artist,performer,1
_,_,0/album,release,1• Bucket name
e.g. artist, album
• Tag (or type) namee.g. performer, producer
• Return (or “keep”)useful in nested traversals
• Wildcarding’_’ for bucket and tag means “any”
• Nestingtraverse on multiple levels
• Multiple Sources?how to walk links from multiple objects?
Traversing Links[3/4]
artist,performer,1_,_,0
/album,release,1
• Bucket namee.g. artist, album
• Tag (or type) namee.g. performer, producer
• Return (or “keep”)useful in nested traversals
• Wildcarding’_’ for bucket and tag means “any”
• Nestingtraverse on multiple levels
• Multiple Sources?how to walk links from multiple objects?
curl -X POST -H “Content-type: application/json” \http://host:8098/mapred �data @- \{"inputs":[
["artist","Ke$ha"],["artist", "Lady+Gaga"]
],"query":[{"link": { "tag": "release", "bucket": "album" }},{"map":{ "language":"javascript",
"source":"function(v) return [v]; "}}
]}^D
curl -X POST -H “Content-type: application/json” \http://host:8098/mapred �data @- \{"inputs":[
["artist","Ke$ha"],["artist", "Lady+Gaga"]
],"query":[{"link": { "tag": "release", "bucket": "album" }},{"map":{ "language":"javascript",
"source":"function(v) return [v]; "}}
]}^D
curl -X POST -H “Content-type: application/json” \http://host:8098/mapred �data @- \{"inputs":[
["artist","Ke$ha"],["artist", "Lady+Gaga"]
],"query":[{"link": { "tag": "release", "bucket": "album" }},{"map":{ "language":"javascript",
"source":"function(v) return [v]; "}}
]}^D
Link Maintenance
• Commit Hookscan use either pre- or post-commit hooks
• Automatic Backlinkscan use commit hooks to generate backlinks automatically
• Trigger Messageslinks could only be used as triggers
• Delete Cleanuppost-commit hook with backlinks can clean up links to new orphans automatically
Link Maintenance
• Commit Hookscan use either pre- or post-commit hooks
• Automatic Backlinkscan use commit hooks to generate backlinks automatically
• Trigger Messageslinks could only be used as triggers
• Delete Cleanuppost-commit hook with backlinks can clean up links to new orphans automatically
Link Maintenance
• Commit Hookscan use either pre- or post-commit hooks
• Automatic Backlinkscan use commit hooks to generate backlinks automatically
• Trigger Messageslinks could only be used as triggers
• Delete Cleanuppost-commit hook with backlinks can clean up links to new orphans automatically
Link Maintenance
• Commit Hookscan use either pre- or post-commit hooks
• Automatic Backlinkscan use commit hooks to generate backlinks automatically
• Trigger Messageslinks could only be used as triggers
• Delete Cleanuppost-commit hook with backlinks can clean up links to new orphans automatically
Possibilities
Object metadataextensions?
Limitations
Not a graph database!
References[1/2]
• Riak Wikihttp://wiki.basho.com/
• Riak Mailing [email protected]
• Riak Contribhttp://contrib.basho.com/
• Riak Code Reposhttp://github.com/basho
References[1/2]
• Riak Wikihttp://wiki.basho.com/
• Riak Mailing [email protected]
• Riak Contribhttp://contrib.basho.com/
• Riak Code Reposhttp://github.com/basho
References[1/2]
• Riak Wikihttp://wiki.basho.com/
• Riak Mailing [email protected]
• Riak Contribhttp://contrib.basho.com/
• Riak Code Reposhttp://github.com/basho
References[1/2]
• Riak Wikihttp://wiki.basho.com/
• Riak Mailing [email protected]
• Riak Contribhttp://contrib.basho.com/
• Riak Code Reposhttp://github.com/basho
References[2/2]
• Secondary index support in Riak?Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806
• riak_core applicationse.g. Riak Search, riak_zab, riak_pipe
• CometKV database supports Active Storage Objects (ASO), handlers
http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
References[2/2]
• Secondary index support in Riak?Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806
• riak_core applicationse.g. Riak Search, riak_zab, riak_pipe
• CometKV database supports Active Storage Objects (ASO), handlers
http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
References[2/2]
• Secondary index support in Riak?Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806
• riak_core applicationse.g. Riak Search, riak_zab, riak_pipe
• CometKV database supports Active Storage Objects (ASO), handlers
http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf
Credits
• Basho & Riak logosCourtesy of Basho
• Icy chained fence photohttp://www.flickr.com/photos/existentist/342177746/
• Frozen chain photohttp://www.flickr.com/photos/skedonk/4197921511/
• Hook and chain photohttp://www.flickr.com/photos/37491467@N07/3871760093/
Questions?
Figure: http://www.flickr.com/photos/42682395@N04/
@SusanPotter
Questions?
Figure: http://www.flickr.com/photos/42682395@N04/
@SusanPotter