link walking with riak

Link Walking with Riak (KV)

Susan Potter / Finsignia

June 25, 2011

Riak Ecosystem

RiakSearch

RiakKV

RiakCore

Riak Ecosystem

RiakSearch

RiakKV

RiakCore

Riak Ecosystem

RiakSearch

RiakKV

RiakCore

Riak Ecosystem

RiakSearch

RiakKV

RiakCore

Riak Ecosystem

RiakSearch

RiakKV

RiakCore

% whoami

# curl http://susanpotter.net/whoami/windycitydb2011.json

• 15 years RDBMS• 3-4 years KV databases (CouchDB and now Riak)

• 1 year Redis• Using graph database in next part of platform

% whoami

Scope of Talk

• Problem Domain• Linking Objects• Traversing Links• Link Maintainence

• not Consistent Hashing or DHTs• not Vector Clocks• not Gossip Protocol• not Hinted Handoff• not Read Repair• not Eventual Consistency

Scope of Talk

Problem Domain

Linking[1/2]

curl -X PUT -H ’Content-type: text/json’-H ’Link: </riak/artist/Lady-Gaga>; riaktag=“performer”’-H ’Link: </riak/artist/DJ-Snake>; riaktag=“producer”’-H ’Link: </riak/artist/DJ-White-Shadow>; riaktag=“producer”’...-H ’Link: </riak/artist/Fernando-Garibay>; riaktag=“producer”’http://host:8098/riak/album/Born-This-Way�data ...

Linking[1/2]

Linking[2/2]• Two Dimensional

both direction and type/tag

• 0..* links per objectof any type/tag

• Metadata traversingnot load, parse, then lookup cycle

• Model sufficiencyotherwise graph database might be more useful

• Link to other buckets(in same Riak cluster)

Traversing Links[1/4]

• GET /riak/album/Born-This-Way/artist,_,1

=> all artists linked to album regardless of link tag/type

• GET /riak/album/Born-This-Way/artist,performer,1

=> artists linked to album with tag/type “performer”

• GET /riak/album/Born-This-Way/artist,producer,1

=> artists linked to album with tag/type “producer”

• GET /riak/album/Born-This-Way/_,performer,1

=> object linked to album with tag/type “performer”

• GET /riak/album/Born-This-Way/_,producer,1

=> object linked to album with tag/type “producer”

• GET /riak/album/Born-This-Way/artist,_,1=> all artists linked to album regardless of link tag/type

• GET /riak/album/Born-This-Way/artist,performer,1=> artists linked to album with tag/type “performer”

• GET /riak/album/Born-This-Way/artist,producer,1=> artists linked to album with tag/type “producer”

• GET /riak/album/Born-This-Way/_,performer,1=> object linked to album with tag/type “performer”

• GET /riak/album/Born-This-Way/_,producer,1=> object linked to album with tag/type “producer”

• GET /riak/album/A/artist,_,1/artist,collaborator,1

=> all artists associated with album A and all artists that have collaborated with these artists

• GET /riak/album/A/artist,performer,0/artist,collaborator,1

=> all “collaborators” of “performers” on album A excluding “performers”

• GET /riak/album/A/artist,producer,0/album,release,1

=> all albums that “producers” on album A have released

• GET /riak/album/A/artist,_,1/artist,collaborator,1

=> all artists associated with album A and all artists that have collaborated with these artists

• GET /riak/album/A/artist,_,1/artist,collaborator,1=> all artists associated with album A and all artists that have collaborated with these artists

• GET /riak/album/A/artist,performer,0/artist,collaborator,1=> all “collaborators” of “performers” on album A excluding “performers”

• GET /riak/album/A/artist,producer,0/album,release,1=> all albums that “producers” on album A have released

artist,performer,1

/album,release,1

• Bucket namee.g. artist, album

• Tag (or type) namee.g. performer, producer

• Return (or “keep”)useful in nested traversals

• Wildcarding’_’ for bucket and tag means “any”

• Nestingtraverse on multiple levels

• Multiple Sources?how to walk links from multiple objects?

artist,performer,1

/album,release,1

artist,performer,1

/album,release,1

artist,performer,1

/album,release,1

artist,performer,1

/album,release,1

artist,performer,1

_,_,0/album,release,1• Bucket name

e.g. artist, album

artist,performer,1_,_,0

/album,release,1

curl -X POST -H “Content-type: application/json” \http://host:8098/mapred �data @- \{"inputs":[

["artist","Ke$ha"],["artist", "Lady+Gaga"]

],"query":[{"link": { "tag": "release", "bucket": "album" }},{"map":{ "language":"javascript",

"source":"function(v) return [v]; "}}

Link Maintenance

• Commit Hookscan use either pre- or post-commit hooks

• Automatic Backlinkscan use commit hooks to generate backlinks automatically

• Trigger Messageslinks could only be used as triggers

• Delete Cleanuppost-commit hook with backlinks can clean up links to new orphans automatically

Link Maintenance

Possibilities

Object metadataextensions?

Limitations

Not a graph database!

References[1/2]

• Riak Wikihttp://wiki.basho.com/

• Riak Mailing [email protected]

• Riak Contribhttp://contrib.basho.com/

• Riak Code Reposhttp://github.com/basho

References[1/2]

References[2/2]

• Secondary index support in Riak?Introducing Secondary Indices: http://www.oscon.com/oscon2011/public/schedule/detail/19806

• riak_core applicationse.g. Riak Search, riak_zab, riak_pipe

• CometKV database supports Active Storage Objects (ASO), handlers

http://vanish.cs.washington.edu/pubs/osdi2010comet.pdf

References[2/2]

Credits

• Basho & Riak logosCourtesy of Basho

• Icy chained fence photohttp://www.flickr.com/photos/existentist/342177746/

• Frozen chain photohttp://www.flickr.com/photos/skedonk/4197921511/

• Hook and chain photohttp://www.flickr.com/photos/37491467@N07/3871760093/

Questions?

Figure: http://www.flickr.com/photos/42682395@N04/

@SusanPotter

Questions?

Figure: http://www.flickr.com/photos/42682395@N04/

@SusanPotter

link walking with riak

Technology