riak search 2: yokozuna
DESCRIPTION
Ricon|West talk about YokozunaTRANSCRIPT
![Page 1: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/1.jpg)
Eric Redmond@coderoshi
Tuesday, November 12, 13
![Page 2: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/2.jpg)
[Slide Intentionally Left Blank]
Tuesday, November 12, 13
![Page 3: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/3.jpg)
A million values isn’t cool.You know what’s cool?...
Tuesday, November 12, 13
![Page 4: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/4.jpg)
A trillion values spread acrosslots of nodes that auto-index and you canalso search in cool ways like geographicallyor maybe you could extract from a pdf on the server side if you used the extractor API
Tuesday, November 12, 13
![Page 5: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/5.jpg)
┬─┬ ( ^_^ノ)
Setup
ノ
Tuesday, November 12, 13
![Page 6: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/6.jpg)
## The enabled Yokozuna set this 'on'.yokozuna = off
## The port number which Solr binds to.yokozuna.solr_port = 10014
## The port number which Solr JMX binds to.yokozuna.solr_jmx_port = 10013
## The arguments to pass to the Solr JVM. Non-standard## arguments, i.e. -XX, may not be portable across JVM## implementations. E.g. -XX:+UseCompressedStrings.yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops
## The data under which to store all Yokozuna related data.## Including the Solr index data.yokozuna.data_dir = ./data/yz
riak.conf
Tuesday, November 12, 13
![Page 7: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/7.jpg)
## The enabled Yokozuna set this 'on'.yokozuna = on
## The port number which Solr binds to.yokozuna.solr_port = 10014
## The port number which Solr JMX binds to.yokozuna.solr_jmx_port = 10013
## The arguments to pass to the Solr JVM. Non-standard## arguments, i.e. -XX, may not be portable across JVM## implementations. E.g. -XX:+UseCompressedStrings.yokozuna.solr_jvm_args = -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops
## The data under which to store all Yokozuna related data.## Including the Solr index data.yokozuna.data_dir = ./data/yz
riak.conf
Tuesday, November 12, 13
![Page 8: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/8.jpg)
riak-admin cluster join [email protected]... and so on...riak-admin cluster planriak-admin cluster commit
Tuesday, November 12, 13
![Page 9: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/9.jpg)
riak-admin cluster join [email protected]... and so on...riak-admin cluster planriak-admin cluster commit
riak-admin security add-user eric 12345riak-admin security add-user admin 123456riak-admin security grant yokozuna.search ON index simple TO ericriak-admin security grant yokozuna.admin ON index TO admin
Tuesday, November 12, 13
![Page 10: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/10.jpg)
require 'riak'client = Riak::Client.new
# create indexclient.create_search_index('simple')
# tie the index to bucket 'cats'bucket = Riak::Bucket.new(client, 'cats')bucket.props = { yz_index: 'simple' }
Tuesday, November 12, 13
![Page 11: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/11.jpg)
Indexing DatatypesTuesday, November 12, 13
![Page 12: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/12.jpg)
UTF8中搜索UTF8חיפוש בΑναζήτηση σε UTF8
Искать в UTF8
Search in UTF8
Tuesday, November 12, 13
![Page 13: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/13.jpg)
Advanced Searchaka. Next Level Search
Tuesday, November 12, 13
![Page 14: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/14.jpg)
solr = RSolr.connect(url: 'http://yokozuna01.bos1:8098/solr/docs')
resp = solr.get('select', params: {q: '*:*'})
Tuesday, November 12, 13
![Page 15: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/15.jpg)
Tuesday, November 12, 13
![Page 16: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/16.jpg)
Facets, Stats, and stuff
Tuesday, November 12, 13
![Page 17: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/17.jpg)
eDisMax
defType=edismax
Tuesday, November 12, 13
![Page 18: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/18.jpg)
Highlighting
Tuesday, November 12, 13
![Page 19: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/19.jpg)
Languages
Tuesday, November 12, 13
![Page 20: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/20.jpg)
Map/ReduceTuesday, November 12, 13
![Page 21: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/21.jpg)
{"inputs": { "module":"yokozuna", "function":"mapred_search", "arg":["docs","title_s:Key* AND language_s:en"] }, "query":[ {"map":{ "language":"javascript", "keep":false, "source":"function(v) { return [1]; }"}}, {"reduce"{ "language":"javascript", "keep":true, "name":"Riak.reduceSum" }} ]}
Tuesday, November 12, 13
![Page 22: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/22.jpg)
Tuesday, November 12, 13
![Page 23: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/23.jpg)
client.create_search_index('simple')
bucket = Riak::Bucket.new(client, 'people')bucket.props = { yz_index: 'simple' }
bucket = Riak::Bucket.new(client, 'cats')bucket.props = { yz_index: 'simple' }
bucket = Riak::Bucket.new(client, 'mountains')bucket.props = { yz_index: 'simple' }
Tuesday, November 12, 13
![Page 24: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/24.jpg)
Schemas
Tuesday, November 12, 13
![Page 25: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/25.jpg)
<schema name="default" version="1.5"> <fields> <field name="kinds" type="string" indexed="true" stored="false" multiValued="true" /> <field name="name" type="string" indexed="true" stored="true" />
<dynamicField name="*_ss" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="_yz_ed" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_pn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_fpn" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_vtag" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_node" type="_yz_str" indexed="true" stored="false"/> <field name="_yz_rk" type="_yz_str" indexed="true" stored="true"/> <field name="_yz_rb" type="_yz_str" indexed="true" stored="true"/> </fields></schema>
Tuesday, November 12, 13
![Page 26: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/26.jpg)
Fallen Nodes
Tuesday, November 12, 13
![Page 27: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/27.jpg)
Why?
Tuesday, November 12, 13
![Page 28: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/28.jpg)
ThermoclineTuesday, November 12, 13
![Page 29: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/29.jpg)
Thermocline of ReplicationTuesday, November 12, 13
![Page 30: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/30.jpg)
GoalsTuesday, November 12, 13
![Page 31: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/31.jpg)
But... why?Tuesday, November 12, 13
![Page 32: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/32.jpg)
Tuesday, November 12, 13
![Page 33: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/33.jpg)
How
Tuesday, November 12, 13
![Page 34: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/34.jpg)
Yokozuna = GlueTuesday, November 12, 13
![Page 35: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/35.jpg)
Extractorsyz_extractor.erlyz_json_extractor.erlyz_noop_extractor.erlyz_text_extractor.erlyz_xml_extractor.erlyz_doc.erl
Tuesday, November 12, 13
![Page 36: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/36.jpg)
Interfaces
yz_pb_admin.erlyz_pb_search.erlyz_schema.erlyz_wm_extract.erlyz_wm_index.erlyz_wm_schema.erlyz_wm_search.erl
Tuesday, November 12, 13
![Page 37: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/37.jpg)
SecurityTuesday, November 12, 13
![Page 38: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/38.jpg)
Permissions
riak-admin security grant yokozuna.admin ON index TO admin
riak-admin security grant yokozuna.search ON index TO user
riak-admin security grant yokozuna.admin ON schema TO admin
riak-admin security grant yokozuna.search ON index wiki TO user
Tuesday, November 12, 13
![Page 39: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/39.jpg)
Administrationyz_pb_admin.erlyz_schema.erlyz_wm_index.erlyz_wm_schema.erl
Tuesday, November 12, 13
![Page 40: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/40.jpg)
Searchesyz_pb_search.erlyz_wm_search.erl
Tuesday, November 12, 13
![Page 41: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/41.jpg)
Backend Magicyokozuna.erlyz_app.erlyz_general_sup.erlyz_kv.erlyz_misc.erlyz_sup.erl
Tuesday, November 12, 13
![Page 42: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/42.jpg)
Distribution
yz_cover.erlyz_events.erl
Tuesday, November 12, 13
![Page 43: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/43.jpg)
AAEyz_entropy.erlyz_entropy_mgr.erlyz_exchange_fsm.erlyz_index.erlyz_index_hashtree.erlyz_index_hashtree_sup.erl
Tuesday, November 12, 13
![Page 44: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/44.jpg)
Solr code
EntropyData.javaMonitor.javayz_solr.erlyz_solr_proc.erlyz_solr_sup.erl
Tuesday, November 12, 13
![Page 45: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/45.jpg)
Stats
yz_stat.erlyz_stat_worker.erl
Tuesday, November 12, 13
![Page 46: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/46.jpg)
“Testing only showsthe presence, not
the absence,of bugs”
aae_test.erlyokozuna_essential.erlyz_errors.erlyz_fallback.erlyz_flag_transitions.erlyz_index_admin.erlyz_languages.erlyz_mapreduce.erlyz_monitor_solr.erlyz_pb.erlyz_rs_migration.erlyz_rt.erlyz_schema_admin.erlyz_siblings.erlyz_solr_start_timeout.erlyz_stat_test.erlyz_wm_extract_test.erlyz_component_tests.erlyz_json_extractor_tests.erlyz_kv_tests.erlyz_misc_tests.erlyz_text_extractor_tests.erlyz_xml_extractor_tests.erlyz_driver.erlyz_file_terms.erl
Dijkstra
Tuesday, November 12, 13
![Page 47: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/47.jpg)
We Want More
• CRDT Support
• Simplify custom extractors
• Bucket Types (?)
• Users
Tuesday, November 12, 13
![Page 48: Riak Search 2: Yokozuna](https://reader034.vdocuments.mx/reader034/viewer/2022052622/55921dcf1a28ab73218b4651/html5/thumbnails/48.jpg)
Thanks
Eric Redmond@coderoshi
Tuesday, November 12, 13