cross datacenter replication in apache solr 6
TRANSCRIPT
![Page 1: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/1.jpg)
![Page 2: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/2.jpg)
Cross Datacenter Replication in Apache Solr 6
Shalin Shekhar Mangar Lucidworks Inc. @shalinmangar
![Page 3: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/3.jpg)
The standard for enterprise search.
of Fortune 500 uses Solr.
90%
![Page 4: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/4.jpg)
Agenda
• Review a typical Solr deployment architecture
• Challenges of running a Solr deployment across data centers
• Cross Data Centre Replication (CDCR) in Solr
• Setup and configuration
• Limitations
• Alternative strategies
• Future work
![Page 5: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/5.jpg)
Client ClientClient
Solr
Zookeeper
Datacenter
![Page 6: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/6.jpg)
CDCR Anti-patterns - Remote Solr instances
C
Solr
Zookeeper
DC 1
C C
DC 2
C C C
![Page 7: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/7.jpg)
CDCR Anti-patterns - Remote ZK and Solr
C
Solr
Zookeeper
DC 1
C C
DC 2
C C C
![Page 8: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/8.jpg)
CDCR Anti-patterns - Remote ZK and Solr
C
Solr
Zookeeper
DC 1
C C
DC 2
C C C
DC 3
![Page 9: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/9.jpg)
Why not a single Solr Cloud?
• Same update is transferred to each replica
• Synchronous indexing means burst-indexing is constrained by cross DC bandwidth
• Increased latency for indexing operations
• Need a ZooKeeper node in a 3rd DC to break ties
• Search requests are not DC-aware, may choose a remote replica
![Page 10: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/10.jpg)
Cross Datacenter Replication in Solr
• Let’s call it CDCR for short
• Accommodate two or more data centres
• Active/passive setup for disaster recovery
• Support limited bandwidth links
• Eventually consistent passive cluster
![Page 11: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/11.jpg)
Source: http://yonik.com/solr-cross-data-center-replication/
![Page 12: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/12.jpg)
CDCR in Solr 6
• Scalable: no SPoF and/or bottleneck
• Peer cluster can have a different replication factor
• Asynchronous updates; no penalty for indexing
• Push operations for low latency replication
• Low overhead — uses existing transaction logs and indexes
• Leader-to-leader communication ensures update is sent only once to peer cluster
![Page 13: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/13.jpg)
Target Cluster
Tune replication
Synchronize logs
CdcrUpdateLog
![Page 14: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/14.jpg)
Enable APIs
Update chains
Update chains
Update log
![Page 15: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/15.jpg)
CDCR APIs
• http://host:port/solr/collection_name/cdcr?action=START
• Control APIs: START, STOP, ENABLEBUFFER, DISABLEBUFFER, STATUS
• Monitoring APIs: QUEUES, OPS, ERRORS
![Page 16: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/16.jpg)
How to failover?
• Change configuration on target to make it the source
• Point indexers to the new target
• Change configuration on source to make it the new target
• May require stopping indexing during the conversion process — especially if you want to revert the change
![Page 17: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/17.jpg)
CDCR support in Solr 6+
• Active/passive setup either for disaster recovery or for low latency querying
• Solr clusters with existing data can be converted to a source cluster from Solr 6.2 onwards
• Low to medium indexing traffic
![Page 18: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/18.jpg)
CDCR Limitations and gotchas
• By default CDCR is disabled — invoke START to enable on both source and target
• Soft commits are not replicated to target — must schedule autoSoftCommit explicitly on target
• Different set of configurations required on source and target
• Daisy-chaining is possible but not well tested — add all targets to the same source cluster
![Page 19: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/19.jpg)
CDCR Limitations and gotchas
• Not suitable for applications requiring high throughput indexing — some knobs exist for tuning replication speeds
• Update log buffers can grow indefinitely when target clusters are down — can work around by disabling buffering for the time being if there is only one target
• No automatic failover between source and target — explicit actions required to modify configurations and point indexing pipelines to the new source
• No Active/active setup
![Page 20: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/20.jpg)
Alternative strategy
• Use a proper queue such as Apache Kafka to feed source and target DCs simultaneously
• Use external versions in conjunction with versions generated by Solr — DocBasedVersionConstraintsProcessorFactory
• Watch the video for “Solr Cross-Datacenter Replication and Consistency at Scale” by Oliver Bates, Apple Inc. — http://sched.co/8ArU
• Pros: Supports high indexing throughputs and active/active replication
• Cons: Additional systems required, managing consistency is difficult and requires in depth Solr expertise, all atomic updates must go to a single DC, cannot support delete-by-query
![Page 21: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/21.jpg)
Problems we solved
• Synchronous indexing to replicas — build separate asynchronous indexing pipeline
• Limited size of the update log — use update log as the queue
• How to track replication progress to preserve consistency on target clusters in case the source leader dies — checkpoints
• Bootstrapping target cluster with indexes when update logs are incomplete
• New replicas on source have no logs to replicate — replicate update logs during recovery
![Page 22: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/22.jpg)
Future work
• Move configuration out of solrconfig.xml and into API calls
• Dynamically add/remove/change target cluster information
• Cap update log to a max size and fall back to index replication if necessary
• Refactor and combine CdcrUpdateLog
• Better monitoring: capture transfer rate and latency info
• Add support for rate limiting replication between source and target
• Active/active?
![Page 23: Cross Datacenter Replication in Apache Solr 6](https://reader034.vdocuments.mx/reader034/viewer/2022042723/5870672e1a28ab48378b52e5/html5/thumbnails/23.jpg)
Resources
• CDCR page on ref guide — https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62687462
• http://yonik.com/solr-cross-data-center-replication/
• https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents