wiredtiger in-memory - vs wiredtiger b-tree...∙ introducing percona memory engine for mongodb ∙...

WiredTiger In-Memoryvs WiredTiger B-Tree

October, 5, 2016 — Movenpick Hotel — AmsterdamSveta Smirnova

∙What is Percona Memory Engine for MongoDB?

∙Typical use cases

∙Advanced Memory Engine

Table of Contents

2

What is Percona Memory Engine for MongoDB?

3

∙ Up to 1000 times faster for OLTP wokloads∙ 10 times faster for read-only workloads∙ Stable throughput∙ No checkpointing

∙ No jitter

Extremely fast In-Memory storage

4

∙ Document-level locking

∙ B-Tree

∙ Practically WiredTiger, but without diskaccess

Based on WiredTiger

5

∙ Doesn’t store data on disk∙ Except small amount of statistics∙ You can control when to log statistics with

option –inMemoryStatisticsLogDelaySecs∙ Still must specify –dbpath

sveta@Thinkie:~/mongo_tests$ ls -lh single/

total 40K

drwxrwxr-x 2 sveta sveta 4,0K Eyl 29 15:00 diagnostic.data

-rw-r--r-- 1 sveta sveta 6 Eyl 29 14:58 mongod.lock

-rw-rw-r-- 1 sveta sveta 93 Eyl 29 14:55 storage.bson

∙ Data does not persist between restarts

WiredTiger without storage

6

∙ –storageEngine=inMemory

∙ Can be only engine on MongoDB server∙ MongoDB restriction, applicable to all engines

∙ Heterogeneous replication and shardingsetups supported

How to enable Memory Engine?

7

∙ Engine can use up to –inMemorySizeGB∙ If data exceeds this amount

∙ WT CACHE FULL error is returned for allkinds of operations that cause user data sizeto grow

INSERTCREATEUPDATE

∙ Reads are not affected

How to control memory usage

8

∙ 100% Open Source

∙ Code available at GitHub

∙ Free for all Percona users and customers

Open Source

9

Typical use cases for Percona Memory Engine

10

∙ Session management∙ Store active sessions in memory∙ Users will receive answer almost immediately∙ Reduce application response time

dramatically

∙ Various temporary collections∙ All you used to store in memcached

Application cache

11

∙ Application runtime data which does notrequire on-disk storage

∙ Intermediary results of calculations

∙ User-specific options

∙ Your idea

Transient Runtime State

12

∙ Thousand-lines aggregations

∙ Temporary collections to store intermediarydata

∙ Complicated queries

Sophisticated data manipulation

13

∙ Large aggregations might be slow∙ Especially if use many collections∙ Often this is not avoidable

∙ To calculate number of distinct values youneed to read whole index

∙ Fast dedicated server is great solution

Real-Time Analytics

14

∙ Data sharing between multi-tier ormulti-language applications

English labels

∙ Articles∙ Pictures∙ Contact

information∙ Other content

Russian labels

Multi-tier object sharing

15

∙ Are you tired to wait when data, needed forapplication test, loads?

∙ Any change in test data causes delay?∙ With Memory engine you can reduce

turnaround time for automated applicationtests.

∙ And still use same syntax

Application Testing

16

Advanced Percona Memory Engine

17

∙ Are you amazed with speed of the Memoryengine?

∙ But still need data to persist betweenrestarts?

∙ You can combine both Memory and WiredTiger in Replica Set or Sharded Cluster

Best of both worlds

18

∙ Setup 2 or more Memory replicas whichcan be Primary

∙ Let WiredTiger to persist data on disk∙ In rare cases if all Memory replicas crash at

the same time you will loose fewtransactions∙ Number of transactions depends on the

latency between In-Memory Primary replicaand WiredTiger replica

Hidden WiredTiger, storing changes in Replica Set

19

Memory

WiredTiger

Memory

Hidden WiredTiger, storing changes in Replica Set

20

rs.initiate(

... {

... "_id" : "rs",

... "members" : [

... {"_id" : 0, "host" : "inMemory1", "priority" : 1},

... {"_id" : 1, "host" : "inMemory2", "priority" : 1},

... {"_id" : 2, "host" : "WiredTiger", "priority" : 0, "hidden" : true}

... ]

... }

)

Hidden WiredTiger to store on disk: example setup

21

∙ Make WiredTiger Primary

∙ Move all reads to read-only Memoryreplicas

∙ Writes will be slow

WiredTiger as Primary in Replica Set

22

Memory

WiredTiger

Memory

WiredTiger as Primary in Replica Set

23

∙ Create Sharded Cluster using Memorynodes only

∙ Split data between nodes

∙ Create copies of data to prevent data loss

Scaling beyond the RAM of a single server

24

Shard 1 Shard 2

Scaling beyond the RAM

25

Shard 1 Shard 2

R1

R2

R3 R4

R5

R6

Scaling beyond the RAM: add redundancy

26

∙ You can use both engines in the ShardedCluster

∙ Split data∙ Session data on Memory nodes∙ Persistent data on WiredTiger node(s)

∙ Duplicate Memory shards to avoid loosingdata

Memory and WiredTiger in Sharded Cluster

27

∙ You can have sharded nodes which useMemory engine

∙ Make them parts of Replica Set

∙ Let hidden WiredTiger member to persistdata on disk

Sharded Cluster Memory and Replica Sets

28

∙ User posts changing rarely are stored ondisk

∙ Session data stored using Memory engine∙ Active comments (last 24 hours) and

actively accessed posts are cached inMemory node

Example: blog application

29

mongos> sh.addShardTag("shard01", "memory") // Memory node

WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )

mongos> sh.addShardTag("shard02", "memory") // Memory node

WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )

mongos> sh.addShardTag("shard03", "persist") // WiredTiger node

WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )

mongos> sh.addShardTag("shard04", "persist") // WiredTiger node

WriteResult( "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 )

Example: tag shards

30

mongos> sh.addTagRange("blog.sessions", { sid: 0 }, { sid: 1000000 }, "memory")

WriteResult({

"nMatched" : 0,

"nUpserted" : 1,

"nModified" : 0,

" id" : {

"ns" : "blog.sessions",

"min" : {

"sid" : 0

}

}

})

Example: split data

31

mongos> sh.addTagRange("blog.comments", { store: "persist", cid: 0 },

... { store: "persist", cid: 1000000 }, "persist")

WriteResult({

...

" id" : {

"ns" : "blog.comments",

"min" : {

"store" : "persist",

"cid" : 0

}}})

mongos> sh.addTagRange("blog.comments", { store: "memory", cid: 0 },

... { store: "memory", cid: 1000000 }, "memory")

WriteResult({

...

Example: split data

31

mongos> sh.addTagRange("blog.posts", { pid: 0 }, { pid: 1000000 }, "persist")

WriteResult({

"nMatched" : 0,

"nUpserted" : 1,

"nModified" : 0,

" id" : {

"ns" : "blog.posts",

"min" : {

"pid" : 0

}

}

})

Example: split data

31

∙ Percona Memory Engine replacesWiredTiger when you need better speedand can afford loosing data

∙ Can be used in setups which combine bothhigh performance of the Memory engineand data persistence of WiredTiger

∙ Open Source

Summary

32

∙ David Bennett

∙ David Murphy

∙ Fernando Ipar

∙ Denis Protyvenskyi

Special thanks

33

∙ Benchmarks for Percona Memory Engine

∙ Introducing Percona Memory Engine forMongoDB

∙ Percona Server for MongoDB manual

∙ Source code for Percona Memory Engine

More informaiton

34

Rate My Session!

35

???

Place for your questions

36

http://www.slideshare.net/SvetaSmirnova

https://twitter.com/svetsmirnova

Thank you!

37