social networks and the richness of data
DESCRIPTION
Social networks by their nature deal with large amounts of user-generated data that must be processed and presented in a time sensitive manner. Much more write intensive than previous generations of websites, social networks have been on the leading edge of non-relational persistence technology adoption. This talk presents how Germany's leading social networks Schuelervz, Studivz and Meinvz are incorporating Redis and Project Voldemort into their platform to run features like activity streams.TRANSCRIPT
![Page 1: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/1.jpg)
Social Networks and the Richness of Data
Getting distributed Webservices Done with NoSQL
Fabrizio Schmidt, Lars GeorgeVZnet Netzwerke Ltd.
Mittwoch, 10. März 2010
![Page 2: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/2.jpg)
Content
• Unique Challenges
• System Evolution
• Architecture
• Activity Stream - NoSQL
• Lessons learned, Future
Mittwoch, 10. März 2010
![Page 3: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/3.jpg)
Unique Challenges
• 16 Million Users
• > 80% Active/Month
• > 40% Active/Daily
• > 30min Daily Time on Site
Mittwoch, 10. März 2010
![Page 4: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/4.jpg)
Mittwoch, 10. März 2010
![Page 5: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/5.jpg)
Unique Challenges• 16 Million Users
• 1 Billion Relationships
• 3 Billion Photos
• 150 TB Data
• 13 Million Messages per Day
• 17 Million Logins per Day
• 15 Billion Requests per Month
• 120 Million Emails per Week
Mittwoch, 10. März 2010
![Page 6: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/6.jpg)
Old System - Phoenix
• LAMP
• Apache + PHP + APC (50 req/s)
• Sharded MySQL Multi-Master Setup
• Memcache with 1 TB+
Monolithic Single Service, Synchronous
Mittwoch, 10. März 2010
![Page 7: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/7.jpg)
Old System - Phoenix
• 500+ Apache Frontends
• 60+ Memcaches
• 150+ MySQL Servers
Mittwoch, 10. März 2010
![Page 8: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/8.jpg)
Old System - Phoenix
Mittwoch, 10. März 2010
![Page 9: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/9.jpg)
DON‘T PANIC
Mittwoch, 10. März 2010
![Page 10: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/10.jpg)
Asynchronous Services
• Basic Services
• Mobile
• CDN Purge
• ...
• Java (e.g. Tomcat)
• RabbitMQ
Mittwoch, 10. März 2010
![Page 11: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/11.jpg)
First Services
Mittwoch, 10. März 2010
![Page 12: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/12.jpg)
Phoenix - RabbitMQ
1. PHP Implementation of AMQP Client
Too slow!
2. PHP C - Extension (php-amqp http://code.google.com/p/php-amqp/)
Fast enough
3. IPC - AMQP Dispatcher C-Daemon
That‘s it! But not released so far
Mittwoch, 10. März 2010
![Page 13: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/13.jpg)
IPC - AMQP Dispatcher
Mittwoch, 10. März 2010
![Page 14: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/14.jpg)
Activity Stream
Mittwoch, 10. März 2010
![Page 15: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/15.jpg)
Old Activity Stream
• Memcache only - no persistence
• Status updates only
• #fail on users with >1000 friends
• #fail on memcache restart
Mittwoch, 10. März 2010
![Page 16: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/16.jpg)
Old Activity Stream
• Memcache only - no persistence
• Status updates only
• #fail on users with >1000 friends
• #fail on memcache restart
We cheated!
Mittwoch, 10. März 2010
![Page 17: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/17.jpg)
Old Activity Stream
• Memcache only - no persistence
• Status updates only
• #fail on users with >1000 friends
• #fail on memcache restart
We cheated!
source: internet
Mittwoch, 10. März 2010
![Page 18: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/18.jpg)
Social Network Problem
• >15 different Events
• Timelines
• Aggregation
• Filters
• Privacy
= Twitter Problem???
Mittwoch, 10. März 2010
![Page 19: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/19.jpg)
Do the Math!
Mittwoch, 10. März 2010
![Page 20: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/20.jpg)
Do the Math!
18M Events/day sent to ~150 friends
Mittwoch, 10. März 2010
![Page 21: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/21.jpg)
Do the Math!
18M Events/day sent to ~150 friends
=> 2700M timeline inserts / day
Mittwoch, 10. März 2010
![Page 22: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/22.jpg)
Do the Math!
18M Events/day sent to ~150 friends
=> 2700M timeline inserts / day
20% during peak hour
Mittwoch, 10. März 2010
![Page 23: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/23.jpg)
Do the Math!
18M Events/day sent to ~150 friends
=> 2700M timeline inserts / day
20% during peak hour
=> 3.6M event inserts/hour - 1000/s
Mittwoch, 10. März 2010
![Page 24: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/24.jpg)
Do the Math!
18M Events/day sent to ~150 friends
=> 2700M timeline inserts / day
20% during peak hour
=> 3.6M event inserts/hour - 1000/s
=> 540M timeline inserts/hour - 150000/s
Mittwoch, 10. März 2010
![Page 25: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/25.jpg)
18M Events/day sent to ~150 friends
=> 2700M timeline inserts / day
20% during peak hour
=> 3.6M event inserts/hour - 1000/s
=> 540M timeline inserts/hour - 150000/s
Mittwoch, 10. März 2010
![Page 26: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/26.jpg)
New Activity Stream
• Social Network Problem
• Architecture
• NoSQL Systems
Mittwoch, 10. März 2010
![Page 27: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/27.jpg)
New Activity Stream
• Social Network Problem
• Architecture
• NoSQL Systems
Do it right!
Mittwoch, 10. März 2010
![Page 28: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/28.jpg)
New Activity Stream
• Social Network Problem
• Architecture
• NoSQL Systems
Do it right!
source: internet
Mittwoch, 10. März 2010
![Page 29: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/29.jpg)
Architecture
Mittwoch, 10. März 2010
![Page 30: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/30.jpg)
FAS Federated Autonomous Services
• Nginx + Janitor
• Embedded Jetty + RESTeasy
• NoSQL Storage Backends
Mittwoch, 10. März 2010
![Page 31: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/31.jpg)
FAS Federated Autonomous
Services
Mittwoch, 10. März 2010
![Page 32: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/32.jpg)
Activity Streamas a service
Requirements:
• Endless scalability
• Storage & cloud independent
• Fast
• Flexible & extensible data model
Mittwoch, 10. März 2010
![Page 33: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/33.jpg)
Thinking in layers...
Mittwoch, 10. März 2010
![Page 34: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/34.jpg)
ActivityStreamas a service
Mittwoch, 10. März 2010
![Page 35: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/35.jpg)
ActivityStreamas a service
Mittwoch, 10. März 2010
![Page 36: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/36.jpg)
NoSQL Schema
Mittwoch, 10. März 2010
![Page 37: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/37.jpg)
Event is sent in by piggybacking the request
NoSQL Schema
Event
Mittwoch, 10. März 2010
![Page 38: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/38.jpg)
Generate itemID - unique ID of the event
NoSQL Schema
EventGenerate ID
Mittwoch, 10. März 2010
![Page 39: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/39.jpg)
NoSQL Schema
EventGenerate IDSave Item itemID => stream_entry - save the event with meta information
Mittwoch, 10. März 2010
![Page 40: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/40.jpg)
Insert into the timeline of each recipient
recipient → [[itemId, time, type],…]
Insert into the timeline of the event originator
sender → [[itemId, time, type],…]
NoSQL Schema
EventGenerate IDSave ItemUpdate Indexes
Mittwoch, 10. März 2010
![Page 41: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/41.jpg)
NoSQL Schema
EventGenerate IDSave Item
Mittwoch, 10. März 2010
![Page 42: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/42.jpg)
MRI
(Redis)
Mittwoch, 10. März 2010
![Page 43: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/43.jpg)
MRI
(Redis)
Mittwoch, 10. März 2010
![Page 44: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/44.jpg)
Push the Message directly to all MRIs
➡ {number of Recipients ~150} updates
Special profiles and some users have >500 recipients
➡>500 pushes to recipient timelines => stress the system!
Architecture: PushMessage Recipient Index (MRI)
Mittwoch, 10. März 2010
![Page 45: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/45.jpg)
ORI
(Voldemort/Redis)
Mittwoch, 10. März 2010
![Page 46: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/46.jpg)
ORI
(Voldemort/Redis)
Mittwoch, 10. März 2010
![Page 47: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/47.jpg)
NO Push to MRIs at all
➡ 1 Message + 1 Originator Index Entry
Special profiles and some users have >500 friends
➡ get >500 ORIs on read => stress the system
Architecture: PullOriginator Index (ORI)
Mittwoch, 10. März 2010
![Page 48: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/48.jpg)
• Identify Users with recipient lists >{limit}
• Only push updates with recipients <{limit} to MRI
• Pull special profiles and users with >{limit} from ORI
• Identify active users with a bloom/bit filter for pull
Architecture: PushPullORI + MRI
Mittwoch, 10. März 2010
![Page 49: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/49.jpg)
Activity Filter
• Reduce read operations on storage
• Distinguish user activity levels
• In memory and shared across keys and types
• Scan full day of updates for16M users on a per minute granularity for 1000 friends in < 100msecs
Lars
Mittwoch, 10. März 2010
![Page 50: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/50.jpg)
Activity Filter
Mittwoch, 10. März 2010
![Page 51: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/51.jpg)
NoSQL
Mittwoch, 10. März 2010
![Page 52: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/52.jpg)
NoSQL: RedisORI + MRI on Steroids
• Fast in memory Data-Structure Server
• Easy protocol
• Asynchronous Persistence
• Master-Slave Replication
• Virtual-Memory
• JRedis - The Java client
Mittwoch, 10. März 2010
![Page 53: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/53.jpg)
Data-Structure Server
• Datatypes: String, List, Sets, ZSets
• We use ZSets (sorted sets) for the Push Recipient Indexes
Insert
for (recipient : recipients) { jredis.zadd(recipient.id, streamEntryIndex); }
Get
jredis.zrange(streamOwnerId, from, to) jredis.zrangebyscore(streamOwnerId, someScoreBegin, someScoreEnd)
NoSQL: RedisORI + MRI on Steroids
Mittwoch, 10. März 2010
![Page 54: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/54.jpg)
Persistence - AOF and Bgsave
AOF - append only file - append on operation
Bgsave - asynchronous snapshot - configurable (timeperiod or every n operations) - triggered directly
We use AOF as itʻs less memory hungryCombined with bgsave for additional backups
NoSQL: RedisORI + MRI on Steroids
Mittwoch, 10. März 2010
![Page 55: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/55.jpg)
Virtual - Memory
Storing Recipient Indexes for 16 mio users à ~500 entries would lead to >250 GB of RAM needed
With Virtual Memory activated Redis swaps less frequented values to disk ➡ Only your hot dataset is in memory➡ 40% logins per day / only 20% of these in peak ~ 20GB needed for hot dataset
NoSQL: RedisORI + MRI on Steroids
Mittwoch, 10. März 2010
![Page 56: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/56.jpg)
Jredis - Redis java client
• Pipelining support (sync and async semantics)
• Redis 1.2.3 compliant
The missing parts
• No consistent hashing
• No rebalancing
NoSQL: RedisORI + MRI on Steroids
Mittwoch, 10. März 2010
![Page 57: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/57.jpg)
Message Store
(Voldemort)
Mittwoch, 10. März 2010
![Page 58: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/58.jpg)
Message Store
(Voldemort)
Mittwoch, 10. März 2010
![Page 59: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/59.jpg)
NoSQL: VoldemortNo #fail Messagestore (MS)
• Key-Value Store
• Replication
• Versioning
• Eventual Consistency
• Pluggable Routing / Hashing Strategy
• Rebalancing
• Pluggable Storage-Engine
Mittwoch, 10. März 2010
![Page 60: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/60.jpg)
Configuring replication, reads and writes
<store> <name>stream-ms</name> <persistence>bdb</persistence> <routing>client</routing> <replication-factor>3</replication-factor> <required-reads>2</required-reads> <required-writes>2</required-writes> <prefered-reads>3</prefered-reads> <prefered-writes>3</prefered-writes> <key-serializer><type>string</type></key-serializer> <value-serializer><type>string</type></value-serializer> <retention-days>8</retention-days></store>
NoSQL: VoldemortNo #fail Messagestore (MS)
Mittwoch, 10. März 2010
![Page 61: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/61.jpg)
Write:client.put(key, myVersionedValue);
Update(read-modify-write):public class MriUpdateAction extends UpdateAction<String, String> { public MriUpdateAction(String key, ItemIndex index) { this.key = key; this.index = index; } @Override public void update(StoreClient<String, String> client) { Versioned<String> versionedJson = client.get(this.key); versionedJson.setObject("my value"); client.put(this.key, versionedJson); }}
NoSQL: VoldemortNo #fail Messagestore (MS)
Mittwoch, 10. März 2010
![Page 62: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/62.jpg)
Eventual Consistency - Readpublic MriInconsistencyResolver implements InconsistencyResolver<Versioned<String>> {
public List<Versioned<String>> resolveConflicts(List<Versioned<String>> items){ Versioned<String> vers0 = items.get(0); Versioned<String> vers1 = items.get(1); if(vers0 == null && vers1 == null) { return null; }
List<Versioned<String>> li = new ArrayList<Versioned<String>>(1); if(vers0 == null) { li.add(vers1); return li; } if(vers1 == null) { li.add(vers0); return li; } // resolve your inconsistency here e.g. merge to lists }}
The default inconsistency resolver automatically takes the newer version
NoSQL: VoldemortNo #fail Messagestore (MS)
Mittwoch, 10. März 2010
![Page 63: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/63.jpg)
Configuration
• Choose a big number of partitions
• Reduce the size of the BDB append log
• Balance Client and Server Threadpools
NoSQL: VoldemortNo #fail Messagestore (MS)
Mittwoch, 10. März 2010
![Page 64: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/64.jpg)
Concurrent ID
Generator
Mittwoch, 10. März 2010
![Page 65: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/65.jpg)
Concurrent ID
Generator
Mittwoch, 10. März 2010
![Page 66: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/66.jpg)
NoSQL: HazelcastConcurrent ID Generator (CIG)
• In Memory Data Grid
• Dynamically Scales
• Distributed java.util.{Queue|Set|List|Map} and more
• Dynamic Partitioning with Backups
• Configurable Eviction
• Persistence
Mittwoch, 10. März 2010
![Page 67: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/67.jpg)
Cluster-wide ID Generation:
• No UUID because of architecture constraint
• IDs of Stream Entries generated via Hazelcast
• Replication to avoid loss of count
• Background persistence used for disaster recovery
NoSQL: HazelcastConcurrent ID Generator (CIG)
Mittwoch, 10. März 2010
![Page 68: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/68.jpg)
Generate Unique Sequencial Numbers (Distributed Autoincrement):
• Nodes get ranges assigned (node1: 10000-19999, node2: 20000-29999 ID's)
• IDs per range locally incremented on the node (thread safe/atomic)
• Distributed locks secure range assignment for nodes
NoSQL: HazelcastConcurrent ID Generator (CIG)
Mittwoch, 10. März 2010
![Page 69: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/69.jpg)
Example Configuration
<map name=".vz.stream.CigHazelcast">
<map-store enabled="true">
<class-name>net.vz.storage.CigPersister</class-name><write-delay-seconds>0</write-delay-seconds>
</map-store>
<backup-count>3</backup-count>
</map>
NoSQL: HazelcastConcurrent ID Generator (CIG)
Mittwoch, 10. März 2010
![Page 70: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/70.jpg)
Future use-cases:
• Advanced preaggregated cache
• Distributed Executions
NoSQL: HazelcastConcurrent ID Generator (CIG)
Mittwoch, 10. März 2010
![Page 71: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/71.jpg)
Lessons Learned
• Start benchmarking and profiling your app early!
• A fast and easy Deployment keeps the motivation up
• Configure Voldemort carefully (especially on large heap machines)
• Read the mailing lists of the #nosql system you use
• No Solution in docs? - read the sources!
• At some point stop discussing and just do it!
Mittwoch, 10. März 2010
![Page 72: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/72.jpg)
In Progress
• Network Stream
• Global Public Stream
• Stream per Location
• Hashtags
Mittwoch, 10. März 2010
![Page 73: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/73.jpg)
Future
• Geo Location Stream
• Third Party API
Mittwoch, 10. März 2010
![Page 74: Social Networks and the Richness of Data](https://reader034.vdocuments.mx/reader034/viewer/2022042623/54b7795e4a795938168b458a/html5/thumbnails/74.jpg)
Questions?
Mittwoch, 10. März 2010