1. Amazon DynamoDB - Advait Deo

2. Agenda Why noSQL ? noSQL World ACID vs CAP DynamoDB What is it ? DynamoDB Architecture Conditional Writes Strongly consistent vs Eventually consistent Provisioned throughput Query vs Scan DynamoDB parameters Secondary Index Item Collections Limitations AWS CLI Table operations 3. Why noSQL ? Data Velocity Huge data getting generated in less time Data Variety Structured, semi-structures and unstructured Data Volume Terabytes and petabytes of data Dynamic Schema Flexible data model Auto-sharding Horizontal scalability Continuous Availability Integrated Caching Replication Automatic replication to support high availability Dynamic provisioned throughput 4. Why noSQL ? Scalability advantage 5. Why noSQL ? Scalability advantage 6. noSQL world Database TypePropertiesExampleKey-values Stores Simplest noSQL datbase Every item is a set of key-value pairDynamoDB (amazon), Amazon SimpleDB (amazon), Valdemort (linkedin), MemcacheDB (zinga), Azure (microsoft), Redis (VMWare)Column Family StoresOptimized for queries over large Hbase (powerset), Cassandra datasets (facebook), BigTable (Google) Contains key-value pair where key if mapped to set of colums In analogy with RDBMS, a column family is a table Document Databases Graph Databasespair each key with a complex data structure called document Documents can contain many different key-value pairsCouchDB, MongoDB (10gen)Stores information about network, social connectionsInfiniteGraph, Neo4J (Neo tech), InfoGrid, FlockDB (Twitter) 7. ACID vs CAP ACID properties relates to relational databases Atomicity Requires each transaction to be all or nothing Consistency Brings databases from one valid state to another Isolation Takes care of concurrent execution and avoid overwritting Durability Commited transactions will remain so in the event of power loss, crashes etc 8. ACID vs CAP CAP theorem relates to noSQL database (pick any two) Consistency All nodes see the same data at the same time Availability A guarantee that every request receives a response about whether it was successful or failed Partition tolerance The system continues to operate despite arbitrary message loss or failure of part of the system) 9. ACID vs CAP 10. Amazon DynamoDB What is it ? Fully managed nosql database service on AWS Data model in the form of tables Data stored in the form of items (name value attributes) Automatic scaling Provisioned throughput Storage scaling Distributed architecture Easy Administration Monitoring of tables using CloudWatch Integration with EMR (Elastic MapReduce) Analyze data and store in S3 11. Amazon DynamoDB What is it ? key=valuekey=valuekey=valuekey=valueTable Item (64KB max) Attributes Primary key (mandatory for every table) Hash or Hash + Range Data model in the form of tables Data stored in the form of items (name value attributes) Secondary Indexes for improved performance Local secondary index Global secondary index Scalar data type (number, string etc) or multi-valued data type (sets) 12. DynamoDB Architecture True distributed architecture Data is spread across hundreds of servers called storage nodes Hundreds of servers form a cluster in the form of a ring Client application can connect using one of the two approaches Routing using a load balancer Client-library that reflects Dynamos partitioning scheme and can determine the storage host to connectAdvantage of load balancer no need for dynamo specific code in client application Advantage of client-library saves 1 network hop to load balancer Synchronous replication is not achievable for high availability and scalability requirement at amazon DynamoDB is designed to be always writable storage solution Allows multiple versions of data on multiple storage nodes Conflict resolution happens while reads and NOT during writes Syntactic conflict resolution Symantec conflict resolution 13. DynamoDB Architecture - Partitioning Data is partitioned over multiple hosts called storage nodes (ring) Uses consistent hashing to dynamically partition data across storage hosts Two problems associated with consistent hashing Hashing of storage hosts can cause imbalance of data and load Consistent hashing treats every storage host as same capacity A [3,4]14B [1]2 3 C[2]Initial Situation 14. DynamoDB Architecture - Partitioning Data is partitioned over multiple hosts called storage nodes (ring) Uses consistent hashing to dynamically partition data across storage hosts Two problems associated with consistent hashing Hashing of storage hosts can cause imbalance of data and load Consistent hashing treats every storage host as same capacity A[4]A [3,4]144 B [1]B[1] D[2,3]22 313 C[2]Initial SituationSituation after C left and D joined 15. DynamoDB Architecture - Partitioning Solution Virtual hosts mapped to physical hosts (tokens) Number of virtual hosts for a physical host depends on capacity of physical host Virtual nodes are function-mapped to physical nodesABDPhysical HostCB CJ I H Virtual Host 16. DynamoDB Architecture Membership Change Gossip protocol used for node membership Every second, storage node randomly contact a peer to bilaterally reconcile persisted membership history Doing so, membership changes are spread and eventually consistent membership view is formed When a new members joins, adjacent nodes adjust their object and replica ownership When a member leaves adjacent nodes adjust their object and replica and distributes keys owned by leaving member HAG BFReplication factor = 3 ECD 17. DynamoDB Architecture Membership Change Gossip protocol used for node membership Every second, storage node randomly contact a peer to bilaterally reconcile persisted membership history Doing so, membership changes are spread and eventually consistent membership view is formed When a new members joins, adjacent nodes adjust their object and replica ownership When a member leaves adjacent nodes adjust their object and replica and distributes keys owned by leaving member HAG BFReplication factor = 3 ECD 18. DynamoDB Architecture Membership Change Gossip protocol used for node membership Every second, storage node randomly contact a peer to bilaterally reconcile persisted membership history Doing so, membership changes are spread and eventually consistent membership view is formed When a new members joins, adjacent nodes adjust their object and replica ownership When a member leaves adjacent nodes adjust their object and replica and distributes keys owned by leaving member HAG BFReplication factor = 3 ECD 19. DynamoDB Architecture ReplicationHash value of Key K APreference list A B DBDPhysical HostCBCI HEach data item is replicated N times replication factor Storage node in charge of storing key K replicates to its N-1 successor clockwise Every node has Preference list which has mapping of which key belongs where based on consistent hashing Preference list skips virtual nodes and contains only distinct physical nodesJ Virtual Host 20. DynamoDB Architecture Data Versioning Eventual Consistency data propagates asynchronously Its possible to have multiple version on different storage nodes If latest data is not available on storage node, client will update the old version of data Conflict resolution is done using vector clock Vector clock is a metadata information added to data when using get() or put() Vector clock effectively a list of (node, counter) pairs One vector clock is associated with every version of every object One can determine two version has causal ordering or parallel branches by examining vector clocks Client specify vector clock information while reading or writing data in the form of context Dynamo tries to resolve conflict among multiple version using syntactic reconciliation If syntactic reconciliation does not work, client has to resolve conflict using semantic reconciliation Application should be designed to acknowledge multiple version and should have logic to reconcile the same 21. DynamoDB Architecture Data VersioningSxClientAdd 2 items [Sx,1]SySzTime 22. DynamoDB Architecture Data VersioningSxClientAdd 2 items [Sx,1]Add 1 item [Sx,2]SySzTime 23. DynamoDB Architecture Data VersioningSxClientAdd 2 items [Sx,1]Add 1 item [Sx,2]Add delete 1 item [Sx,2][Sy,1]SySzTime 24. DynamoDB Architecture Data VersioningSxClientAdd 2 items [Sx,1]Add 1 item [Sx,2]SyAdd delete 1 item [Sx,2][Sy,1]SzAdd 3 items [Sx,2][Sz,1]Time 25. DynamoDB Architecture Data VersioningSxClientAdd 2 items [Sx,1]Final Cart 5 items [Sx,3][Sy,1][Sz,1]Add 1 item [Sx,2]SyAdd delete 1 item [Sx,2][Sy,1]SzAdd 3 items [Sx,2][Sz,1]Time 26. DynamoDB Architecture Data Versioning Syntactic reconciliation (handled by dynamoDB as [Sx,2] > [Sx,1])SxClientAdd 2 items [Sx,1]Final Cart 5 items [Sx,3][Sy,1][Sz,1]Add 1 item [Sx,2]SyAdd delete 1 item [Sx,2][Sy,1]SzAdd 3 items [Sx,2][Sz,1]Time 27. DynamoDB Architecture Data Versioning Syntactic reconciliation (handled by dynamoDB as [Sx,2] > [Sx,1])SxClientAdd 2 items [Sx,1]Symantec reconciliation (handled by client application) Is [Sx,2][Sy,1] > [Sx,2][Sz,1] ??? Final Cart 5 items [Sx,3][Sy,1][Sz,1]Add 1 item [Sx,2]SyAdd delete 1 item [Sx,2][Sy,1]SzAdd 3 items [Sx,2][Sz,1]Time 28. DynamoDB Architecture Data Versioning Syntactic reconciliation (handled by dynamoDB as [Sx,2] > [Sx,1])SxClientSySzAdd 2 items [Sx,1]Symantec reconciliation (handled by client application) Is [Sx,2][Sy,1] > [Sx,2][Sz,1] ??? Final Cart 5 items [Sx,3][Sy,1][Sz,1]Add 1 item [Sx,2]Add delete 1 item [Sx,2][Sy,1]Syntactic reconciliation (handled by dynamoDB as [Sx,1][Sz,1] > [Sx,1])Add 3 items [Sx,2][Sz,1]Add 1 item [Sx,1][Sz,1]Time 29. DynamoDB Architecture Data Versioning Syntactic reconciliation (handled by dynamoDB as [Sx,2] > [Sx,1])SxClientSySzAdd 2 items [Sx,1]Symantec reconciliation (handled by client application) Is [Sx,2][Sy,1] > [Sx,2][Sz,1] ??? Final Cart 5 items [Sx,3][Sy,1][Sz,1]Add 1 item [Sx,2]Add delete 1 item [Sx,2][Sy,1]Syntactic reconciliation (handled by dynamoDB as [Sx,1][Sz,1] > [Sx,1])Add 3 items [Sx,2][Sz,1]Add 1 item [Sx,1][Sz,1]TimeIf counters on first objects vector clock are less than or equal to all of the nodes in second vector clock, then first is an ancestor of second and can be forgotten 30. DynamoDB Architecture get() and put() 2 strategies client uses to select a node Node handling read and write operation is called coordinator Coordinator should be among first in top N nodes in preference list For reads and writes dynamo uses consistency protocol similar to quorum system Consistency protocol has 3 variables N Number of replicas of data to be read or written W Number of nodes that must participate in successful write operation R Number of machines contacted in read operationPut() Generic load-balancer that will select a node based on load Partition aware client-library which routes request to right nodeUpon receiving put() request coordinator generates vector clock and writes data locally Coordinator sends new version (along with vector clock) to top N nodes from preference list If at least W-1 nodes respond, write is successfulGet() Coordinator requests all existing version of data from top N nodes from preference list Waits for at least R nodes to respond In case of multiple versions, coordinator send all casually unrelated versions to client Client reconcile divergent versions and supersede current version with reconciled version 31. Conditional Writes Multiple clients can access the same item and try to update the same attributes 32. Conditional Writes Multiple clients can access the same item and try to update the same attributes idempotent operation Running same request multiple times will have not effect 33. Strongly consistent vs Eventually Consistent Multiple copies of same item to ensure durability Takes time for update to propagate multiple copies Eventually consistent After write happens, immediate read may not give latest value Strongly consistency Requires additional read capacity unit Gets most up-to-date version of item value Eventually consistent read consumes half the read capacity unit as strongly consistent read 34. Provisioned throughput Configuring read and write capacity Read capacity 1 read unit = 4KB (item will be rounded off to multiple of 4KB) 1 read capacity = 1 read unit for strong consistency or 2 read unit for eventual consistency 1 read capacity = 4KB for strong consistency or 8KB for eventual consistency Write capacity 1 write unit = 1KB (item will be rounded off to multiple of 1KB) 1 write capacity = 1 write unit 1 write capacity = 1KB 35. Provisioned throughput Read Capacity provisioning Expected Item SizeConsistencyDesired Reads Per Provisioned Throughput Second Required4 KBStrongly consistent50508 KBStrongly consistent501004 KBEventually consistent50258 KBEventually consistent5050Write Capacity provisioning Expected Item SizeDesired Writes Per SecondProvisioned Throughput Required1 KB50502 KB50100 36. Operations vs Provisioned throughput OperationDescriptionProvisioning throughputCalculationGetItemReads only 1 itemItem rounded off to multiple of 4KBItem size = 400 bytes Read capacity = 1 (eventually consistent)BatchGetItemReads multiple items (max 1MB or 100 items)Every item rounded off to multiple of 4KBItem size = 400 bytes Number of items 100 Read capacity = 50 (eventually consistent)QueryReads multiple items (max 1MB or 100 items)Total output is rounded off to multiple of 4KBItem size = 400 bytes Number of items 100 Read capacity = 10 (eventually consistent)ScanReads entire table and then apply filterSize of table rounded off to multiple of 4KBItem size = 400 bytes Number of records in table = 10000 Read capacity = 500 (eventually consistent) 37. Query vs Scan Query Search based on primary key and examines only matching data Maximum 1MB result Query result sorted by range key Query result can be opted to give strong consistency Query can be done based on secondary index Requires less provisioned throughput Query performance depends on amount of data retrieved Scan Examines every item and then applies filter Maximum 1MB result Scan result is always eventually consistent Required high provisioned throughput Scan performance depends on table size Secondary index does not have any impact on scan performance Parallel scan can be performed using TotalSegments 38. DynamoDB Parameters LastEvaluatedKey Key of last returned value for operation > 1 MB If LastEvaluatedKey is null then all data is transfered ExclusiveStartKey Starting point of second query request for operation > 1MB Count During request if count = true, dynamoDB provides only count(*) from table ScannedCount Total number of items scanned in scan operation before applying any filter Limit Limit number of rows in result Segments Segment to be scanned by a worker in parallel scan TotalSegments Degree of parallelism set in parallel scan 39. Secondary Index For efficiently accessing data using keys other than primary key Can only be created on scalar data type Can include projections from table into index Primary key attributes always part of secondary index DynamoDB copies items from table to index 2 types of secondary index Local secondary index Global secondary index Access to index similar as access to table Indexes are maintained automatically Maximum 5 local and 5 global secondary indexes for a table 40. Local vs Global Secondary Index Local Secondary Index Hash + Range key Has the same hash key as primary key. Range key defers For each hash key total size of all indexed items >> import boto.dynamodb >>> conn = boto.dynamodb.connect_to_region( ... 'us-east-1', ... aws_access_key_id=', ... aws_secret_access_key=') >>> conn >>> conn.list_tables() ['emp', 'test', 'test1'] Create table:>>> message_table_schema = conn.create_schema( ... hash_key_name='forum_name', ... hash_key_proto_value=str, ... range_key_name='subject', ... range_key_proto_value=str ... ) >>> table = conn.create_table( ... name='messages', ... schema=message_table_schema, ... read_units=10, ... write_units=10 ... ) >>> print table Table(messages) 59. References https://docs.aws.amazon.com/amazondynamodb/ http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf http://home.aubg.bg/students/ENL100/Cloud%20Computing/Research%20Paper/nosqldbs.p df http://docs.aws.amazon.com/cli/latest/reference/dynamodb/index.html - CLI reference http://aws.amazon.com/sdkforpython/ - SDK for python http://docs.pythonboto.org/en/latest/dynamodb_tut.html - boto tutorial and command reference http://en.wikipedia.org/wiki/Consistent_hashing