c* summit eu 2013: cassandra internals

CASSANDRA EU 2013

CASSANDRA INTERNALS

Aaron Morton @aaronmorton

!

Co-Founder & Principal Consultant www.thelastpickle.com

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License #CassandraEU

http://www.thelastpickle.com

http://creativecommons.org/licenses/by-nc/3.0/nz/


About The Last Pickle. Work with clients to deliver and improve

Apache Cassandra based solutions.

Apache Cassandra Committer, DataStax MVP, Hector Maintainer, Apache Usergrid Committer.

Based in New Zealand & Austin, TX.

#CassandraEUwww.thelastpickle.com



Architecture Code




Cassandra Architecture.

API's

Cluster Aware

Cluster Unaware

Clients

Disk




Cassandra Cluster Architecture.

API's

Cluster Aware

Cluster Unaware

Clients

Disk

API's

Cluster Aware

Cluster Unaware

Disk

Node 1 Node 2




Dynamo Cluster Architecture.

API's

Dynamo

Database

Clients

Disk

API's

Dynamo

Database

Disk

Node 1 Node 2

www.thelastpickle.com #CassandraEU



Architecture API

Dynamo Database




API Transports. !

Thrift Native Binary

!




Thrift Transport. !

//Custom TServer implementations o.a.c.thrift.CustomTThreadPoolServer o.a.c.thrift.CustomTHsHaServer




API Transports.

Thrift Native Binary




Native Binary Transport. !

Beta in Cassandra 1.2, now GA. Uses Netty. CQL 3 only.




o.a.c.transport.Server.run() !

//Setup the Netty server new ExecutionHandler() new NioServerSocketChannelFactory() ServerBootstrap.setPipelineFactory()




o.a.c.transport.Message.Dispatcher.messageReceived() !

//Process message from client ServerConnection.validateNewMessage() Request.execute() ServerConnection.applyStateTransition() Channel.write()




Messages. !

Defined in the Native Binary Protocol

$SRC/doc/native_protocol.spec




API Services. !

JMX Thrift

CQL 3 !




JMX Management Beans. !

Spread around the code base.

Interfaces named *MBean




JMX Management Beans. !

Registered with names such as org.apache.cassandra.db:

type=StorageProxy




API Services. !

JMX Thrift CQL 3

!




o.a.c.thrift.CassandraServer !

// Implements Thrift Interface // Access control // Input validation // Mapping to/from Thrift and internal types




Thrift Interface. !

Thrift IDL $SRC/interface/cassandra.thrift




o.a.c.thrift.CassandraServer.get_slice() !

// get columns for one row Tracing.begin() ClientState cState = state() cState.hasColumnFamilyAccess() multigetSliceInternal() !




CassandraServer.multigetSliceInternal() !

// get columns for may rows ThriftValidation.validate*() // Create ReadCommands getSlice() !




CassandraServer.getSlice() !

// Process ReadCommands // return Thrift types !

readColumnFamily() thriftifyColumnFamily() !




CassandraServer.readColumnFamily() !

// Process ReadCommands // Return ColumnFamilies !

StorageProxy.read() !




API Services. !

JMX Thrift

CQL 3 !




o.a.c.cql3.QueryProcessor !

// Prepares and executes CQL3 statements // Used by Thrift & Native transports // Access control // Input validation // Returns transport.ResultMessage

!

!




CQL3 Grammar. !

ANTLR Grammar $SRC/o.a.c.cql3/Cql.g




o.a.c.cql3.statements.ParsedStatement !

// Subclasses generated by ANTLR // Tracks bound term count // Prepare CQLStatement prepare()




o.a.c.cql3.statements.CQLStatement !

checkAccess(ClientState state) validate(ClientState state) execute(ConsistencyLevel cl, QueryState state, List<ByteBuffer> variables)




statements.SelectStatement.RawStatement !

// Implements ParsedStatement // Input validation prepare()




statements.SelectStatement.execute() !

// Create ReadCommands StorageProxy.read()

www.thelastpickle.com #CassandraEU



Architecture API

Dynamo Database




Dynamo Layer. o.a.c.service

o.a.c.net !

o.a.c.dht o.a.c.gms

o.a.c.locator o.a.c.stream




o.a.c.service.StorageProxy !

// Cluster wide storage operations // Select endpoints & check CL available // Send messages to Stages // Wait for response // Store Hints




o.a.c.service.StorageService !

// Ring operations // Track ring state // Start & stop ring membership // Node & token queries




o.a.c.service.IResponseResolver !

preprocess(MessageIn<T> message) resolve() throws DigestMismatchException !

RowDigestResolver RowDataResolver RangeSliceResponseResolver




Response Handlers / Callback.

implements IAsyncCallback<T> !

response(MessageIn<T> msg) !




o.a.c.service.ReadCallback.get()

//Wait for blockfor & data response condition.await(timeout, TimeUnit.MILLISECONDS) !

throw ReadTimeoutException() !

resolver.resolve()




o.a.c.service.StorageProxy.fetchRows() !

getLiveSortedEndpoints() new RowDigestResolver() new ReadCallback() MessagingService.sendRR() --------------------------------------- ReadCallback.get() # blocking catch (DigestMismatchException ex) catch (ReadTimeoutException ex)




Dynamo Layer !

o.a.c.service o.a.c.net

!

o.a.c.dht o.a.c.gms

o.a.c.locator o.a.c.stream




o.a.c.net.MessagingService.verb<<enum>> !

MUTATION READ REQUEST_RESPONSE TREE_REQUEST TREE_RESPONSE

(And more...)




o.a.c.net.MessagingService.verbHandlers !

new EnumMap<Verb, IVerbHandler>(Verb.class)




o.a.c.net.IVerbHandler<T> !

doVerb(MessageIn<T> message, String id);

!




o.a.c.net.MessagingService.verbStages !

new EnumMap<MessagingService.Verb, Stage>(MessagingService.Verb.class)




o.a.c.net.MessagingService.receive() !

runnable = new MessageDeliveryTask( message, id, timestamp); !

StageManager.getStage( message.getMessageType()); !

stage.execute(runnable);




o.a.c.net.MessageDeliveryTask.run() !

// If dropable and rpc_timeout MessagingService.incrementDroppedMessages(v

erb); return; !

MessagingService.getVerbHandler(verb) verbHandler.doVerb(message, id)




Architecture API Layer

Dynamo Layer Database Layer




Database Layer !

o.a.c.concurrent o.a.c.db

!

o.a.c.cache o.a.c.io

o.a.c.trace




o.a.c.concurrent.StageManager !

stages = new EnumMap<Stage, ThreadPoolExecutor>(Stage.class); !

getStage(Stage stage)




o.a.c.concurrent.Stage !

READ MUTATION GOSSIP REQUEST_RESPONSE ANTI_ENTROPY

(And more...)#CassandraEUwww.thelastpickle.com



Database Layer. o.a.c.concurrent

o.a.c.db !

o.a.c.cache o.a.c.io

o.a.c.trace




o.a.c.db.Table !

// Keyspace open(String table) getColumnFamilyStore(String cfName) !

getRow(QueryFilter filter) apply(RowMutation mutation, boolean writeCommitLog)




o.a.c.db.ColumnFamilyStore !

// Column Family getColumnFamily(QueryFilter filter) getTopLevelColumns(...) !

apply(DecoratedKey key, ColumnFamily columnFamily, SecondaryIndexManager.Updater indexer)




o.a.c.db.IColumnContainer !

addColumn(IColumn column) remove(ByteBuffer columnName) !

ColumnFamily SuperColumn !

(Removed in 2.0)




o.a.c.db.ISortedColumns !

addColumn(IColumn column, Allocator allocator) removeColumn(ByteBuffer name) !

ArrayBackedSortedColumns AtomicSortedColumns TreeMapBackedSortedColumns




o.a.c.db.Memtable !

put(DecoratedKey key, ColumnFamily columnFamily, SecondaryIndexManager.Updater indexer) !

flushAndSignal(CountDownLatch latch, Future<ReplayPosition> context)




o.a.c.db.ReadCommand !

getRow(Table table) !

SliceByNamesReadCommand SliceFromReadCommand RangeSliceCommand

(Additional classes for paging in 2.0)




o.a.c.db.IDiskAtomFilter !

getMemtableColumnIterator(...) getSSTableColumnIterator(...) !

IdentityQueryFilter NamesQueryFilter SliceQueryFilter




Summary CustomTThreadPoolServer Message.Dispatcher

CassandraServer QueryProcessor

ReadCommand

StorageProxy

IResponseResolver

IAsyncCallback

MessagingService

IVerbHandler

Table ColumnFamilyStore IDiskAtomFilter

API

Dynamo

Database




Thanks. !




Aaron Morton @aaronmorton

www.thelastpickle.com !

Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License


http://creativecommons.org/licenses/by-nc/3.0/nz/

c* summit eu 2013: cassandra internals

Technology

thrift transport

thrift native binary

jmx thrift cql

thrift types

cassandra architecture

locator o

cassandra cluster architecture

cql3 statements