distributed computing with hazelcast - javaone 2014

DISTRIBUTED COMPUTINGWITH HAZELCAST

www.hazelcast.com

DISCLAIMERThis presentation may include certain forward-looking statements and projections provided bythe Company. Any such statements and projections reflect various estimates and assumptionsby the Company concerning anticipated results. These statements involve risks, uncertainties

and assumptions and are based on the current estimates and assumptions of the managementof the Company as of the date of this presentation and are subject to uncertainty and changes.

No representations or warranties are made by the Company as to the accuracy of any suchstatements or projections. Whether or not any such forward-looking statements or projectionsare in fact achieved will depend upon future events some of which are not within the control of

the Company. Accordingly, actual results may vary from the projected results and suchvariations may be material. Statements contained herein describing documents and

agreements are summaries only and such summaries are qualified in their entirety by referenceto such documents and agreements.

www.hazelcast.com

THAT'S MEChris(toph) EngelbertTwitter: @noctarius2k8+ years of Java WeirdonessPerformance, GC, traffic topicsApache CommitterGaming, Travel Management, ...

www.hazelcast.com

OUR SPACE TRIP ROADMAPHazelcastDistributed ComputingDistributed ExecutorServiceEntryProcessorMap & ReduceQuestions

www.hazelcast.com

HAZELCASTPICKIN' DIAMONDS

www.hazelcast.com

WHAT IS HAZELCAST?In-Memory Data-GridData Partioning (Sharding)Java Collections ImplementationDistributed Computing Platform

www.hazelcast.com

WHY HAZELCAST?Automatic PartitioningFault ToleranceSync / Async BackupsFully DistributedIn-Memory for Highest Speed

www.hazelcast.com

WHY HAZELCAST?

www.hazelcast.com

www.hazelcast.com

WHY IN-MEMORYCOMPUTING?

www.hazelcast.com

TREND OF PRICES

Data Source: http://www.jcmit.com/memoryprice.htm

www.hazelcast.com

SPEED DIFFERENCE

Data Source: http://i.imgur.com/ykOjTVw.png

www.hazelcast.com

DISTRIBUTEDCOMPUTING

OR

MULTICORE CPU ON STEROIDS

www.hazelcast.com

THE IDEA OF DISTRIBUTED COMPUTING

Source: https://www.flickr.com/photos/stefan_ledwina/1853508040

www.hazelcast.com

THE BEGINNING

Source: http://en.wikipedia.org/wiki/File:KL_Advanced_Micro_Devices_AM9080.jpg

www.hazelcast.com

MULTICORE IS NOT NEW

Source: http://en.wikipedia.org/wiki/File:80386with387.JPG

www.hazelcast.com

CLUSTER IT

Source: http://rarecpus.com/images2/cpu_cluster.jpg

www.hazelcast.com

SUPER COMPUTER

Source: http://www.dkrz.de/about/aufgaben/dkrz-geschichte/rechnerhistorie-1

www.hazelcast.com

CLOUD COMPUTING

Source: https://farm6.staticflickr.com/5523/11407118963_e0e0870846_b_d.jpg

www.hazelcast.com

DISTRIBUTEDEXECUTORSERVICE

THE WHOLE CLUSTER IN YOUR HANDS

www.hazelcast.com

WHY A DISTRIBUTED EXECUTORSERVICE?j.l.Runnable / j.u.c.CallableOnly needs to be serializableSame Task all / multiple NodesShould not work on Data

www.hazelcast.com

Talk to me from all nodes

QUICK EXAMPLE

Runnable runnable = () -> println( "Hello from my Cluster-World" );IExecutorService es = hzInstance.getExecutorService( "default" );es.executeOnAllMembers( runnable );

www.hazelcast.com

DEMONSTRATION

www.hazelcast.com

ENTRYPROCESSORLOCKFREE DATA OPERATIONS

www.hazelcast.com

WHY ENTRYPROCESSOR?Prevents external LockingGuarantees AtomicityKind of a "Cluster-wide Thread-Safe"

www.hazelcast.com

Incrementing a counter atomically

QUICK EXAMPLE

private int increment( Map.Entry entry ) { val newValue = entry.getValue() + 1; entry.setValue( newValue ); return newValue;}

IMap map = hazelcastInstance.getMap( "default" );int newId = map.executeOnKey( "idgen", this::increment );

www.hazelcast.com

DEMONSTRATION

www.hazelcast.com

MAP & REDUCETHE BLACK MAGIC FROM PLANET GOOGLE

www.hazelcast.com

USE CASESLog AnalysisData QueryingAggregation and summingDistributed SortETL (Extract Transform Load)and more...

www.hazelcast.com

SIMPLE STEPSReadMap / TransformReduce

www.hazelcast.com

FULL STEPSReadMap / TransformCombiningGrouping / ShufflingReduceCollating

www.hazelcast.com

MAPREDUCE WORKFLOW

www.hazelcast.com

Data are mapped / transformed in a set of key-value pairs

SOME PSEUDO CODE (1/3)

MAPPING

map( key:String, document:String ):Void -> for each w:Word in document: emit( w, 1 )

www.hazelcast.com

Multiple values are combined to an intermediate result to preserve traffic

SOME PSEUDO CODE (2/3)

COMBINING

combine( word:Word, counts:List[Int] ):Void -> emit( word, sum( counts ) )

www.hazelcast.com

Values are reduced / aggregated to the requested result

SOME PSEUDO CODE (3/3)

REDUCING

reduce( word:String, counts:List[Int] ):Int -> return sum( counts )

www.hazelcast.com

FOR MATHEMATICIANSProcess: (K x V)* → (L x W)* ⇒ [(l1, w1), …, (lm, wm)]

Mapping: (K x V) → (L x W)* ⇒ (k, v) → [(l1, w1), …, (ln, wn)]

Reducing: L x W* → X* ⇒ (l, [w1, …, wn]) → [x1, …,xn]

www.hazelcast.com

MAPREDUCE PROGRAMS INGOOGLE SOURCE TREE

Source: http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0005.html

www.hazelcast.com

DEMONSTRATION

www.hazelcast.com

@noctarius2k@hazelcast

http://www.sourceprojects.comhttp://github.com/noctarius

THANK YOU!ANY QUESTIONS?

www.hazelcast.com