Performance Evaluation of a MongoDB and Hadoop Platform for Scientific Data Analysis

E. Dede, M.Govindaru (University Binghamton), D.Gunter, R. Canon, L. Ramakrishnan (Lawrence Berekely National Lab)

http://dl.acm.org/citation.cfm?id=2465849

Introduction

MongoDB : NoSQL DBMSHadoop : Framework -> Big Data ProcessingHDFS : Hadoop FileSystemMapReduce : for scalable Parallel Analysis

MotivationAdvanced Light Source (ALS) & Joint Genome Institute have project -> Materials Project.Now ALS & Materials Projects use MongoDB+MapReduceHadoop -> most popular FOSS implementation of MapReduce.How if we make a comparison of both ->

evaluate the performancescalabilityfault tolerance

System OverviewNow the ALS have system with native MogoDB+MapReduce, and they want to know and compare about the Hadoop MongoDB vs (native Hadoop)Hadoop HDFS

Experiment Setup● Hadoop Setup using Standard Setup● Data: US Census data 300 GB● MongoDB : run with 2 mode, single and sharding● Hopper Machine : Cray XE6 with 153,216 compute

cores, 217 TB RAM, 2 PetaBytes Disk, Each Node has 24 cores, 2 twelve-core AMD 2.1 GHz processor and 32 GB RAM.

Result

Evaluate PerformanceScalabilityFault Tolerance

Evaluate Performance

Scalability 37.2 milion input records

Fault Tolerancetotal 32 node processing 37 milion input records.

Hadoop after 8 faulted nodes Hadoop loses too many data nodes and fails to complete the MapReduce Job.

MogoDB-Hadoop can finish job even with half of the cluster lost.

Conclusion & Contribution

Case -> Scientific Big Data Processing● HDFS-Hadoop is better than MongoDB for Read and

Write Performance.● HDFS-Hadoop is better than MongoDB-hadoop for

Scalability case.● MongoDB-hadoop is better than HDFS-Hadoop for

Fault Tolerance.

Thank you

Any Questions?